Mailing List Archive

On Thu, May 2, 2013 at 3:33 PM, Frank Peters <frank.peters@comcast.net> wrote:
> During kernel configuration there is an option called CONFIG_TRANSPARENT_HUGEPAGE
> which allows the use of Transparent Hugepages or application memory pages larger
> than the traditional 4K size. This option has been available for a while but
> I've never enabled it. However, it would seem like a good idea to use.
>
> Searching for more information I can find no comments on actual Hugepage
> performance for a simple desktop Linux system. Can anyone verify or refute
> the idea that Transparent Hugepages will lead to improvements on a desktop
> system? Does an application have to specifically request Hugepages or
> does the allocation occur for all applications?

AFAIR there were claims of up to 10% performance improvement in
certain test scenarios, and that you don't need to do anything special
to exploit THP in existing software (but applications can be rewritten
in such a way as to specifically make better use of it).

There is a setting that lets you expose THP _only_ to those programs
which specifically request it, rather than backwards-compatible to all
programs if you are worried about that.

Check out the in-kernel documentation (vm/transhuge.txt) for details.

Anecdotal: I have turned it off and on and don't "feel" any difference
in the operation of my computer either way...

Frank Peters posted on Thu, 02 May 2013 16:33:09 -0400 as excerpted:

> During kernel configuration there is an option called
> CONFIG_TRANSPARENT_HUGEPAGE which allows the use of Transparent
> Hugepages or application memory pages larger than the traditional 4K
> size. This option has been available for a while but I've never enabled
> it. However, it would seem like a good idea to use.
>
> Searching for more information I can find no comments on actual Hugepage
> performance for a simple desktop Linux system. Can anyone verify or
> refute the idea that Transparent Hugepages will lead to improvements on
> a desktop system? Does an application have to specifically request
> Hugepages or does the allocation occur for all applications?

In addition to what Paul says (which is generally correct, AFAIK)...

As I understand it, the whole idea of "huge pages" is based on making
better use of a relatively limited hardware resource, the TLB entries,
translation lookaside buffer entries. The TLB is a sort of memory
mapping cache index, the memory mapping itself being between virtual and
physical memory. The technical details of TLB are beyond me, so I'll
simply refer you to wikipedia:

http://en.wikipedia.org/wiki/Translation_lookaside_buffer

But basically, the TLB is an index with a limit of (from wikipedia)
12-4096 entries, depending on architecture and specific CPU.

The key is that each memory page requires an entry, and normal memory
pages are 4k in size, so normal TLB coverage is relatively small, a few
megabytes of RAM can be indexed at a time, that's it.

Which is where huge pages come in, since each huge page entry requires
the same index line as a normal page would, but indexes and allows fast
access to a far larger bit of memory, often 2 MB. If all TLB line
entries were for huge-pages, then, coverage would be several GB instead
of several MB.

The problem was to implement that in the kernel. Originally, there was a
huge page interface that individual apps could use, but only if they
specifically requested it, which of course required that they be
rewritten in ordered to do so.

Which is where the transparent comes in. With transparent huge-pages,
apps work as normal -- it's the kernel that manages the huge pages
"transparently".

Practical value depends on your work load, but in general, it won't hurt.

Two further notes: 1) Because I regularly run mainline git kernels, I was
exposed to the option early on, and based on the coverage in (2) below, I
enabled them. But at one point in a pre-release kernel there was a bug,
of course long since fixed. The details aren't important as it is long
fixed, but due to that bug, I *DID* discover ONE app that ends up using
huge-pages a LOT -- firefox! Evidently, huge pages only really come into
play when individual app memory usage is relatively high, and firefox hit
that threshold, as well as some other requirements for huge pages, plus
some corner-cases that were required to hit this particular bug (which
was with zero-pages, in the second kernel that had huge-page support as
they added zero-page support, a different "trick" that wasn't initially
applied to huge-pages, only normal pages).

So for that period, I simply disabled huge-page support again, until the
bug was fixed. Then I enabled it and haven't disabled it since.

2) LWN has several articles covering kernel huge-page support as well as
transparent-huge-pages, including a 5-part-series covering the initial
non-transparent version, the first part (with further links to the other
parts) of which is found here:

http://lwn.net/Articles/374424/

That'll let you get pretty deep into the technology, if you like.
There's benchmarks, ways to discover what the TLBs are for your hardware,
etc.

Here's a google search limited to site:lwn.net for transparent huge pages:

http://www.google.com/search?q=site%3Alwn.net+transparent+huge+pages

Of interest, down the first page of hits a bit you can see overage of the
"adding a huge zero page" discussion, the source of the bug I mentioned
above, again, now long fixed.

That should give you quite enough information to get you started. =:^)

--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman

On Thu, 2 May 2013 16:05:10 -0500
Paul Hartman <paul.hartman+gentoo@gmail.com> wrote:

>
> There is a setting that lets you expose THP _only_ to those programs
> which specifically request it, rather than backwards-compatible to all
> programs if you are worried about that.
>

I don't really expect an answer, but what programs incorporate the
MADV_HUGEPAGE directive in their source code? Since THP is relatively
new for the Linux kernel, it would seem that most Linux software has not
yet included MADV_HUGEPAGE and could not benefit from THP if it is
limited to madvise only.

Also, if a program does include MADV_HUGEPAGE would it need to be
recompiled with a THP enabled kernel?

These are some questions I will have to research.

However, the madvise manpage has this to say about THP:

"This feature is primarily aimed at applications that use large mappings
of data and access large regions of that memory at a time (e.g., virtualization
systems such as QEMU)."

I wonder if image or sound processing applications are using MADV_HUGEPAGE.

Frank Peters

Frank Peters posted on Thu, 02 May 2013 18:51:42 -0400 as excerpted:

> On Thu, 2 May 2013 16:05:10 -0500 Paul Hartman
> <paul.hartman+gentoo@gmail.com> wrote:
>
>> There is a setting that lets you expose THP _only_ to those programs
>> which specifically request it, rather than backwards-compatible to all
>> programs if you are worried about that.
>>
> I don't really expect an answer, but what programs incorporate the
> MADV_HUGEPAGE directive in their source code? Since THP is relatively
> new for the Linux kernel, it would seem that most Linux software has not
> yet included MADV_HUGEPAGE and could not benefit from THP if it is
> limited to madvise only.

As suggested by the manpage you quote below, it's primarily big-memory
applications, including databases, virtualization, etc. Those apps stand
to gain enough from it (and have enough big-money sponsored development
behind them) that despite the feature being relatively new and Linux
specific, their newest releases are likely to support it.

> Also, if a program does include MADV_HUGEPAGE would it need to be
> recompiled with a THP enabled kernel?

AFAIK it wouldn't actually need rebuilt against a supporting kernel,
specifically, but it may need built against a supporting linux-headers
package. When built against a supporting linux-headers, the app would
detect at runtime whether the running kernel has the feature available or
not, and use it if so, but should still run just fine, only without that
optimization, on earlier kernels or those built without the feature
enabled.

> These are some questions I will have to research.
>
> However, the madvise manpage has this to say about THP:
>
> "This feature is primarily aimed at applications that use large
> mappings of data and access large regions of that memory at a time
> (e.g., virtualization systems such as QEMU)."
>
> I wonder if image or sound processing applications are using
> MADV_HUGEPAGE.

I don't know about media apps. It may be that despite their memory
usage, their access pattern doesn't benefit from huge-pages... or maybe
it does... I simply don't know.

Meanwhile, when you wrote the above you probably hadn't read my earlier
reply as they likely crossed on the net, and I don't know if it
specifically uses MADV_HUGEPAGE or not, but as I said in that reply,
firefox definitely uses transparent huge-pages if they're generally
available. I know this due to the zero-huge-page related bug I had with
an early pre-release of the kernel that enabled that, since it was firefox
triggering that bug. I temporarily turned transparent-hugepage support
off due to that, but turned it back on again when the bug was fixed,
before that kernel was released as a .0, and haven't had a huge-pages
related problem since. But due to that now fixed bug, I definitely know
at least one app that makes use of huge-pages, here. =:^)

If you plan on doing a bunch of research on the subject, I strongly
suggest that you look at the links I posted in the previous reply. LWN's
well respected in the Linux and particularly kernel community as a
relatively authoritative source, and you could certainly do worse than LWN
as a source for research on the subject, but may not be able to find
better.

--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman

On Fri, 3 May 2013 05:13:44 +0000 (UTC)
Duncan <1i5t5.duncan@cox.net> wrote:

>
> firefox definitely uses transparent huge-pages if they're generally
> available.
>

According to this early benchmark at Phoronix, the improvement for
system-wide THP use is not that impressive:

http://www.phoronix.com/scan.php?page=article&item=linux_transparent_hugepages&num=1

I think it would probably be best to enable THP in the kernel limited
to MADV_HUGEPAGE only.

Frank Peters

Am 03.05.2013 18:12, schrieb Frank Peters:
> On Fri, 3 May 2013 05:13:44 +0000 (UTC)
> Duncan <1i5t5.duncan@cox.net> wrote:
>
>>
>> firefox definitely uses transparent huge-pages if they're generally
>> available.
>>
> According to this early benchmark at Phoronix, the improvement for
> system-wide THP use is not that impressive:
>
> http://www.phoronix.com/scan.php?page=article&item=linux_transparent_hugepages&num=1
>
> I think it would probably be best to enable THP in the kernel limited
> to MADV_HUGEPAGE only.
>

why? it has no downsides. Apps use transparent hugepages without knowing
it (that is what the magical transparent means). It does not slow down
things.

Just turn it on.