The translation of the article was prepared for students of the Linux Administrator course.

Earlier, I talked about how to test and enable the use of Hugepages in Linux.
This article will be useful only if you really have where to use Hugepages. I have met many people who are deceived by the prospect that Hugepages will magically improve performance. Still, hugepaging is a complex topic, and if used improperly, it can reduce performance.

Part 1: Verify that hugepages are included in Linux (the original is here )

Problem:
You need to check if HugePages are enabled on your system.

Decision:
It is quite simple:

cat /sys/kernel/mm/transparent_hugepage/enabled 

You will get something like this:

 always [madvise] never 

You will see a list of available options ( always, madvise, never ), while the current active option will be enclosed in parentheses (default madvise ).

madvise means that transparent hugepages enabled only for memory areas that explicitly request hugepages using madvise (2) .

always means that transparent hugepages are always on and for all processes. This usually improves performance, but if you have a use case where many processes consume a small amount of memory, then the total memory load may increase dramatically.

never means that transparent hugepages will not turn on even when requested using madvise. To learn more, consult your Linux kernel documentation .

How to change the default value

Option 1 : Directly change sysfs (after a reboot, the parameter will return to the default value):

 echo always >/sys/kernel/mm/transparent_hugepage/enabled echo madvise >/sys/kernel/mm/transparent_hugepage/enabled echo never >/sys/kernel/mm/transparent_hugepage/enabled 

Option 2 : Change the system default by recompiling the kernel with the changed configuration (this option is recommended only if you are using your own kernel):

• To set always to default, use:
  

   CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS=y # Comment out CONFIG_TRANSPARENT_HUGEPAGE_MADVISE=y 

• To set madvise by default, use:
  

   CONFIG_TRANSPARENT_HUGEPAGE_MADVISE=y # Comment out CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS=y 

Part 2: Advantages and disadvantages of HugePages

We will try to selectively explain the advantages, disadvantages and possible errors when using Hugepages. Since a technologically complex and pedantic article is likely to be hard for people to understand who are deceived, considering Hugepages to be a panacea, I will sacrifice accuracy for the sake of simplicity. Just keep in mind that many topics are really complex and therefore greatly simplified.

Please note that we are talking about 64-bit x86 systems running on Linux, and that I simply assume that the system supports transparent hugepages (since it is not a drawback that hugepages are not replaced), as it happens in almost any modern Linux environment.

In the links below I will attach more technical descriptions.

Virtual memory

If you are a C ++ programmer, you know that objects in memory have specific addresses (pointer values).

However, these addresses do not necessarily reflect the physical addresses in memory (addresses in RAM). They are addresses in virtual memory. The processor has a special MMU (memory management unit) module that helps the kernel match virtual memory with physical location.

This approach has many advantages, but the most basic of them are:

• Performance (for various reasons);
• Isolation of programs, that is, none of the programs can read from the memory of another program.

What are pages?

Virtual memory is divided into pages. Each individual page points to a specific physical memory, it may indicate an area in RAM, or it may indicate an address assigned to a physical device, such as a video card.

Most of the pages you are dealing with point to either RAM or are swapped, that is, they are stored on a hard disk or SSD. The kernel controls the physical layout of each page. If you access a spoofed page, the kernel stops the thread that is trying to access the memory, reads the page from the hard disk / SSD into RAM, and then continues the thread.

This process is transparent to the stream, that is, it does not necessarily read directly from the hard disk / SSD. The size of normal pages is 4096 bytes. Hugepages size is 2 megabytes.

Associative translation buffer (TLB)

When a program accesses a page of memory, the CPU must know which physical page to read data from (that is, have a virtual address card).

The kernel has a data structure (page table) that contains all the information about the pages used. Using this data structure, you can map a virtual address to a physical address.

However, the page table is quite complex and slow, so we simply cannot analyze the entire data structure every time when a process accesses memory.

Fortunately, there is a TLB in our processor that caches the mapping of virtual and physical addresses. This means that despite the fact that we need to analyze the page table during the first attempt to gain access, all subsequent references to the page can be processed in the TLB, which ensures quick work.

Since it is implemented as a physical device (which makes it primarily fast), its capacity is limited. Therefore, if you want to access more pages, TLB will not be able to store a match for all of them, and as a result your program will run much slower.

Hugepages come to the rescue

So what can we do to avoid a TLB overflow? (We assume that the program still needs the same amount of memory).

This is where the Hugepages appear. Instead of 4096 bytes, requiring just one TLB entry, one TLB entry can now indicate a whopping 2 megabytes. We will assume that the TLB has 512 entries; here, without Hugepages, we can match:

 4096 b⋅512=2 MB 

Whereas we can compare with them:

 2 MB⋅512=1 GB 

That is why Hugepages is awesome. They can improve performance without much effort. But there are significant reservations.

Substitution Hugepages

The kernel automatically keeps track of how often each page of memory is used. If there is not enough physical memory (RAM), the kernel will move less important (less commonly used) pages to the hard disk in order to free up some of the RAM for more important pages.
In principle, the same goes for Hugepages. However, the kernel can swap only whole pages, not individual bytes.

Suppose we have this program:

 char* mymemory = malloc(2*1024*1024); //     Hugepage! //  mymemory -  //    , //      mymemory // ... //       putchar(mymemory[0]); 

In this case, the kernel will need to replace (read) as much as 2 megabytes of information from the hard disk / SSD only for you to read one byte. As for ordinary pages, from a hard disk / SSD it is necessary to read only 4096 bytes.

Therefore, if the hugepage is replaced, its reading is faster only if you need to access the entire page. This means that if you are trying to access randomly to different parts of the memory and just read a couple of kilobytes, you should use regular pages and do not worry about anything else.

On the other hand, if you need to access much memory consistently, hugepages will increase your productivity. However, you need to check it yourself (and not using the example of abstract software) and see what works faster.

Memory allocation

If you write in C, you know that you can request arbitrarily small (or almost arbitrarily large) heap memory with malloc() . Suppose you need 30 bytes of memory:

 char* mymemory = malloc(30); 

It may seem to the programmer that you are “requesting” 30 bytes of memory from the operating system and returning a pointer to some virtual memory. But in fact, malloc () is simply a C function that calls the inside of the brk and sbrk functions to request or free memory from the operating system.

However, it is inefficient to request more and more memory for each location; Most likely, a memory segment has already been freed (free()) , and we can reuse it. malloc() implements quite complex algorithms for reusing freed memory.

At the same time, everything happens unnoticed for you, so why should you care? Because the free() call does not mean that the memory is returned immediately to the operating system .

There is such a thing as memory fragmentation. In extreme cases, there are heap segments where only a few bytes are used, while everything between them has been freed (free()) .

Please note that memory fragmentation is an incredibly complex topic, and even minor changes in the program can significantly affect it. In most cases, programs do not cause significant memory fragmentation, but you should keep in mind that if there is a problem with fragmentation in a certain area of ​​the heap, hugepages can only aggravate the situation.

Selective application of hugepages

After reading the article, you have determined which parts of your program can benefit from using hugepages, and which parts are not. So should hugepages even be included?

Fortunately, you can use madvise() to enable hugepaging only for those memory areas where it will be useful.

To get started, check that hugepages work in madvise () mode, using the instructions at the beginning of the article.

Then, use madvise() to tell the kernel exactly where to use hugepages.

 #include <sys/mman.h> //    ,    size_t size = 256*1024*1024; char* mymemory = malloc(size); //   hugepages… madvise(mymemory, size, MADV_HUGEPAGE); // …    madvise(mymemory, size, MADV_HUGEPAGE | MADV_SEQUENTIAL) 

Note that this method is just a recommendation to the kernel for managing the memory. This does not mean that the kernel will automatically use hugepages for a given memory.

Refer to the madvise manpage documentation to learn more about memory management and madvise() , this topic has an incredibly steep learning curve. Therefore, if you intend to really understand it well, prepare to read and test for a few weeks before relying on at least some positive result.

What to read?

• IBM OpenStack article on hugepages
• Transparent hugepages vs non-transparent hugepages
• WLipedia article about TLB
• Linux kernel documentation on transparent hugepages
• StackOverflow: Hugepages are prone to memory fragmentation
• Microsoft Virtual Memory Management Article

Have a question? Write in the comments!