How malloc memory eats

No, there will not be anything from the series “Aaaa, I made malloc (new), and forgot to make free (delete)!”
There will be something sophisticated here: we will cut the pieces of memory a little bit, hide them in a secluded place ... And when the operating system pays the ransom, says, “Enough!”, We will try to get everything back. It would seem that the simplest operation of allocating and freeing memory does not portend trouble.
Those who are interested in how to destroy memory clog - I ask for habrakat

A little background

On duty, you have to work a lot with large memory buffers (imagine an image of 5000x40000 pixels). Sometimes (due to fragmentation) it’s impossible to allocate a continuous piece of memory for everything. Therefore, a memory manager was written, which allocated as much as possible, perhaps in a few pieces. Naturally, the memory manager must both allocate and delete. Then the following interesting thing was discovered: The Task Manager, after being released, shows the level of memory usage the same as before block allocation. However, no new block of memory in the program can be allocated. The use of virtual memory analysis tools (VMMap from Mark Rusinovich) shows that the memory remains busy despite its release in the code and despite the TM readings.

Analysis

Let's write quickly some program that allocates and frees memory. Something like a “Hello World!”:

int main(void) { const int blockCount = 1024; const int blockSize = 1024*1024; char **buf; printf("Hit something...\n"); getchar(); buf = (char**)malloc(blockCount*sizeof(char*)); for (int i=0; i<blockCount; i++) { buf[i] = (char*)malloc(blockSize*sizeof(char)); } printf("Memory allocated\n"); printf("Hit something...\n"); getchar(); for (int i=0; i<blockCount; i++) { free(buf[i]); } free(buf); printf("Hit something...\n"); printf("Memory freed\n"); getchar(); return 0; } 

Simple calculations can make sure that the program should allocate 1 GB of memory, and then release everything. After start and check all memory is released. Hmm, it seems the blackmail system does not give in. However, we cut large pieces.
')
Now let's take a bit and fix the source code:

 const int blockSize = 520133 // ...; 

In this case, we get that the memory is allocated, but not freed:
To "Memory freed":

After “Memory freed”:


Inquisitive mind of the programmer did not stop there! I began to look for the threshold at which this effect occurs. After a brief binary selection, it turned out that with a size equal to

• 520168 bytes and above - the release is normal
• 520167 bytes and below - we have the problem described

Looking ahead, I will say that I couldn’t explain this threshold in any way. It is not even divided into 1024!

Possible explanation

After long vigils over Google and studying forums, I came to the following conclusions.
It turns out that after allocating memory using malloc / new functions, if a small piece is allocated, the memory is not freed by free / delete functions, but is transferred from the category of committed to the category of reserved. And if we access this memory right there after deletion (apparently within the framework of one heap), then it can be re-allocated. However, when trying to allocate memory from another class (or a static function), we get an exception - not enough memory. Apparently, when allocating memory from a static function, the memory is not allocated in the same heap as in the normal allocation from the inside of the application class.
As a result, after creating a large block of memory (from small pieces), we run out of memory and cannot further allocate to ourselves a little more at least a little bit! of memory.

Wrong decision

Using the functions of VirtualAlloc / VirtualFree ( MSDN ) solves this problem, the memory is fully returned to the process after use (MEM_RELEASE key), however when using VirtualAlloc there is a strong memory fragmentation, and somewhere 800MB of memory is not available for use, because The maximum size of a free block is 28KB. Classic malloc works better in this regard, because there is some defragmenter.

Final decision

Found a third-party implementation of malloc and free (as it turns out, widely known in narrow circles), which has a classic lack of defragmentation of memory, but in a place that frees up completely memory after use. Plus, it also works much faster.
For the curious and thirsty there is a link

Remarks

Under OS * NIX (Ubuntu, Debian, CentOS), the problem could not be repeated)
Under Windows, the problem was reproduced on Windows Server 2003 x64, Windows 7 x64, Windows XP x32.
You shouldn’t immediately trust in long-proven functions, they can be a trick.

UPD: MS VS 2010 is used for compilation on Windows