Anatomy of the program in memory
Memory management is one of the main tasks of the OS. It is critical for both programming and system administration. I will try to explain how the OS works with memory. The concepts will be of a general nature, and I will take examples from Linux and Windows on 32-bit x86. First I will describe how the programs are located in memory.
Each process in a multitasking OS works in its sandbox in memory. This is a virtual address space, which in 32-bit mode is a 4GB block of addresses. These virtual addresses are mapped to physical memory with page tables that are supported by the OS kernel. Each process has its own set of tables. But if we begin to use virtual addressing, we have to use it for all programs running on the computer — including the kernel itself. Therefore, a portion of the virtual address space must be reserved for the kernel.
')
This does not mean that the kernel uses so much physical memory - it just has a part of the address space at its disposal that can be assigned to the necessary amount of physical memory. The memory space for the kernel is marked in the page tables as exclusively used by the privileged code, so if a program tries to access it, a page fault happens. In Linux, the memory space for the kernel is constantly present, and sets in correspondence the same part of the physical memory of all processes. Kernel code and data always have addresses, and are ready to handle interrupts and system calls at any time. For user programs, on the contrary, the correspondence of virtual addresses of real memory changes when the switching process occurs:
Blue indicates virtual addresses corresponding to physical memory. White is the space to which addresses are not assigned. In our example, Firefox uses much more space in virtual memory because of its legendary gluttony. The strips in the address space correspond to memory segments such as heap, stack, and so on. These segments are just memory address intervals, and have nothing to do with segments from Intel. Here is the standard segment diagram for a process under Linux:
When programming was white and fluffy, the initial virtual segment addresses were the same for all processes. This made it easy to remotely exploit security vulnerabilities. Malicious programs often need to access memory at absolute addresses — the stack address, the library function address, and so on. Remote attacks had to be done blindly, counting on the fact that all address spaces remain on permanent addresses. In this regard, the random address selection system has gained popularity. Linux randomizes the stack, the memory map segment and the heap by adding offsets to their starting addresses. Unfortunately, in the 32-bit address space you can’t really expand, and there is not enough space to assign random addresses, which makes this system not very effective.
The topmost segment in the process's address space is the stack, which in most languages ​​stores local variables and function arguments. Calling a method or function adds a new stack frame (stack frame) to an existing stack. After returning from the function, the frame is deleted. This simple scheme leads to the fact that to track the contents of the stack does not require any complex structure - just enough pointer to the beginning of the stack. Adding and deleting data becomes a simple and unambiguous process. The constant reuse of memory areas for the stack causes these parts to be cached in the CPU, which adds speed. Each thread in the process gets its own stack.
You can come to a situation in which the memory allocated for the stack ends. This results in a page fault, which in Linux is handled by the expand_stack () function, which, in turn, calls acct_stack_growth () to check if the stack can still be expanded. If its size does not exceed RLIMIT_STACK (usually it is 8 MB), then the stack grows and the program continues execution, as if nothing had happened. But if the maximum stack size is reached, we get a stack overflow and the program receives a Segmentation Fault error. At the same time, the stack can only grow - like the state budget, it does not decrease back.
Dynamic stack growth is the only situation in which free memory can be accessed, which is shown in white in the diagram. All other attempts to access this memory cause a page fault error leading to a Segmentation Fault. And some occupied memory areas are read-only, so attempts to write to these areas also result in a Segmentation Fault.
After the stack comes the memory mapping segment. Here the kernel places the contents of the files directly in memory. Any application can request to do this through the mmap () system call on Linux or CreateFileMapping () / MapViewOfFile () on Windows. This is a convenient and fast way to organize input and output operations in files, so it is used to load dynamic libraries. It is also possible to create an anonymous memory location, not associated with the files that will be used for the program data. If you make a request to Linux for large amounts of memory through malloc (), the C library will create such an anonymous mapping instead of using heap memory. By “large” is meant a volume larger than MMAP_THRESHOLD (128 kB by default, it is configured via mallopt ().)
The heap itself is located at the following positions in memory. It provides memory allocation during program execution, as well as the stack - but, unlike it, it stores the data that must survive the function that allocates them. Most languages ​​have tools for managing the heap. In this case, the satisfaction of the request for the allocation of memory is performed jointly by the program and the kernel. In With the heap interface, malloc () is used with friends, and in a language that has automatic garbage collection, such as C #, the interface is the keyword new.
If there is not enough space on the heap to fulfill the request, the program itself can handle the problem without the intervention of the kernel. Otherwise, the heap is incremented by the brk () system call. Heap management is a tricky business, it requires ingenious algorithms that tend to work quickly and efficiently to cater to the chaotic method of data placement used by the program. The time to process a request for a heap can vary widely. Real-time systems have special tools for working with it. Heaps can also be fragmented:
And so we got to the very bottom of the scheme - BSS, data and program text. BSS and data store static (global) variables in C. The difference is that BSS stores the contents of non-initialized static variables whose values ​​were not set by the programmer. In addition, the BSS is anonymous, it does not correspond to any file. If you write
The data segment, on the contrary, contains those variables that were initialized in the code. This part of the memory corresponds to the binary image of the program containing the initial static values ​​specified in the code. If you write
The sample data in the diagram will be a bit more complicated because it uses a pointer. In this case, the contents of the pointer, the 4-byte memory address, lives in the data segment. And the line on which it shows lives in a segment of text that is intended only for reading. It stores all the code and various other details, including string literals. It also keeps your binary in memory. Attempts to write to this segment end with an error Segmentation Fault. This prevents pointer-related errors (although not as effective as if you didn’t use C at all). The diagram shows these segments and examples of variables:
You can study the memory areas of a Linux process by reading the / proc / pid_of_process / maps file. Note that a single segment can contain many areas. For example, each file duplicated into memory has its own region in the mmap segment, and dynamic libraries have additional regions that resemble BSS and data. By the way, sometimes when people say "data segment", they mean data + bss + heap.
Binary images can be studied using the commands nm and objdump - you will see characters, their addresses, segments, etc. The virtual address scheme described in this article is so-called. The “flexible” scheme, which has been used by default for several years. It implies that a value has been assigned to the RLIMIT_STACK variable. Otherwise, Linux uses the “classic” scheme:
Each process in a multitasking OS works in its sandbox in memory. This is a virtual address space, which in 32-bit mode is a 4GB block of addresses. These virtual addresses are mapped to physical memory with page tables that are supported by the OS kernel. Each process has its own set of tables. But if we begin to use virtual addressing, we have to use it for all programs running on the computer — including the kernel itself. Therefore, a portion of the virtual address space must be reserved for the kernel.
')
This does not mean that the kernel uses so much physical memory - it just has a part of the address space at its disposal that can be assigned to the necessary amount of physical memory. The memory space for the kernel is marked in the page tables as exclusively used by the privileged code, so if a program tries to access it, a page fault happens. In Linux, the memory space for the kernel is constantly present, and sets in correspondence the same part of the physical memory of all processes. Kernel code and data always have addresses, and are ready to handle interrupts and system calls at any time. For user programs, on the contrary, the correspondence of virtual addresses of real memory changes when the switching process occurs:
Blue indicates virtual addresses corresponding to physical memory. White is the space to which addresses are not assigned. In our example, Firefox uses much more space in virtual memory because of its legendary gluttony. The strips in the address space correspond to memory segments such as heap, stack, and so on. These segments are just memory address intervals, and have nothing to do with segments from Intel. Here is the standard segment diagram for a process under Linux:
When programming was white and fluffy, the initial virtual segment addresses were the same for all processes. This made it easy to remotely exploit security vulnerabilities. Malicious programs often need to access memory at absolute addresses — the stack address, the library function address, and so on. Remote attacks had to be done blindly, counting on the fact that all address spaces remain on permanent addresses. In this regard, the random address selection system has gained popularity. Linux randomizes the stack, the memory map segment and the heap by adding offsets to their starting addresses. Unfortunately, in the 32-bit address space you can’t really expand, and there is not enough space to assign random addresses, which makes this system not very effective.
The topmost segment in the process's address space is the stack, which in most languages ​​stores local variables and function arguments. Calling a method or function adds a new stack frame (stack frame) to an existing stack. After returning from the function, the frame is deleted. This simple scheme leads to the fact that to track the contents of the stack does not require any complex structure - just enough pointer to the beginning of the stack. Adding and deleting data becomes a simple and unambiguous process. The constant reuse of memory areas for the stack causes these parts to be cached in the CPU, which adds speed. Each thread in the process gets its own stack.
You can come to a situation in which the memory allocated for the stack ends. This results in a page fault, which in Linux is handled by the expand_stack () function, which, in turn, calls acct_stack_growth () to check if the stack can still be expanded. If its size does not exceed RLIMIT_STACK (usually it is 8 MB), then the stack grows and the program continues execution, as if nothing had happened. But if the maximum stack size is reached, we get a stack overflow and the program receives a Segmentation Fault error. At the same time, the stack can only grow - like the state budget, it does not decrease back.
Dynamic stack growth is the only situation in which free memory can be accessed, which is shown in white in the diagram. All other attempts to access this memory cause a page fault error leading to a Segmentation Fault. And some occupied memory areas are read-only, so attempts to write to these areas also result in a Segmentation Fault.
After the stack comes the memory mapping segment. Here the kernel places the contents of the files directly in memory. Any application can request to do this through the mmap () system call on Linux or CreateFileMapping () / MapViewOfFile () on Windows. This is a convenient and fast way to organize input and output operations in files, so it is used to load dynamic libraries. It is also possible to create an anonymous memory location, not associated with the files that will be used for the program data. If you make a request to Linux for large amounts of memory through malloc (), the C library will create such an anonymous mapping instead of using heap memory. By “large” is meant a volume larger than MMAP_THRESHOLD (128 kB by default, it is configured via mallopt ().)
The heap itself is located at the following positions in memory. It provides memory allocation during program execution, as well as the stack - but, unlike it, it stores the data that must survive the function that allocates them. Most languages ​​have tools for managing the heap. In this case, the satisfaction of the request for the allocation of memory is performed jointly by the program and the kernel. In With the heap interface, malloc () is used with friends, and in a language that has automatic garbage collection, such as C #, the interface is the keyword new.
If there is not enough space on the heap to fulfill the request, the program itself can handle the problem without the intervention of the kernel. Otherwise, the heap is incremented by the brk () system call. Heap management is a tricky business, it requires ingenious algorithms that tend to work quickly and efficiently to cater to the chaotic method of data placement used by the program. The time to process a request for a heap can vary widely. Real-time systems have special tools for working with it. Heaps can also be fragmented:
And so we got to the very bottom of the scheme - BSS, data and program text. BSS and data store static (global) variables in C. The difference is that BSS stores the contents of non-initialized static variables whose values ​​were not set by the programmer. In addition, the BSS is anonymous, it does not correspond to any file. If you write
static int cntActiveUsers , then the contents of cntActiveUsers live in the BSS.
The data segment, on the contrary, contains those variables that were initialized in the code. This part of the memory corresponds to the binary image of the program containing the initial static values ​​specified in the code. If you write
static int cntWorkerBees = 10 , then the contents of cntWorkerBees live in the data segment, and begin their lives as 10. But, although the data segment corresponds to the program file, this is a private memory mapping - which means that the updates memory is not reflected in the corresponding file. Otherwise, changes to the value of variables would be reflected in a file stored on disk.
The sample data in the diagram will be a bit more complicated because it uses a pointer. In this case, the contents of the pointer, the 4-byte memory address, lives in the data segment. And the line on which it shows lives in a segment of text that is intended only for reading. It stores all the code and various other details, including string literals. It also keeps your binary in memory. Attempts to write to this segment end with an error Segmentation Fault. This prevents pointer-related errors (although not as effective as if you didn’t use C at all). The diagram shows these segments and examples of variables:
You can study the memory areas of a Linux process by reading the / proc / pid_of_process / maps file. Note that a single segment can contain many areas. For example, each file duplicated into memory has its own region in the mmap segment, and dynamic libraries have additional regions that resemble BSS and data. By the way, sometimes when people say "data segment", they mean data + bss + heap.
Binary images can be studied using the commands nm and objdump - you will see characters, their addresses, segments, etc. The virtual address scheme described in this article is so-called. The “flexible” scheme, which has been used by default for several years. It implies that a value has been assigned to the RLIMIT_STACK variable. Otherwise, Linux uses the “classic” scheme:
Source: https://habr.com/ru/post/265509/
All Articles