ReactOS News Release # 89

RPC

The Remote Procedure Call includes many different components of Windows, especially services. Thus, the remote procedure call (rpcrt4) dynamic library is quite important for ensuring the continuous operation and stability of ReactOS. In addition, it is a component in which the project is extremely dependent on Wine. Unfortunately, the fact that the Wine code works in Linux does not mean that it will work in ReactOS, and all previous attempts to synchronize rpcrt4 led to many problems and failures in ReactOS. Thomas Faber, a student who participated in Google Summer of Code with a project to implement a new system for testing kernel components, discovered, he believes, the main cause of the critical RPC failures in ReactOS. When a process makes a call to a remote procedure, it receives a handle to that call. This descriptor is a unique identifier for each RPC operation, and its uniqueness should have been guaranteed by a random number generator. Wine for these purposes uses a Linux random number generator located in / dev / urandom, which of course is not in ReactOS. In addition to everything else, it turned out that the ReactOS code lacks the NT function, which Wine uses to generate random numbers, so when querying for unique identifiers for different RPC operations, the same values ​​could be issued. This confused the operating system and broke down the functionality of everything that depended on RPC. Thomas is addressing this issue, and we hope that in the future, attempts to synchronize rpcrt4 with Wine code will not lead to disastrous consequences for the operating system.

Memory manager

Many testers expressed their dissatisfaction with the hangup of the ReactOS installation process during an attempt to copy the mshtml.dll file. Attempts to diagnose the problem led developers deep into the depths of the memory manager and revealed a fairly wide range of problems. The reason for the hang was to reset the data structures, which led to the fact that the heap manager lost its ability to track memory blocks while copying the mshtml.dll file. Without this information, the heap dispatcher could not get the next block of memory and hung in an infinite loop. Already the third (and maybe even more) development team tried their hand at eliminating this problem, and all of them came to the same conclusion: the rewriting of the memory manager, started in ARM3, should be completed. Thomas was one of the team members to fix this problem, and first decided that the error was caused by the setupapi component, which installs the drivers, libraries, and related files. This was due to the rather cluttered state and complexity of the setupapi code, so Thomas suggested that the error occurs due to memory overflow and similar problems. To test this theory, Thomas created his own heap dispatcher and ran setupapi on top of it. As a result, setupapi was rehabilitated as failures began to appear that were not related to setupapi. This meant that the problem was much deeper in the memory manager. Even more disappointing was the fact that the faults seemed to depend on timing, which in itself suggested a likely conflict of resources.
')
Cameron Gutman (Cameron Gutman) took part in the Crusade for MSHTML, but he had to be distracted by the patch provided by Nikolai Myltsev and related to the swap job. In general, the amount of virtual memory requested by the application and the amount of this memory actually located in physical memory are two different things. In Windows, a working set is the virtual memory pages of a process that has recently been working on, and all processes have a maximum and minimum working set size. Windows will try to keep the working set of pages in physical memory, assuming that there is enough free space for this. The purpose of the working set is to prevent all physical memory from being consumed by a single application, and the memory manager balances the memory consumption of each process by removing pages from the working sets of different processes, if necessary. Pages whose contents have been modified in RAM must be written back to disk before the block of memory they use can be reused. When it comes time to remove pages from the working set, the operating system will look for a sufficient number of clean, unmodified pages that it could reuse, thereby obtaining the necessary amount of free memory. The system should also try to record changes in occupied pages if it has insufficient free pages to satisfy the request for memory allocation, however, this was not what happened in ReactOS. This error ultimately led to the fact that the balancer of the working set ended with pages that he could have withdrawn from the working set, as a result of which the operating system crashed. After reviewing the patch created by Nicholas and making several of his changes in it, Cameron added the results of his work to the code base of the project, and now the balancer correctly records the changes in the occupied pages, which allows them to be used again.

Processor microcodes

This topic requires a bit of insight into the design and manufacture of CPUs. Most programmers understand that each of the processor families is subject to its Instruction Set Architecture architecture, which determines what instructions and operations the processor can perform. However, many people do not know or do not think that most processors do not actually follow the instructions defined by the ISA directly. In fact, there is a lower level of commands, these are the so-called microcodes that control the operation of the processor so that the result conforms to the requirements of the ISA. These microcodes are for the most part completely undocumented, since they are a commercial secret with both AMD and Intel. The architecture of modern CPUs is extremely complex, and there may be hardware errors in it, leading to incorrect microcode processing. Solving such a problem in hardware would require a change in the architecture, but in modern CPUs it is possible to correct these errors by updating the microcode. AMD and Intel provide these patches to developers of operating systems in the form of binary files, but the process in which Windows applies these patches is undocumented. Pierre Schweitzer took up this issue and had some thoughts on how ReactOS could update the microcode, but at the moment the problem has a rather low priority. In any case, this is a rather interesting problem, distinguished by a number of difficulties that can be encountered at the lowest levels of hardware / software interfaces.

Installation

Long ago, an adopted architectural solution that allows installing ReactOS on systems with a small amount of RAM can significantly increase the time required to copy ReactOS files. The compressed cab file containing the system files is extracted during the first stage of the installation. The initial developer of the code for extracting files from the cab-archive did not suggest the possibility that the cab-file may have a larger volume than the amount of available free physical memory. As a result, every time the unpacking program needs to extract a new file from the cab archive, it must start searching the archive from the very beginning and alternately iterate through all the blocks until it finds one it needs. This is far from the best solution, since ReactOS is forced to upload and download the same data from memory pages to it, over and over again. Cameron noticed this behavior and modified the program for extracting cab-archives so that she could memorize her current position in the cab-file. Thus, the extraction program does not need to start reading the archive from the very beginning each time it is necessary to continue extracting files, which provides the memory manager with much faster access to the files.

S / G DMA

In traditional direct memory access (DMA) operations, a single continuous block of memory is required to which data will be transferred. Unfortunately, the system does not always have a sufficiently large memory block. Direct access to the memory by the “Scatter / Gather” mechanism is intended to correct this problem by using smaller pieces, saving data on their location and writing data to them. The main part of the S / G DMA support code was already in ReactOS, however, an API that represented its functionality was missing. Cameron undertook and developed this API, which allowed the drivers using the S / G DMA mechanism to work in ReactOS, for the most part these are network card drivers.

Memory Optimization in FreeLoader

Recently Timo Kreuzer (Timo Kreuzer) noticed that freeldr uses memory rather inefficiently. In fact, there are two types of data for which memory is required by freeldr: data that is transferred to it to the kernel, and data that is only needed by freeldr itself to complete the loading of the operating system. Initially, freeldr allocated a single block of 4 MB in size and used it for both types of data, while the kernel had to read all 4 MB of data. Considering this somewhat wasteful, Timo broke a single piece into two different heaps, one for each data type. This allows freeldr to free up all non-core memory, which results in a significant reduction in the amount of used memory. It seems to be a small change, but small changes like this can change a lot in the long run.

Path fixes

Many regressions caused by the new loader were associated with the not quite correct behavior of some functions dealing with paths. Since the loader during its work performs many operations of access to files necessary for loading executable files and libraries, it actively uses path processing functions. Alex Ionescu recently spent time rewriting a large part of the basic functions associated with differences in string encoding and their transformation, parsing abbreviations and correcting error codes during critical failures. All this taken together resulted in the elimination of a significant number of regressions associated with the loader, and, we hope, will allow even more applications to function correctly.

PS Participated in the translation: evilslon , serrox , tipys .

PPS The project accepts donations now in Bitcoins www.reactos.org/ru/foundation_donate.html