Windows programming difficulties

When you are involved in the development of fairly complex projects, then writing the incorrectly working code is easier than ever. The whole snag is that you start to look for a mistake in the darkest back streets of the project, in its most difficult parts. At the same time, even thoughts do not come to mind that the simplest code, the basis of the whole project, the framework, may not work.
In this post, I will describe two problems that I encountered in practice: the inability to create another stream and rename the file. Used programming language: C / C ++.

Fight, thread, do not be afraid

So, the first situation.

Given : on one of the laptops in the office (not developers) while the program is running, at one point, the CreateThread () function stops working with the return code “Failed to create thread. Not enough storage is available to process this command. ”

It is worth noting that when this error occurred, a substantial part of the functionality did not work on the target laptop. While all the errors were cleared and the result was remotely checked using logs (it was not possible to install the IDE on the laptop and debug the program), quite a long time passed. And so, there was only the problem of creating a stream.
')
As soon as it became clear that the program did not have enough memory for this task (judging by the return code), the witch hunt began: suspicion began to fall on all the complex libraries, in the code of which the devil himself would break his leg. This was facilitated by the fact that when the program used the GPU, the problem manifested itself. If only the CPU was used, there was no problem.
Naturally, we began to carefully look at the library, providing work with the GPU. Along the way, several important errors were found and fixed that caused a memory leak. However, this did not solve the original problem. Flows all the same at one point ceased to be created.

Another snag was that the memory didn't end there! Task Manager showed that the program at the critical moment used about 220 MB, while the laptop was equipped with 8 GB of RAM.

Stress Testing

It turned out that the problem manifested itself when the program created more than 90 threads. They began to sin on a pool of threads: they thought that it was starting to create threads indefinitely. It was decided to remove the limit on the number of threads created by the program at a time, and select, say, 1000 threads. On the debugging machine, all a thousand threads were created with a bang, but on the target laptop there were even more problems: the GUI began to fall apart.

But all hopes that the source of the problem was found were broken by the TestLimit program from Sysinternals: on the target laptop, this test program created 1000 threads or more without problems. Even the hope that in our program each thread is allocated more memory than the default, melted away like a light haze. The error was in our program.

RTFM

We have already begun to think that this problem is associated exclusively with the target notebook and is manifested only on a specific hardware. However, there is also a misfire: if you create a new user, then under it our program works without fail.

When the hands had already begun to fall, one of the employees once again (for the umpteenth time) read the help for CreateThread () in MSDN. And the casket just opened:
"If you have created a CRT, you can terminate the process in low-memory conditions ." ( Proof )

Solution: use _beginthreadex () to create threads, instead of CreateThread () . The same was written in the comments to the help page with the theme "_beginthread vs CreateThread: which should you use?"

Immediately I remembered that on the target laptop really heavyweight applications were spinning, which corresponds to the low-memory condition. The thread pool was rewritten to use _beginthreadex () and eventually ... the problem persists . But already manifested less frequently. And all because in some parts of the program threads were created manually, and were not taken from the pool.
When all calls to CreateThread () were replaced with calls to _beginthreadex () (including in third-party libraries), the problem was finally solved.

The main difficulty in solving this problem was that the problem was reproduced only on a specific machine with a certain use. At the same time, all our code tests were useless, since in this case we also needed emulation of the load on the OS.

Are you sure you want to rename the file?

Situation two.

Given : at an indefinite moment of the program, after the file <filename> has been deleted, the function for checking the existence of the file <filename> returns true .

First we look at the function of checking the existence of a file:

bool exist( const wchar_t* filename ) { bool result = !_waccess( filename, 0 ); if( result ) { return true; } WIN32_FIND_DATAW attrs; HANDLE handle = FindFirstFileW( filename, &attrs ); if( handle == INVALID_HANDLE_VALUE ) { return false; } FindClose( handle ); return !!attrs.dwFileAttributes; } 

Remembering the problem with CRT that arose in the previous example, I begin to sin on the _waccess () function. However, after writing a small test for creating / deleting a file, it becomes clear that it has nothing to do with it.

It turned out that after the file was deleted using the _wremove () function, the FindFirstFileW () function can still find attributes for this file! It turns out that the NTFS file system driver requires some non-zero time to remove the file attributes. Therefore, one cannot rely on the existence of file attributes in the function of checking for existence.
OK, no problems. Just remove this attribute availability check.

The exist () function now works correctly. Problem solved? Not here it was!

I can not create a file, a crib!

Initially, the problem was in the following algorithm:

• delete the file <dest_file>
• rename file <source_file> to <dest_file>

namely, in the renaming function: it had the function exist () described above before starting the renaming. After correction, the algorithm began to work correctly. The test for this piece of code looked like this:

 void test() { const wchar_t firstName[] = "alice.txt"; const wchar_t secondName[] = "bob.txt"; int mode = _S_IREAD | _S_IWRITE; //    int handle = _wcreat( firstName, mode ); ::close( handle ); //    handle = _wcreat( secondName, mode ); ::close( handle ); while( true ) { //    _wremove( secondName ); // ,    assert( !exist( secondName ) ); //      _wrename( firstName, secondName ); //    _wrename( secondName, firstName ); //    handle = _wcreat( secondName, mode ); ::close( handle ); } } 

However, the original test looked like this:

 void test() { const wchar_t fileName[] = "test.txt"; int mode = _S_IREAD | _S_IWRITE; while( true ) { //   int handle = _wcreat( fileName, mode ); ::close( handle ); // ,    assert( exist( fileName ) ); //   _wremove( fileName ); // ,    assert( !exist( fileName )); } } 

So, at first, the function exist () did not work correctly: over time, checking for the absence of a file produced an incorrect result. We fixed this by removing the check for the attributes of the requested file. And then the _wcreat () function started to fail ! After about the 800th iteration of the loop (or even earlier), it ended with an error ...

It is known that _wcreat () is a wrapper over CreateFile () . So at the time of the error, the last function terminated with the return code "ACCESS_DENIED" ! That is, despite the fact that we told the system to delete the file and even checked that the file is no longer there, in fact, the file system driver still deleted the file. Hence the denial of access.

Solution: we did not find it. Because in a real project there was no similar situation with the creation / deletion of the same file, then they decided to score a mistake. It is noteworthy that in such a situation, MoveFile () works correctly, i.e. can rename a third-party file to the remote!
Another solution to this situation: turn off antivirus and / or other programs that may request access to the file being created.

Epilogue

I like programming and solving various algorithmic problems. But how, however, it is difficult to solve the problem in the program when the OS is buggy! Well, not OS, but libraries and drivers for it. And in such moments, when you come across such errors, only one picture comes to mind:

Links

If anyone is interested in reading about other interesting cases, here is a good blog (thanks for the link thanks unkinddragon ):
blogs.msdn.com/b/oldnewthing

Literature:
Richter J. "Windows. Creating effective Win32-based applications, taking into account the specifics of the 64-bit version of Windows
Mark Russinovich, David Solomon, Alex Ionescu "Windows Internals (5th Edition)"

UPD 1: if anyone is interested, here is my test:

According to the results of testing, I can say that on my laptop the second situation is reproduced stably. From antiviruses: Windows Security Essentials.

UPD 2: Situation 2 can be avoided by disabling the antivirus and / or other programs that may request access to the file being created (thanks to the community for the tip to this solution).