Do I need undocumented APIs?
What is the beauty of high-level programming languages?
The programmer stops thinking about “how it actually functions” and gives out a ton of unaccompanied code to the mountain, which sometimes even works, but sometimes slows down. Moreover, quite often, he is not even going to think about it, since he simply does not have the necessary knowledge (alas, as they are teaching now).
Then from all of this there are such clever words as “design patterns”, internal frameworks that you can’t see without tears, and other chiaroscuro, including technical documentation that you need to write so that some third-party peyzanin can understand it, which is essentially programming (yes and on a post), has the most remote relation.
The words are clever, for a high-level code it may even fit, but ...
And then, having executed the next clever “pattern”, he begins to figure out where the algorithm is slowing down. Moreover, if the programmer is more stubborn, he studies the implementation of the VCL and sometimes even gets to the bottom of it, where it turns out that the brakes rest on the challenges known to him from the API documentation, passing to which he calmly stops and closes the ticket in the bugtracker with the phrase: “function XXX slows down, there are no options to bypass. "
')
Have not met with the situation?
So lucky ...
I will show one of the variants of such a closed ticket, which at one time greatly influenced the speed of the target application. The problem, of course, was different, so I’m pulling this code a little bit behind my ears, but in principle it also reflects the essence of the problem.
Suppose we want to write an application that needs an up-to-date list of the current heaps of the specified process (no matter why, let’s say antivirus :)).
For simplicity, we will get a list of heaps from the current process, for this we will create an empty console application and write the following code there:
The code is simple to impossible - the standard TlHelp32 functions without frills are used.
But the nuance, try to execute it and look at the time of its work.
About 150 milliseconds. The jokes about the fact that programmers are increasing the power of computers are trying to level by non-optimized code, of course ridiculous, but ...
We tried to take data only from our process, but how many will there be on the user's machine? Yes, though, would be fifty dollars, and what will we come to?
By an almost five-second delay in requesting only heaps of the process (I'm not talking about just listing thread processes, etc.).
Well, really, and what can we do?
We are not to blame - this is how the Heap32xxx function works, and in fact these brakes are due to a delay in their call (this is true). And here we are sitting with our service on the server, we are gorging on the CPU resources and shrug: “I lost all the stuff - sheff”.
There’s no - we are guilty of dwelling on the barriers we created.
A programmer always has a debugger at hand, thanks to which you can analyze exactly what is happening, at least by expanding the call stack using CPU-View.
Given that I am writing articles for people interested in software protection, I will not tell you how to use it (you yourself should know this), so let's set the breakpoint on the Heap32First function call and simply route the program through F7.
After a few dozen tracing steps, we will see:
Let's take a little more and go to:
After that, we will perform the same steps, but only with the Heap32Next function.
What do you think we will find there?
Alas - the same RtlCreateQueryDebugBuffer + RtlQueryProcessDebugInformation and at the end of RtlDestroyQueryDebugBuffer.
But this is already a hitch, what is happening? It turns out that for each of our own when calling API functions from the tlhelp32.dll library, all three API functions are actually called , the first of which collects information about the process, the second makes a selection of just one entry from it, and then everything closes. Is it too expensive?
And now let's remember - how much time did it take us to list the heaps of the process? 150 milliseconds? Well, yes, why not, when for every Heap32Next call all the information about the process was collected again.
We will fight - we write the code (I’m just warning you, the knee-length code is confusing a little with the output of addresses, but for the demo it will go).
And what do we have here?
However, acceleration is more than four times, and in fact they have done nothing but a trifle.
Undocumented?
That and the clown with him, do not hesitate to delve into the subtleties :)
Just do not forget about the distance between "when you can" and "when you need" :)
What are the disadvantages when using undocumented calls?
Well, first of all, no one can guarantee that the parameters and structures for calling these functions will not change with the next patch, and in general no one guarantees that it will remain on the library's export lists. True, to be honest, I have never met with such behavior, except for one single function - AllocateAndGetTcpExTableFromStack. But with her, everything turned out somehow unclear, initially it appeared in Windows XP. Until Vista came out, it was considered undocumented, but it was documented with the release of Vista, but was immediately excluded from the Iphlpapi.dll library by writing "This function is not longer available."
The second minus: it is not always possible to find such a more productive analogue, as a substitute for the documented function, and in fact in most cases it is not needed. Optimization for the sake of optimization is a rather bad approach to software development ...
The source code can be viewed here: rouse.drkb.ru/blog/heap.zip
Alexander (Rouse_) Bagel
December 2012
The programmer stops thinking about “how it actually functions” and gives out a ton of unaccompanied code to the mountain, which sometimes even works, but sometimes slows down. Moreover, quite often, he is not even going to think about it, since he simply does not have the necessary knowledge (alas, as they are teaching now).
Then from all of this there are such clever words as “design patterns”, internal frameworks that you can’t see without tears, and other chiaroscuro, including technical documentation that you need to write so that some third-party peyzanin can understand it, which is essentially programming (yes and on a post), has the most remote relation.
The words are clever, for a high-level code it may even fit, but ...
And then, having executed the next clever “pattern”, he begins to figure out where the algorithm is slowing down. Moreover, if the programmer is more stubborn, he studies the implementation of the VCL and sometimes even gets to the bottom of it, where it turns out that the brakes rest on the challenges known to him from the API documentation, passing to which he calmly stops and closes the ticket in the bugtracker with the phrase: “function XXX slows down, there are no options to bypass. "
')
Have not met with the situation?
So lucky ...
I will show one of the variants of such a closed ticket, which at one time greatly influenced the speed of the target application. The problem, of course, was different, so I’m pulling this code a little bit behind my ears, but in principle it also reflects the essence of the problem.
Suppose we want to write an application that needs an up-to-date list of the current heaps of the specified process (no matter why, let’s say antivirus :)).
For simplicity, we will get a list of heaps from the current process, for this we will create an empty console application and write the following code there:
procedure QueryProcessHeap1; var hSnapShot: THandle; HeapList: THeapList32; HeapEntry: THeapEntry32; Start: DWORD; begin Start := GetTickCount; Writeln('Heap info:'); Writeln; hSnapShot := CreateToolhelp32Snapshot(TH32CS_SNAPHEAPLIST, 0); try HeapList.dwSize := SizeOf(THeapList32); if Heap32ListFirst(hSnapShot, HeapList) then repeat HeapEntry.dwSize := SizeOf(THeapEntry32); if Heap32First(HeapEntry, GetCurrentProcessId, HeapList.th32HeapID) then repeat Writeln(Format('Heap addr: 0x%p, size: %d', [Pointer(HeapEntry.dwAddress), HeapEntry.dwBlockSize])); until not Heap32Next(HeapEntry); until not Heap32ListNext(hSnapShot, HeapList); finally CloseHandle(hSnapShot); end; Writeln; Writeln('DONE. Time elapsed: ', GetTickCount - Start); end; The code is simple to impossible - the standard TlHelp32 functions without frills are used.
But the nuance, try to execute it and look at the time of its work.
About 150 milliseconds. The jokes about the fact that programmers are increasing the power of computers are trying to level by non-optimized code, of course ridiculous, but ...
We tried to take data only from our process, but how many will there be on the user's machine? Yes, though, would be fifty dollars, and what will we come to?
By an almost five-second delay in requesting only heaps of the process (I'm not talking about just listing thread processes, etc.).
Well, really, and what can we do?
We are not to blame - this is how the Heap32xxx function works, and in fact these brakes are due to a delay in their call (this is true). And here we are sitting with our service on the server, we are gorging on the CPU resources and shrug: “I lost all the stuff - sheff”.
There’s no - we are guilty of dwelling on the barriers we created.
A programmer always has a debugger at hand, thanks to which you can analyze exactly what is happening, at least by expanding the call stack using CPU-View.
Given that I am writing articles for people interested in software protection, I will not tell you how to use it (you yourself should know this), so let's set the breakpoint on the Heap32First function call and simply route the program through F7.
After a few dozen tracing steps, we will see:
Let's take a little more and go to:
After that, we will perform the same steps, but only with the Heap32Next function.
What do you think we will find there?
Alas - the same RtlCreateQueryDebugBuffer + RtlQueryProcessDebugInformation and at the end of RtlDestroyQueryDebugBuffer.
But this is already a hitch, what is happening? It turns out that for each of our own when calling API functions from the tlhelp32.dll library, all three API functions are actually called , the first of which collects information about the process, the second makes a selection of just one entry from it, and then everything closes. Is it too expensive?
And now let's remember - how much time did it take us to list the heaps of the process? 150 milliseconds? Well, yes, why not, when for every Heap32Next call all the information about the process was collected again.
We will fight - we write the code (I’m just warning you, the knee-length code is confusing a little with the output of addresses, but for the demo it will go).
procedure QueryProcessHeap2; var I, A: Integer; pDbgBuffer: PRtlDebugInformation; pHeapInformation: PRtlHeapInformation; pHeapEntry: PRtrHeapEntry; dwHeapStartAddr, dwAddr, dwLastSize: DWORD; hit_seg_count: Integer; Start: DWORD; begin Start := GetTickCount; Writeln('Heap info:'); Writeln; pDbgBuffer := RtlCreateQueryDebugBuffer(0, False); if pDbgBuffer <> nil then try // if RtlQueryProcessDebugInformation(GetCurrentProcessId, RTL_QUERY_PROCESS_HEAP_SUMMARY or RTL_QUERY_PROCESS_HEAP_ENTRIES, pDbgBuffer) = 0 then begin // pHeapInformation := @pDbgBuffer^.Heaps^.Heaps[0]; // ... for I := 0 to pDbgBuffer^.Heaps^.NumberOfHeaps - 1 do begin // pHeapEntry := pHeapInformation^.Entries; dwAddr := DWORD(pHeapEntry^.u.s2.FirstBlock) + pHeapInformation^.EntryOverhead; dwLastSize := 0; for A := 0 to pHeapInformation^.NumberOfEntries - 1 do begin dwHeapStartAddr := dwAddr; hit_seg_count := 0; while (pHeapEntry^.Flags and RTL_HEAP_SEGMENT) = RTL_HEAP_SEGMENT do begin // RTL_HEAP_SEGMENT, // dwAddr := DWORD(pHeapEntry^.u.s2.FirstBlock) + pHeapInformation^.EntryOverhead; Inc(pHeapEntry); Inc(hit_seg_count); // if A + hit_seg_count >= Integer(pHeapInformation^.NumberOfEntries - 1) then Continue; end; // , , // + if hit_seg_count = 0 then Inc(dwAddr, dwLastSize); Writeln(Format('Heap addr: 0x%p, size: %d', [Pointer(dwHeapStartAddr), dwAddr - dwHeapStartAddr])); // dwLastSize := pHeapEntry^.Size; // Inc(pHeapEntry); end; // Inc(pHeapInformation); end; end; finally RtlDestroyQueryDebugBuffer(pDbgBuffer); end; Writeln; Writeln('DONE. Time elapsed: ', GetTickCount - Start); end; And what do we have here?
However, acceleration is more than four times, and in fact they have done nothing but a trifle.
Undocumented?
That and the clown with him, do not hesitate to delve into the subtleties :)
Just do not forget about the distance between "when you can" and "when you need" :)
What are the disadvantages when using undocumented calls?
Well, first of all, no one can guarantee that the parameters and structures for calling these functions will not change with the next patch, and in general no one guarantees that it will remain on the library's export lists. True, to be honest, I have never met with such behavior, except for one single function - AllocateAndGetTcpExTableFromStack. But with her, everything turned out somehow unclear, initially it appeared in Windows XP. Until Vista came out, it was considered undocumented, but it was documented with the release of Vista, but was immediately excluded from the Iphlpapi.dll library by writing "This function is not longer available."
The second minus: it is not always possible to find such a more productive analogue, as a substitute for the documented function, and in fact in most cases it is not needed. Optimization for the sake of optimization is a rather bad approach to software development ...
The source code can be viewed here: rouse.drkb.ru/blog/heap.zip
Alexander (Rouse_) Bagel
December 2012
Source: https://habr.com/ru/post/181588/
All Articles