24-core CPU, and I can not dial an email
I was not looking for trouble. I didn’t run Chrome’s build process a thousand times over the weekend , but I’ve only done the most common tasks of the 21st century — I just wrote an email at 10:30 in the morning. And suddenly Gmail hung. I kept typing for a few seconds, but no characters appeared on the screen. Then, suddenly, Gmail opened up - and I went back to my very important letter. But later everything repeated, only this time Gmail did not respond to requests even longer. This is weird ...
It is difficult to resist the opportunity to conduct a good investigation, but in this case the challenge is especially strong. After all, I'm working on Google to improve the performance of Chrome for Windows. To discover the cause of the hang is my job . And after a lot of false starts and hard efforts, I still managed to figure out how Chrome, Gmail, Windows and our IT department together prevented me from typing an email. Along the way, I found a way to save a significant amount of memory for some web pages in Chrome.
The investigation turned out to have so many nuances that I will leave some for another article , and now I will fully explain the reasons for the suspensions.
As usual, UIforETW works with me in the background and monitors cyclic buffers, so all that remained was to enter Ctrl + Win + R - and the buffers of the system were saved to disk in about the last thirty seconds. I downloaded them to the Windows Performance Analyzer (WPA), but could not finally install the hang.
')
When the Windows program stops transmitting messages, ETW events are issued indicating where exactly this happened , so these types of hangs are trivial to find. But, apparently, Chrome continued to transmit messages. I looked for the moment when one of the key streams of Chrome either entered the active cycle, or was completely idle, but found nothing obvious. There were some places where Chrome was mostly idle, but even then all key streams continued to work, so it was impossible to be sure where the hang occurred - Chrome could have been idle just in the absence of events:
UIforETW has a built-in keylogger, which is often useful when identifying key points in a trace. However, for obvious security reasons, by default it anonymizes keystrokes, regarding each entered number as “1” and every letter as “A”. This makes it difficult to find the exact time of the hang, so I changed the type of tracing from “private” to “full” and waited for the hang. At about 10:30 the next morning the hangup repeated. I saved the trace buffers and saved this mark in the UIforETW trace information field:
This is a common discussion of how to get to the office , but it is important that there is now a way to find a hang in the ETW trace. I load the trace, look at the keylogger data in the Generic Events field (events are issued by UIforETW, and each is a purple diamond in the screenshot below) - and I immediately see where the hang occurred, which is clearly correlated by the failure in using the CPU:
OK, but why did Chrome stop? Here are some hints: the screenshots do not show that every time WmiPrvSE.exe fully uses the hyper- thread CPU. But that shouldn't matter. In my machine, there are 24 cores / 48 threads, so the consumption of one hyperflow means that the system is still 98% free.
Then I approached a clearly important period when Chrome was idle - and looked, in particular, at the CrRendererMain process in chrome.exe (27368) corresponding to the Gmail tab.
The problem has cleared up. Within 2.81 seconds of hangup, this thread was launched according to the schedule 440 times. Usually, running every 6 ms is enough to make the program responsive, but for some reason this did not happen. I noticed that every time he woke up, I was on the same stack. If you simplify:
Chrome VirtualAlloc, “CfgBits” . , Chrome VirtualAlloc 440 , . . Chrome VirtualAlloc — . Chrome , — 439 — Chrome , . , .
Windows — , , . . .
, Chrome , WmiPrvSE., :
WMI ( ), WMI. CPU, , WmiPrvSE.exe ( ):
. NtQueryVirtualMemory, GetProcessVaData, Va, , . VirtualScan NtQueryVirtualMemory , Gmail (10-15 ) — . ?
, . NtQueryVirtualMemory . « », « » .. Gmail 26 000 , ( , WPA) 16 000 , .
- Gmail vmmap , Gmail (361 836 ) (49 719), — 2 147 483 648 , 2 . ?
, 2 Control Flow Guard (CFG), , “CFG” , Gmail Chrome — MiCommitVadCfgBits. , CFG !
Control Flow Guard (CFG) . , 128- . , CFG ( 2 ), . CFG , CFG . 98 24 866 CFG-. :
, CFG ? CFG , ? .
— VAllocStress, . 64- CFG, , , , . , / , , . VAllocStress:
. . , . , VirtualScan VAllocStress. , CFG , . VAllocStress !
CFG, . !
, JavaScript- v8 CodeRange , CodeRange 128 . , , CFG, .
, CodeRange, , ? CodeRange, Gmail — . ( ) CodeRange. , WorkerThread::Start . :
CFG Windows 10 RS4 ( 2018 ), , . , .
CFG — . CFG , . . , CFG ! , . CFG . , 2 !
, — Gmail 353,3 CFG 59,2 , 400 . - , .
v8 ( JavaScript Chrome) , CodeRange, . Microsoft CFG. , - Microsoft CFG , , , . vmmap .
. :
, , , . , , , , .
, 10:30 , IT- . , Control Panel → Configuration Manager → Actions, Hardware Inventory Cycle Run Now.
, VAllocStress VirtualScan Github.
. ( ), (), ( WMI). vmmap. — — crbug.com/870054.
UPD.
It is difficult to resist the opportunity to conduct a good investigation, but in this case the challenge is especially strong. After all, I'm working on Google to improve the performance of Chrome for Windows. To discover the cause of the hang is my job . And after a lot of false starts and hard efforts, I still managed to figure out how Chrome, Gmail, Windows and our IT department together prevented me from typing an email. Along the way, I found a way to save a significant amount of memory for some web pages in Chrome.
The investigation turned out to have so many nuances that I will leave some for another article , and now I will fully explain the reasons for the suspensions.
As usual, UIforETW works with me in the background and monitors cyclic buffers, so all that remained was to enter Ctrl + Win + R - and the buffers of the system were saved to disk in about the last thirty seconds. I downloaded them to the Windows Performance Analyzer (WPA), but could not finally install the hang.
')
When the Windows program stops transmitting messages, ETW events are issued indicating where exactly this happened , so these types of hangs are trivial to find. But, apparently, Chrome continued to transmit messages. I looked for the moment when one of the key streams of Chrome either entered the active cycle, or was completely idle, but found nothing obvious. There were some places where Chrome was mostly idle, but even then all key streams continued to work, so it was impossible to be sure where the hang occurred - Chrome could have been idle just in the absence of events:
UIforETW has a built-in keylogger, which is often useful when identifying key points in a trace. However, for obvious security reasons, by default it anonymizes keystrokes, regarding each entered number as “1” and every letter as “A”. This makes it difficult to find the exact time of the hang, so I changed the type of tracing from “private” to “full” and waited for the hang. At about 10:30 the next morning the hangup repeated. I saved the trace buffers and saved this mark in the UIforETW trace information field:He typed “put aside for those who have more experience with scuba diving” - and Gmail hung at the end of the word “those”, and then resumed work in the area of the word “experience”. Gmail tab with PID 27368.
This is a common discussion of how to get to the office , but it is important that there is now a way to find a hang in the ETW trace. I load the trace, look at the keylogger data in the Generic Events field (events are issued by UIforETW, and each is a purple diamond in the screenshot below) - and I immediately see where the hang occurred, which is clearly correlated by the failure in using the CPU:
OK, but why did Chrome stop? Here are some hints: the screenshots do not show that every time WmiPrvSE.exe fully uses the hyper- thread CPU. But that shouldn't matter. In my machine, there are 24 cores / 48 threads, so the consumption of one hyperflow means that the system is still 98% free.
Then I approached a clearly important period when Chrome was idle - and looked, in particular, at the CrRendererMain process in chrome.exe (27368) corresponding to the Gmail tab.
Note: I want to say thank you to myself from 2015 for requesting Microsoft to improve the naming mechanisms for threads, and thanks to Microsoft for implementing all of the suggestions - the names of the threads in WPA are just great!
The problem has cleared up. Within 2.81 seconds of hangup, this thread was launched according to the schedule 440 times. Usually, running every 6 ms is enough to make the program responsive, but for some reason this did not happen. I noticed that every time he woke up, I was on the same stack. If you simplify:
chrome_child.dll (stack base)
KernelBase.dll!VirtualAlloc
ntoskrnl.exe!MiCommitVadCfgBits
ntoskrnl.exe!MiPopulateCfgBitMap
ntoskrnl.exe!ExAcquirePushLockExclusiveEx
ntoskrnl.exe!KeWaitForSingleObject (stack leaf)Chrome VirtualAlloc, “CfgBits” . , Chrome VirtualAlloc 440 , . . Chrome VirtualAlloc — . Chrome , — 439 — Chrome , . , .
Windows — , , . . .
, Chrome , WmiPrvSE., :
ntoskrnl.exe!KiSystemServiceCopyEnd (stack base)
ntoskrnl.exe!NtQueryVirtualMemory
ntoskrnl.exe!MmQueryVirtualMemory
ntoskrnl.exe!MiUnlockAndDereferenceVad
ntoskrnl.exe!ExfTryToWakePushLock (stack leaf)WMI ( ), WMI. CPU, , WmiPrvSE.exe ( ):
WmiPerfClass.dll!EnumSelectCounterObjects (stack base)
WmiPerfClass.dll!ConvertCounterPath
pdh.dll!PdhiTranslateCounter
pdh.dll!GetSystemPerfData
KernelBase.dll!blah-blah-blah
advapi32.dll!blah-blah-blah
perfproc.dll!blah-blah-blah
perfproc.dll!GetProcessVaData
ntdll.dll!NtQueryVirtualMemory
ntoskrnl.exe!NtQueryVirtualMemory
ntoskrnl.exe!MmQueryVirtualMemory
ntoskrnl.exe!MiQueryAddressSpan
ntoskrnl.exe!MiQueryAddressState
ntoskrnl.exe!MiGetNextPageTable (stack leaf). NtQueryVirtualMemory, GetProcessVaData, Va, , . VirtualScan NtQueryVirtualMemory , Gmail (10-15 ) — . ?
, . NtQueryVirtualMemory . « », « » .. Gmail 26 000 , ( , WPA) 16 000 , .
- Gmail vmmap , Gmail (361 836 ) (49 719), — 2 147 483 648 , 2 . ?
, 2 Control Flow Guard (CFG), , “CFG” , Gmail Chrome — MiCommitVadCfgBits. , CFG !
Control Flow Guard (CFG) . , 128- . , CFG ( 2 ), . CFG , CFG . 98 24 866 CFG-. :
Scan time, Committed, page tables, committed blocks
Total: 41.763s, 1457.7 MiB, 67.7 MiB, 32112, 98 code blocks
CFG: 41.759s, 353.3 MiB, 59.2 MiB, 24866vmmap , — vmmap 49 684 , 24 866
, CFG ? CFG , ? .
— VAllocStress, . 64- CFG, , , , . , / , , . VAllocStress:
- :
- VirtualAlloc .
- .
- :
- 500 ( ).
- VirtualAlloc .
- , VirtualAlloc ~500
- .
. . , . , VirtualScan VAllocStress. , CFG , . VAllocStress !
CFG, . !
, JavaScript- v8 CodeRange , CodeRange 128 . , , CFG, .
, CodeRange, , ? CodeRange, Gmail — . ( ) CodeRange. , WorkerThread::Start . :
- Gmail -, , .
- , -.
- CodeRange, JITted- JavaScript 47- .
- CFG 2 .
- CFG .
- NtQueryVirtualMemory CFG ( 1 ) , .
CFG Windows 10 RS4 ( 2018 ), , . , .
CFG — . CFG , . . , CFG ! , . CFG . , 2 !
, — Gmail 353,3 CFG 59,2 , 400 . - , .
v8 ( JavaScript Chrome) , CodeRange, . Microsoft CFG. , - Microsoft CFG , , , . vmmap .
. :
- Gmail.
- Windows 10.
- IT- WMI- .
- .
- .
, , , . , , , , .
, 10:30 , IT- . , Control Panel → Configuration Manager → Actions, Hardware Inventory Cycle Run Now.
, VAllocStress VirtualScan Github.
. ( ), (), ( WMI). vmmap. — — crbug.com/870054.
UPD.
Source: https://habr.com/ru/post/420579/
All Articles