How to analyze Thread Dump
There are a lot of topics on the internals of the JVM in the Java Developer course program. We understand the mechanisms of collections, bytecode, garbage collection, etc. Today we offer your heed a translation of a rather interesting article on thread dump. What it is, how to get it and how to use it.
Want to learn how to analyze a thread dump? Come under the cat to learn more about how to get thread dump in Java and what to do with it then.
Most modern Java applications are multi-threaded. Multithreading can significantly expand the functionality of the application, at the same time it introduces significant complexity.
In a single-threaded application, all resources (shared memory, input / output operations, etc.) can be used without synchronization, since at any time, only one thread uses the resource.
')
In the case of multi-threaded applications, it is necessary to find a compromise between the complexity of the program and the possible performance improvement, when several threads can use all available (often more than one) cores of the central processing unit (CPU). If you do everything right, then using multithreading (formalized in Amdahl's Law ), you can achieve a significant increase in application performance. However, it should be remembered about ensuring synchronous access of several threads to a shared resource. In most cases, frameworks such as Spring encapsulate work with threads and hide many technical details from users. However, in the case of the use of modern complex frameworks, something can go wrong, and we, as users, will encounter difficult-to-solve multi-threading bugs.
Fortunately, Java is equipped with a special mechanism for obtaining information about the current state of all threads at any given time - this is a thread dump (a kind of snapshot). In this article, we will learn how to get a thread dump for an application of realistic dimensions and how to analyze this dump.
It is assumed that the reader has basic information about multi-threaded programming and is aware of the problems of thread synchronization and the use of shared resources. Nevertheless, it will not be superfluous to brush up on some basic terms and concepts.
At first glance, Java thread dumps may seem like a “Chinese letter”, the key to which I understand is the following concepts. In general, let's repeat the main terms of multithreading, which we will use for the analysis of dumps.
The terminology presented is not exhaustive for describing the world of multithreading, but this is enough to begin analyzing thread dumps.
More detailed information can be found in these sources:
Section 17 of the JLS and Java Concurrency in Practice
By applying these simple notions about flow in Java, we can create a test application. For this application we will collect thread dump. The resulting dump will analyze and extract useful information about the current flows of the application.
Before creating a thread dump, we need to develop a Java application. The traditional “hello, world!” Is too simple for our purpose, and a medium sized application dump may be too difficult to demonstrate. Based on this, we will create a fairly simple application in which two streams are created. And the threads get into the deadlock:
This program creates two resources: resourceA and resourceB, and starts two threads: threadLockingResourceAFirst and threadLockingResourceBFirst, which block each other’s resources.
The cause of the deadlock is a "cross" blocking of resources by threads.
The cause of the deadlock is an attempt to "mutual" capture of resources, i.e. thread threadLockingResourceAFirst captures resource resourceA, thread threadLockingResourceBFirst captures resource resourceB. After that, the thread threadLockingResourceAFirst, without releasing its resource, tries to grab resourceB, and the thread threadLockingResourceBFirst, without releasing its resource, tries to grab resource ResourceA. As a result, threads are blocked. A 1s delay is added to ensure blocking occurs. Streams are waiting for the release of the necessary resources, but this will never happen.
The output of the program will be as follows (the numbers after java.lang.Object @ will be different for each launch):
After the output of these messages, the program will look like working (the process that executes this program is not completed), while the program does not perform any work. This is how deadlock looks like in practice. To solve the problem, we need to manually create a tread dump and analyze the state of the threads.
In practice, a Java program can crash and create a thread dump. However, in some cases (for example, in the case of deadlocks), the program does not terminate and the thread dump does not create, it just hangs. To create a dump of such hung programs, first of all you need to find out the program process ID, i.e. Process ID (PID). To do this, you can use the JVM Process Status (JPS) utility, which, since version 7, is part of the Java Development Kit (JDK). To find the PID process of our hung program, we simply execute jps in the terminal (Windows or Linux):
The first column is the local virtual machine identifier (Local VM ID, i.e. lvmid) for the running Java process. In the context of the local JVM, lvmid points to the PID of the Java process.
It should be noted that this value is likely to differ from the value above. The second column is the name of the application, which may indicate the name of the main class, jar file or be equal to “Unknown” It all depends on how the application was launched.
In our case, the application name DeadlockProgram is the name of the main classes that was launched when the program started. In the example above, the PID of the program is 11568, this information is enough to generate a thread dump. To generate a dump, we will use the jstack utility, which is included in the JDK, starting with version 7. To get the dump, we will pass in the jstack as the PID parameter of our program and specify the -l flag (create a long listing). The utility output is redirected to a text file, i.e. thread_dump.txt:
The resulting thread_dump.txt file contains a thread dump of our hung program and contains important information for diagnosing the causes of the deadlock.
If you use JDK up to version 7, then you can use the Linux utility - kill with the -3 flag to generate a dump. Calling kill -3 will send a SIGQUIT signal to the program.
In our case, the call will be like this:
Opening the file thread_dump.txt, we will see something like the following content:
Although at first glance this file may seem too complicated and confusing, in reality it is quite simple, if you take it apart in parts, step by step.
The first line indicates the time when the dump was generated, the second - the diagnostic information about the JVM, on which the dump was received:
There is no thread information in this section. This sets the general context of the system in which the dump was collected.
The next section provides information about the threads that were running in the system at the time of the dump:
The following section lists:
Safe Memory Reclamation (SMR) information
It contains information about threads outside the JVM, i.e. these are not virtual machine threads or garbage collector threads. If you look at the addresses of these streams, you can see that they correspond to the value of the tid - “natural, iron” (native) address in the operating system, not the Thread ID.
Triplets are used to hide unnecessary information:
Immediately after the SMR block follows a list of threads. The first thread on our list is the Reference Handler:
The first line for each stream provides a general description. The description contains the following items:
The second line is the current state of the stream. Possible flow conditions are given in enum:
Thread.State :
NEW
RUNNABLE
BLOCKED
WAITING
TIMED_WAITING
TERMINATED
For more information, see the documentation .
The next section contains the stack trace trace at the time of the dump. This stack trace is very similar to the trace that is generated by an uncaught exception. It contains the names of the classes and lines that were executed at the time of the formation of the dump. In the case of the Reference Handler stream, we see nothing interesting.
However, there is something interesting in the trace of Thread-02 that differs from the standard trace:
In the trace, we see that information about the blocking has been added. This thread waits for a lock on the object with the address 0x00000000894465b0 (object type java.lang.Object). Moreover, the thread itself holds the lock with the address 0x00000000894465a0 (also a java.lang.Object object). This information is useful to us further for the diagnosis of deadlock.
The last section provides a list of synchronization primitives captured by the stream. These are objects that can be used to synchronize threads, for example, locks.
According to the official Java documentation, Ownable Synchronizer is the heirs of the AbstractOwnableSynchronizer (or its subclass), which can be exclusively captured by the stream for synchronization purposes.
ReentrantLock and write-lock , but not the read-lock class of the ReentrantReadWriteLock are two good examples of such “ownable synchronizers” offered by the platform.
For more information on this, you can refer to this.
post .
The next section of the dump contains information about JVM technical threads that are not part of the application and are associated with operating system threads. Since These threads work outside the application, they do not have thread identifiers. Most often these are garbage collector streams and other technical JVM streams:
This section indicates the number of global links used by the JVM via JNI. These links are not served by the garbage collector and in certain circumstances may cause a memory leak.
In most simple cases, this information is not used. However, the importance of global links must be understood. For more information, see this post .
The last section contains information about deadlocks found.
If these do not show up, then the section will be empty. Since We specifically developed an application with locks, in our case this section is there. A lock was detected during dump generation and is represented by the following message:
The first subsection describes the deadlock scenario:
Thread-0 expects the ability to capture the monitor (this is a call to the synchronized (secondResource) block in our application), at the same time, this thread is holding the monitor that is trying to capture the Thread-1 thread (this is the same code snippet: synchronized (secondResource ) in our application).
This circular lock is otherwise called deadlock . The picture below
This situation is presented graphically:
In the second section, the stack trace is given for both blocked threads.
This stack trace allows us to follow the operation of each thread until a lock appears.
In our case, if we look at the line:
at DeadlockProgram $ DeadlockRunnable.run (DeadlockProgram.java:34) , then we will see the problem area of ​​the code:
This line is the first line of the synchronized block, which is the cause of the lock, and tells us that the synchronization on the secondResource is the cause of the interlocking. To remedy the situation, we must ensure the same synchronization order on resourceA and resourceB in both flows. If we do this, we will come to the following application:
This application will end without interlocking, and as a result we will get the following output (note that the addresses of instances of the Object class have changed):
As a result, using only the information obtained from the thread dump, we were able to find the cause and fix the deadlock in our application. ( deadlock-). , .
.
JVM . ( , ).
.
- — Thread Dump Analyzers (TDAs). Java thread dump- - , . , . , .
TDA:
. .
Thread dump- — Java-, . , .
deadlock, . . , — .
, Java- thread dump-. , .
, thread dump — « » , , Java-.
Java . , deadlock- .
, .
Want to learn how to analyze a thread dump? Come under the cat to learn more about how to get thread dump in Java and what to do with it then.
Most modern Java applications are multi-threaded. Multithreading can significantly expand the functionality of the application, at the same time it introduces significant complexity.
In a single-threaded application, all resources (shared memory, input / output operations, etc.) can be used without synchronization, since at any time, only one thread uses the resource.
')
In the case of multi-threaded applications, it is necessary to find a compromise between the complexity of the program and the possible performance improvement, when several threads can use all available (often more than one) cores of the central processing unit (CPU). If you do everything right, then using multithreading (formalized in Amdahl's Law ), you can achieve a significant increase in application performance. However, it should be remembered about ensuring synchronous access of several threads to a shared resource. In most cases, frameworks such as Spring encapsulate work with threads and hide many technical details from users. However, in the case of the use of modern complex frameworks, something can go wrong, and we, as users, will encounter difficult-to-solve multi-threading bugs.
Fortunately, Java is equipped with a special mechanism for obtaining information about the current state of all threads at any given time - this is a thread dump (a kind of snapshot). In this article, we will learn how to get a thread dump for an application of realistic dimensions and how to analyze this dump.
It is assumed that the reader has basic information about multi-threaded programming and is aware of the problems of thread synchronization and the use of shared resources. Nevertheless, it will not be superfluous to brush up on some basic terms and concepts.
Basic terminology
At first glance, Java thread dumps may seem like a “Chinese letter”, the key to which I understand is the following concepts. In general, let's repeat the main terms of multithreading, which we will use for the analysis of dumps.
- Thread or thread is a discrete unit of multithreading driven by a Java Virtual Machine (JVM). JVM threads correspond to threads in the operating system (OS) - native threads (“natural flows”), which implement the mechanism for executing code.
Each thread has a unique identifier and name. Streams can be "demons" and "not demons."
The program ends its work when all non-daemons threads are terminated or the Runtime.exit method is called . Working "demons" do not affect the completion of the program. Those. JVMs are waiting for all the “non-demons” to be completed and the work is completed; the “non-demons” pays no attention.
For more information, see the documentation for the class Thread .
A thread can be in one of the following states:- Alive thread or “live” is a thread that does some work (normal state).
- Blocked thread or “blocked” - a stream that tried to enter the synchronization section (synchronized), however, another stream has already managed to enter this block first, and all the following streams that try to enter the same block turn out to be blocked.
- Waiting thread or “waiting” is the thread that called the wait method (possibly with a timeout indication) and is now waiting for another method to execute notify or nonifyAll on the same object.
Note that the thread is not considered "waiting" if it has called a wait with a timeout and this timeout has expired. - Sleeping thread or "sleeping" - a thread that is not currently running, because performed the Thread.sleep method (indicating the duration of the "sleep").
- Monitor is the mechanism used in the JVM to provide multi-threaded access to a single object. The mechanism is started using the special synchronized keyword .
Each object in Java has a monitor with which the stream can synchronize, i.e. set a lock, which guarantees that no other thread will gain access to this object until the lock is released, i.e. thread - lock owner will not exit synchronized block.
For more information, see the Synchronization section (17.1) of the Java Langauge Specification (JLS) . - Deadlock is a situation in which a thread, say A, blocks a resource, it needs another resource that is blocked by another thread, say B. Stream B does not release this resource, because to complete a certain operation, it needs a resource that is blocked by stream A. It turns out that stream A is waiting for the resource to be unlocked by thread B, which is waiting for the unlocking of another resource by stream A. And thus, the threads are waiting for each other. As a result, the whole program “hangs” and waits for the threads to somehow unlock and continue working. Mutual blocking can have many threads. This problem is well known as the “Problem of the Dining Philosophers” .
- Livelock is a situation where flow A causes flow B to perform some action, which in turn results in execution A of the original action, which once again causes the action of flow B. A cyclical relationship is obtained. This can be imagined as a dog running after its tail. Similar to Deadlock , in the Livelock situation, the program does not make progress, i.e. does not perform a useful action, however, in this situation, threads are not blocked.
The terminology presented is not exhaustive for describing the world of multithreading, but this is enough to begin analyzing thread dumps.
More detailed information can be found in these sources:
Section 17 of the JLS and Java Concurrency in Practice
By applying these simple notions about flow in Java, we can create a test application. For this application we will collect thread dump. The resulting dump will analyze and extract useful information about the current flows of the application.
Creating an example program
Before creating a thread dump, we need to develop a Java application. The traditional “hello, world!” Is too simple for our purpose, and a medium sized application dump may be too difficult to demonstrate. Based on this, we will create a fairly simple application in which two streams are created. And the threads get into the deadlock:
public class DeadlockProgram { public static void main(String[] args) throws Exception { Object resourceA = new Object(); Object resourceB = new Object(); Thread threadLockingResourceAFirst = new Thread(new DeadlockRunnable(resourceA, resourceB)); Thread threadLockingResourceBFirst = new Thread(new DeadlockRunnable(resourceB, resourceA)); threadLockingResourceAFirst.start(); Thread.sleep(500); threadLockingResourceBFirst.start(); } private static class DeadlockRunnable implements Runnable { private final Object firstResource; private final Object secondResource; public DeadlockRunnable(Object firstResource, Object secondResource) { this.firstResource = firstResource; this.secondResource = secondResource; } @Override public void run() { try { synchronized(firstResource) { printLockedResource(firstResource); Thread.sleep(1000); synchronized(secondResource) { printLockedResource(secondResource); } } } catch (InterruptedException e) { System.out.println("Exception occurred: " + e); } } private static void printLockedResource(Object resource) { System.out.println(Thread.currentThread().getName() + ": locked resource -> " + resource); } } } This program creates two resources: resourceA and resourceB, and starts two threads: threadLockingResourceAFirst and threadLockingResourceBFirst, which block each other’s resources.
The cause of the deadlock is a "cross" blocking of resources by threads.
The cause of the deadlock is an attempt to "mutual" capture of resources, i.e. thread threadLockingResourceAFirst captures resource resourceA, thread threadLockingResourceBFirst captures resource resourceB. After that, the thread threadLockingResourceAFirst, without releasing its resource, tries to grab resourceB, and the thread threadLockingResourceBFirst, without releasing its resource, tries to grab resource ResourceA. As a result, threads are blocked. A 1s delay is added to ensure blocking occurs. Streams are waiting for the release of the necessary resources, but this will never happen.
The output of the program will be as follows (the numbers after java.lang.Object @ will be different for each launch):
Thread-0: locked resource -> java.lang.Object@149bc794 Thread-1: locked resource -> java.lang.Object@17c10009 After the output of these messages, the program will look like working (the process that executes this program is not completed), while the program does not perform any work. This is how deadlock looks like in practice. To solve the problem, we need to manually create a tread dump and analyze the state of the threads.
Thread Dump Generation
In practice, a Java program can crash and create a thread dump. However, in some cases (for example, in the case of deadlocks), the program does not terminate and the thread dump does not create, it just hangs. To create a dump of such hung programs, first of all you need to find out the program process ID, i.e. Process ID (PID). To do this, you can use the JVM Process Status (JPS) utility, which, since version 7, is part of the Java Development Kit (JDK). To find the PID process of our hung program, we simply execute jps in the terminal (Windows or Linux):
$ jps 11568 DeadlockProgram 15584 Jps 15636 The first column is the local virtual machine identifier (Local VM ID, i.e. lvmid) for the running Java process. In the context of the local JVM, lvmid points to the PID of the Java process.
It should be noted that this value is likely to differ from the value above. The second column is the name of the application, which may indicate the name of the main class, jar file or be equal to “Unknown” It all depends on how the application was launched.
In our case, the application name DeadlockProgram is the name of the main classes that was launched when the program started. In the example above, the PID of the program is 11568, this information is enough to generate a thread dump. To generate a dump, we will use the jstack utility, which is included in the JDK, starting with version 7. To get the dump, we will pass in the jstack as the PID parameter of our program and specify the -l flag (create a long listing). The utility output is redirected to a text file, i.e. thread_dump.txt:
jstack -l 11568 > thread_dump.txt The resulting thread_dump.txt file contains a thread dump of our hung program and contains important information for diagnosing the causes of the deadlock.
If you use JDK up to version 7, then you can use the Linux utility - kill with the -3 flag to generate a dump. Calling kill -3 will send a SIGQUIT signal to the program.
In our case, the call will be like this:
kill -3 11568 Analysis of simple thread dump
Opening the file thread_dump.txt, we will see something like the following content:
2018-06-19 16:44:44
Full thread dump Java HotSpot (TM) 64-Bit Server VM (10.0.1 + 10 mixed mode):
Threads class SMR info:
_java_thread_list = 0x00000250e5488a00, length = 13, elements = {
0x00000250e4979000, 0x00000250e4982800, 0x00000250e52f2800, 0x00000250e4992800,
0x00000250e4995800, 0x00000250e49a5800, 0x00000250e49ae800, 0x00000250e5324000,
0x00000250e54cd800, 0x00000250e54cf000, 0x00000250e54d1800, 0x00000250e54d2000,
0x00000250e54d0800
}
"Reference Handler" # 2 daemon prio = 10 os_prio = 2 tid = 0x00000250e4979000 nid = 0x3c28 waiting on condition [0x000000b82a9ff000]
java.lang.Thread.State: RUNNABLE
at java.lang.ref.Reference.waitForReferencePendingList (java.base@10.0.1/Native Method)
at java.lang.ref.Reference.processPendingReferences (java.base@10.0.1/Reference.java: 174)
at java.lang.ref.Reference.access $ 000 (java.base@10.0.1/Reference.java: 44)
at java.lang.ref.Reference $ ReferenceHandler.run (java.base@10.0.1/Reference.java: 138)
Locked ownable synchronizers:
- None
"Finalizer" # 3 daemon prio = 8 os_prio = 1 tid = 0x00000250e4982800 nid = 0x2a54 in Object.wait () [0x000000b82aaff000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait (java.base@10.0.1/Native Method)
- waiting on <0x0000000089509410> (a java.lang.ref.ReferenceQueue $ Lock)
at java.lang.ref.ReferenceQueue.remove (java.base@10.0.1/ReferenceQueue.java: 151)
- waiting for re-lock in wait () <0x0000000089509410> (a java.lang.ref.ReferenceQueue $ Lock)
at java.lang.ref.ReferenceQueue.remove (java.base@10.0.1/ReferenceQueue.java: 172)
at java.lang.ref.Finalizer $ FinalizerThread.run (java.base@10.0.1/Finalizer.java: 216)
Locked ownable synchronizers:
- None
"Signal Dispatcher" # 4 daemon prio = 9 os_prio = 2 tid = 0x00000250e52f2800 nid = 0x2184 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Attach Listener" # 5 daemon prio = 5 os_prio = 2 tid = 0x00000250e4992800 nid = 0x1624 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread0" # 6 daemon prio = 9 os_prio = 2 tid = 0x00000250e4995800 nid = 0x4198 waiting on condition [0x000000000000000000]
java.lang.Thread.State: RUNNABLE
No compile task
Locked ownable synchronizers:
- None
"C2 CompilerThread1" # 7 daemon prio = 9 os_prio = 2 tid = 0x00000250e49a5800 nid = 0x3b98 waiting on condition [0x000000000000000000]
java.lang.Thread.State: RUNNABLE
No compile task
Locked ownable synchronizers:
- None
"C1 CompilerThread2" # 8 daemon prio = 9 os_prio = 2 tid = 0x00000250e49ae800 nid = 0x1a84 waiting on condition [0x000000000000000000]
java.lang.Thread.State: RUNNABLE
No compile task
Locked ownable synchronizers:
- None
"Sweeper thread" # 9 daemon prio = 9 os_prio = 2 tid = 0x00000250e5324000 nid = 0x5f0 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Service Thread" # 10 daemon prio = 9 os_prio = 0 tid = 0x00000250e54cd800 nid = 0x169c runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
Common cleaner # 11 daemon prio = 8 os_prio = 1 tid = 0x00000250e54cf000 nid = 0x1610 in Object.wait () [0x000000b82b2fe000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait (java.base@10.0.1/Native Method)
- waiting on <0x000000008943e600> (a java.lang.ref.ReferenceQueue $ Lock)
at java.lang.ref.ReferenceQueue.remove (java.base@10.0.1/ReferenceQueue.java: 151)
- waiting for re-lock in wait () <0x000000008943e600> (a java.lang.ref.ReferenceQueue $ Lock)
at jdk.internal.ref.CleanerImpl.run (java.base@10.0.1/CleanerImpl.java: 148)
at java.lang.Thread.run (java.base@10.0.1/Thread.java: 844)
at jdk.internal.misc.InnocuousThread.run (java.base@10.0.1/InnocuousThread.java: 134)
Locked ownable synchronizers:
- None
"Thread-0" # 12 prio = 5 os_prio = 0 tid = 0x00000250e54d1800 nid = 0xdec waiting for monitor entry [0x000000b82b4ff000]
java.lang.Thread.State: BLOCKED (on object monitor)
at DeadlockProgram $ DeadlockRunnable.run (DeadlockProgram.java:34)
- waiting to lock <0x00000000894465b0> (a java.lang.Object)
- locked <0x00000000894465a0> (a java.lang.Object)
at java.lang.Thread.run (java.base@10.0.1/Thread.java: 844)
Locked ownable synchronizers:
- None
"Thread-1" # 13 prio = 5 os_prio = 0 tid = 0x00000250e54d2000 nid = 0x415c waiting for monitor entry [0x000000b82b5ff000]
java.lang.Thread.State: BLOCKED (on object monitor)
at DeadlockProgram $ DeadlockRunnable.run (DeadlockProgram.java:34)
- waiting to lock <0x00000000894465a0> (a java.lang.Object)
- locked <0x00000000894465b0> (a java.lang.Object)
at java.lang.Thread.run (java.base@10.0.1/Thread.java: 844)
Locked ownable synchronizers:
- None
"DestroyJavaVM" # 14 prio = 5 os_prio = 0 tid = 0x00000250e54d0800 nid = 0x2b8c waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"VM Thread" os_prio = 2 tid = 0x00000250e496d800 nid = 0x1920 runnable
"GC Thread # 0" os_prio = 2 tid = 0x00000250c35b5800 nid = 0x310c runnable
"GC Thread # 1" os_prio = 2 tid = 0x00000250c35b8000 nid = 0x12b4 runnable
"GC Thread # 2" os_prio = 2 tid = 0x00000250c35ba800 nid = 0x43f8 runnable
"GC Thread # 3" os_prio = 2 tid = 0x00000250c35c0800 nid = 0x20c0 runnable
"G1 Main Marker" os_prio = 2 tid = 0x00000250c3633000 nid = 0x4068 runnable
"G1 Conc # 0" os_prio = 2 tid = 0x00000250c3636000 nid = 0x3e28 runnable
"G1 Refine # 0" os_prio = 2 tid = 0x00000250c367e000 nid = 0x3c0c runnable
"G1 Refine # 1" os_prio = 2 tid = 0x00000250e47fb800 nid = 0x3890 runnable
"G1 Refine # 2" os_prio = 2 tid = 0x00000250e47fc000 nid = 0x32a8 runnable
"G1 Refine # 3" os_prio = 2 tid = 0x00000250e47fd800 nid = 0x3d00 runnable
"G1 Young RemSet Sampling" os_prio = 2 tid = 0x00000250e4800800 nid = 0xef4 runnable
"VM Periodic Task Thread" os_prio = 2 tid = 0x00000250e54d6800 nid = 0x3468 waiting on condition
JNI global references: 2
Found one java-level deadlock:
=============================
"Thread-0":
monitor 0x00000250e4982480 (object 0x00000000894465b0, a java.lang.Object) waiting for lock
which is held by "Thread-1"
"Thread-1":
monitor 0x00000250e4982380 (object 0x00000000894465a0, a java.lang.Object),
which is held by "Thread-0"
Java stack information listed above:
================================================= =
"Thread-0":
at DeadlockProgram $ DeadlockRunnable.run (DeadlockProgram.java:34)
- waiting to lock <0x00000000894465b0> (a java.lang.Object)
- locked <0x00000000894465a0> (a java.lang.Object)
at java.lang.Thread.run (java.base@10.0.1/Thread.java: 844)
"Thread-1":
at DeadlockProgram $ DeadlockRunnable.run (DeadlockProgram.java:34)
- waiting to lock <0x00000000894465a0> (a java.lang.Object)
- locked <0x00000000894465b0> (a java.lang.Object)
at java.lang.Thread.run (java.base@10.0.1/Thread.java: 844)
Found 1 deadlock.
Introductory Information
Although at first glance this file may seem too complicated and confusing, in reality it is quite simple, if you take it apart in parts, step by step.
The first line indicates the time when the dump was generated, the second - the diagnostic information about the JVM, on which the dump was received:
2018-06-19 16:44:44 Full thread dump Java HotSpot(TM) 64-Bit Server VM (10.0.1+10 mixed mode): There is no thread information in this section. This sets the general context of the system in which the dump was collected.
General flow information
The next section provides information about the threads that were running in the system at the time of the dump:
Threads class SMR info:
_java_thread_list = 0x00000250e5488a00, length = 13, elements = {
0x00000250e4979000, 0x00000250e4982800, 0x00000250e52f2800, 0x00000250e4992800,
0x00000250e4995800, 0x00000250e49a5800, 0x00000250e49ae800, 0x00000250e5324000,
0x00000250e54cd800, 0x00000250e54cf000, 0x00000250e54d1800, 0x00000250e54d2000,
0x00000250e54d0800
}
The following section lists:
Safe Memory Reclamation (SMR) information
It contains information about threads outside the JVM, i.e. these are not virtual machine threads or garbage collector threads. If you look at the addresses of these streams, you can see that they correspond to the value of the tid - “natural, iron” (native) address in the operating system, not the Thread ID.
Triplets are used to hide unnecessary information:
"Reference Handler" # 2 ... tid = 0x00000250e4979000 ... "Finalizer" # 3 ... tid = 0x00000250e4982800 ... "Signal Dispatcher" # 4 ... tid = 0x00000250e52f2800 ... "Attach Listener" # 5 ... tid = 0x00000250e4992800 ... "C2 CompilerThread0" # 6 ... tid = 0x00000250e4995800 ... "C2 CompilerThread1" # 7 ... tid = 0x00000250e49a5800 ... "C1 CompilerThread2" # 8 ... tid = 0x00000250e49ae800 ... "Sweeper thread" # 9 ... tid = 0x00000250e5324000 ... "Service Thread" # 10 ... tid = 0x00000250e54cd800 ... Common Cleaner # 11 ... tid = 0x00000250e54cf000 ... "Thread-0" # 12 ... tid = 0x00000250e54d1800 ... "Thread-1" # 13 ... tid = 0x00000250e54d2000 ... "DestroyJavaVM" # 14 ... tid = 0x00000250e54d0800 ...
Streams
Immediately after the SMR block follows a list of threads. The first thread on our list is the Reference Handler:
"Reference Handler" # 2 daemon prio = 10 os_prio = 2 tid = 0x00000250e4979000 nid = 0x3c28 waiting on condition [0x000000b82a9ff000]
java.lang.Thread.State: RUNNABLE
at java.lang.ref.Reference.waitForReferencePendingList (java.base@10.0.1/Native Method)
at java.lang.ref.Reference.processPendingReferences (java.base@10.0.1/Reference.java: 174)
at java.lang.ref.Reference.access $ 000 (java.base@10.0.1/Reference.java: 44)
at java.lang.ref.Reference $ ReferenceHandler.run (java.base@10.0.1/Reference.java: 138)
Locked ownable synchronizers:
- None
Brief stream description
The first line for each stream provides a general description. The description contains the following items:
| Section | Example | Description |
|---|---|---|
| Name | "Reference Handler" | The human-readable name of the thread. The name can be set by calling the Thread object's setName method. And get through a call to getName |
| ID | # 2 | Unique ID assigned to each object of class Thread . An ID is generated for threads in the system. The initial value is 1. Each newly created thread is assigned its own ID, previously increased by 1. This property of the “read-only” stream can be obtained using the getId function of the object of the class Thread . |
| Daemon status | daemon | The flag is a sign that the thread is a daemon. If it is a demon, then the flag will be set. For example, Thread-0 is not a daemon. |
| Priority | prio = 10 | The numeric priority of the java stream. Note that this priority does not necessarily correspond to the priority of the associated thread in the operating system. To set the priority you can use the setPriority method of an object of class Thread , and to get method getPriority . |
| OS Thread Priority | os_prio = 2 | The priority of the thread in the operating system. This priority may differ from the one assigned to the coherent java stream. |
| Address | tid = 0x00000250e4979000 | Java stream address This address is a pointer to a Java Native Interface (JNI) native object of class Thread (a C ++ Thread object that is associated with a Java stream via JNI). This value is obtained by casting a pointer to this. (a C ++ object that is associated with this Java stream) to integer. Cm. line 879 in hotspot / share / runtime / thread.cpp : st-> print ("tid =" INTPTR_FORMAT "", p2i (this));
Although the key for this object ( tid ) may be similar to a stream ID, in fact, this is the address of the connected JNI C ++ Thread object, and this is not the value that returns the getId method of the Java Thread object. |
| OS Thread ID | nid = 0x3c28 | The unique identifier for the operating system thread to which the Java thread is bound. This value is output by the following code: line 42 in hotspot / share / runtime / osThread.cpp : st-> print ("nid = 0x% x", thread_id ());
|
| Status | waiting on condition | The human-readable status of the current thread. This line prints additional information to the simple status of the stream (see below), which can be used to understand what the thread was going to do (i.e., did the thread try to get the lock or waited for the unlock condition to be met). |
| Last Known Java Stack Pointer | [0x000000b82a9ff000] | The last known pointer to the stack (SP) associated with this thread. This value is obtained using native C ++ code mixed with Java code using JNI. The value returned by the function last_Java_sp () , line 2886 in hotspot / share / runtime / thread.cpp : st-> print_cr ("[" INTPTR_FORMAT "]",
(intptr_t) last_Java_sp () & ~ right_n_bits (12));
For simple thread dumps, this information is almost useless. However, in difficult cases, SP can be used to track locks. |
Flow condition
The second line is the current state of the stream. Possible flow conditions are given in enum:
Thread.State :
NEW
RUNNABLE
BLOCKED
WAITING
TIMED_WAITING
TERMINATED
For more information, see the documentation .
Thread stack trace
The next section contains the stack trace trace at the time of the dump. This stack trace is very similar to the trace that is generated by an uncaught exception. It contains the names of the classes and lines that were executed at the time of the formation of the dump. In the case of the Reference Handler stream, we see nothing interesting.
However, there is something interesting in the trace of Thread-02 that differs from the standard trace:
"Thread-0" # 12 prio = 5 os_prio = 0 tid = 0x00000250e54d1800 nid = 0xdec waiting for monitor entry [0x000000b82b4ff000]
java.lang.Thread.State: BLOCKED (on object monitor)
at DeadlockProgram $ DeadlockRunnable.run (DeadlockProgram.java:34)
- waiting to lock <0x00000000894465b0> (a java.lang.Object)
- locked <0x00000000894465a0> (a java.lang.Object)
at java.lang.Thread.run (java.base@10.0.1/Thread.java: 844)
Locked ownable synchronizers:
- None
In the trace, we see that information about the blocking has been added. This thread waits for a lock on the object with the address 0x00000000894465b0 (object type java.lang.Object). Moreover, the thread itself holds the lock with the address 0x00000000894465a0 (also a java.lang.Object object). This information is useful to us further for the diagnosis of deadlock.
Captured synchronization primitives (Ownable Synchronizer)
The last section provides a list of synchronization primitives captured by the stream. These are objects that can be used to synchronize threads, for example, locks.
According to the official Java documentation, Ownable Synchronizer is the heirs of the AbstractOwnableSynchronizer (or its subclass), which can be exclusively captured by the stream for synchronization purposes.
ReentrantLock and write-lock , but not the read-lock class of the ReentrantReadWriteLock are two good examples of such “ownable synchronizers” offered by the platform.
For more information on this, you can refer to this.
post .
JVM threads
The next section of the dump contains information about JVM technical threads that are not part of the application and are associated with operating system threads. Since These threads work outside the application, they do not have thread identifiers. Most often these are garbage collector streams and other technical JVM streams:
"VM Thread" os_prio = 2 tid = 0x00000250e496d800 nid = 0x1920 runnable "GC Thread # 0" os_prio = 2 tid = 0x00000250c35b5800 nid = 0x310c runnable "GC Thread # 1" os_prio = 2 tid = 0x00000250c35b8000 nid = 0x12b4 runnable "GC Thread # 2" os_prio = 2 tid = 0x00000250c35ba800 nid = 0x43f8 runnable "GC Thread # 3" os_prio = 2 tid = 0x00000250c35c0800 nid = 0x20c0 runnable "G1 Main Marker" os_prio = 2 tid = 0x00000250c3633000 nid = 0x4068 runnable "G1 Conc # 0" os_prio = 2 tid = 0x00000250c3636000 nid = 0x3e28 runnable "G1 Refine # 0" os_prio = 2 tid = 0x00000250c367e000 nid = 0x3c0c runnable "G1 Refine # 1" os_prio = 2 tid = 0x00000250e47fb800 nid = 0x3890 runnable "G1 Refine # 2" os_prio = 2 tid = 0x00000250e47fc000 nid = 0x32a8 runnable "G1 Refine # 3" os_prio = 2 tid = 0x00000250e47fd800 nid = 0x3d00 runnable "G1 Young RemSet Sampling" os_prio = 2 tid = 0x00000250e4800800 nid = 0xef4 runnable "VM Periodic Task Thread" os_prio = 2 tid = 0x00000250e54d6800 nid = 0x3468 waiting on condition
JNI global links
This section indicates the number of global links used by the JVM via JNI. These links are not served by the garbage collector and in certain circumstances may cause a memory leak.
JNI global references: 2
In most simple cases, this information is not used. However, the importance of global links must be understood. For more information, see this post .
Deadlocked streams
The last section contains information about deadlocks found.
If these do not show up, then the section will be empty. Since We specifically developed an application with locks, in our case this section is there. A lock was detected during dump generation and is represented by the following message:
Found one java-level deadlock:
=============================
"Thread-0":
monitor 0x00000250e4982480 (object 0x00000000894465b0, a java.lang.Object) waiting for lock
which is held by "Thread-1"
"Thread-1":
monitor 0x00000250e4982380 (object 0x00000000894465a0, a java.lang.Object),
which is held by "Thread-0"
Java stack information listed above:
================================================= =
"Thread-0":
at DeadlockProgram $ DeadlockRunnable.run (DeadlockProgram.java:34)
- waiting to lock <0x00000000894465b0> (a java.lang.Object)
- locked <0x00000000894465a0> (a java.lang.Object)
at java.lang.Thread.run (java.base@10.0.1/Thread.java: 844)
"Thread-1":
at DeadlockProgram $ DeadlockRunnable.run (DeadlockProgram.java:34)
- waiting to lock <0x00000000894465a0> (a java.lang.Object)
- locked <0x00000000894465b0> (a java.lang.Object)
at java.lang.Thread.run (java.base@10.0.1/Thread.java: 844)
Found 1 deadlock.
The first subsection describes the deadlock scenario:
Thread-0 expects the ability to capture the monitor (this is a call to the synchronized (secondResource) block in our application), at the same time, this thread is holding the monitor that is trying to capture the Thread-1 thread (this is the same code snippet: synchronized (secondResource ) in our application).
This circular lock is otherwise called deadlock . The picture below
This situation is presented graphically:
In the second section, the stack trace is given for both blocked threads.
This stack trace allows us to follow the operation of each thread until a lock appears.
In our case, if we look at the line:
at DeadlockProgram $ DeadlockRunnable.run (DeadlockProgram.java:34) , then we will see the problem area of ​​the code:
printLockedResource (secondResource);
This line is the first line of the synchronized block, which is the cause of the lock, and tells us that the synchronization on the secondResource is the cause of the interlocking. To remedy the situation, we must ensure the same synchronization order on resourceA and resourceB in both flows. If we do this, we will come to the following application:
public class DeadlockProgram { public static void main(String[] args) throws Exception { Object resourceA = new Object(); Object resourceB = new Object(); Thread threadLockingResourceAFirst = new Thread(new DeadlockRunnable(resourceA, resourceB)); Thread threadLockingResourceBFirst = new Thread(new DeadlockRunnable(resourceA, resourceB)); threadLockingResourceAFirst.start(); Thread.sleep(500); threadLockingResourceBFirst.start(); } private static class DeadlockRunnable implements Runnable { private final Object firstResource; private final Object secondResource; public DeadlockRunnable(Object firstResource, Object secondResource) { this.firstResource = firstResource; this.secondResource = secondResource; } @Override public void run() { try { synchronized (firstResource) { printLockedResource(firstResource); Thread.sleep(1000); synchronized (secondResource) { printLockedResource(secondResource); } } } catch (InterruptedException e) { System.out.println("Exception occurred: " + e); } } private static void printLockedResource(Object resource) { System.out.println(Thread.currentThread().getName() + ": locked resource -> " + resource); } } } This application will end without interlocking, and as a result we will get the following output (note that the addresses of instances of the Object class have changed):
Thread-0: locked resource -> java.lang.Object@1ad895d1 Thread-0: locked resource -> java.lang.Object@6e41d7dd Thread-1: locked resource -> java.lang.Object@1ad895d1 Thread-1: locked resource -> java.lang.Object@6e41d7dd
As a result, using only the information obtained from the thread dump, we were able to find the cause and fix the deadlock in our application. ( deadlock-). , .
Thread Dump-
.
JVM . ( , ).
.
- — Thread Dump Analyzers (TDAs). Java thread dump- - , . , . , .
TDA:
. .
Conclusion
Thread dump- — Java-, . , .
deadlock, . . , — .
, Java- thread dump-. , .
, thread dump — « » , , Java-.
Java . , deadlock- .
, .
Source: https://habr.com/ru/post/427513/
All Articles