PermGen memory leak analysis in Java
What is it about?
Who was engaged in web development on Java, for certain faced such problem as
java.lang.OutOfMemoryError: PermGen space . It usually occurs after restarting the web application inside the server without restarting the server itself. Restarting the web application without restarting the server may be necessary during the development process, in order not to wait for the server to start too much time. If you have multiple web applications delayed, restarting the entire server can be much longer than restarting a single web application. Or the entire server simply cannot be restarted, since other web applications are used. The first solution that comes to mind is to increase the maximum amount of PermGen memory available to the JVM (you can do this with the -XX:MaxPermSize ), but this will only delay the fall, after a few restarts you will again get OutOfMemoryError . It would be nice to be able to restart and redistribute the web application on a running server as many times as you like. How to overcome PermGen, and will further discussion.What is PermGen?
PermGen - Permanent Generation is a JVM memory area for storing descriptions of Java classes and some additional data. Thus, when restarting a web application, all classes are loaded with a new one and fill PermGen memory. A web application can contain a bunch of libraries, and class descriptions can occupy dozens of megabytes. Those who follow Java innovations may have heard that PermGen was abandoned in Java 8. Here you would think that the eternal problem was finally fixed, and there will be no more falls due to the lack of PermGen memory. Unfortunately, this is not true, roughly speaking, PermGen is now simply called Metaspace, and you still get an
OutOfMemoryError .Stop. But what about the garbage collector?
We all know that Java has a garbage collector that collects all unused objects. It should also collect unused classes in PermGen, but only if it is configured correctly and there are no memory leaks.
')
Regarding the configuration, there is quite a bit of official documentation, there are many tips on the Internet to use various options, for example
-XX:+CMSClassUnloadingEnabled , -XX:+CMSPermGenSweepingEnabled , -XX:+UseConcMarkSweepGC . I didn’t dig deep and look for official documentation, but through trial and error I determined that for Java 7 and Tomcat 7 it is necessary and sufficient to add the JVM option -XX:+UseConcMarkSweepGC . This option will change the garbage collection algorithm, if you are not sure that your application will not work worse because of this, then look for documentation and comparisons of different garbage collection algorithms to determine whether this option is worth using or not. You may need to enable this option to get rid of problems with PermGen. If not, then you most likely have a memory leak, what to do with it - read on.Why does PermGen memory leak occur?
First, a few words about the class loader. Class loaders are objects in Java that are responsible for loading classes. In web servers, there is a hierarchy of class loaders, for each web application there is one class loader, plus several general class loaders. Classes within the web application are loaded with a class loader, which corresponds to this web application. System classes and the classes needed by the server itself are loaded by common class loader. For example, how the class loader hierarchy for Tomcat is arranged can be read here .
In order for the garbage collector to collect all classes of a web application, they should not be referenced outside this web application. Now remember that every object in Java stores a reference to its class, i.e. to an object of the
java.lang.Class class, and each class stores a reference to the class loader that loaded this class, and each class loader stores references to all the classes it has loaded. It turns out that only one link from the outside to the web application object pulls along all the classes of the web application and the inability to collect them by the garbage collector.
Another reason for the leak may be a thread that was launched from a web application and which could not be stopped when the web application was stopped. It also stores a link to the class loader of the web application.
Another popular leak option is the
ThreadLocal variable, which assigns an object from a web application to a stream from a common pool. In this case, the thread stores an object reference. The stream from the common pool cannot be destroyed, the object cannot be destroyed, it means that the whole class loader with all classes cannot be destroyed.Standard Tomcat Tools
Fortunately, there are a number of tools in Tomcat that analyze and prevent PermGen memory leaks.
First , in the standard Tomcat Manager Application, there is a “Find leaks” button ( details ) that will analyze which web applications left trash after a restart.
But this will only show which web applications may contain a leak, there is little sense from this.
Secondly , in Tomcat there is a
JreMemoryLeakPreventionListener - a solution for well-known possible variants of memory leaks, configured in server.xml ( details ). Perhaps turning on any options on this listener will help get rid of memory leaks.And thirdly, the most important thing is that when a web application is stopped, Tomcat writes to the log what exactly could have caused a memory leak. For example:
SEVERE: The web application [/drp] appears to have started a thread named [AWT-Windows] but has failed to stop it. This is very likely to create a memory leak.SEVERE: The web application [/drp] created a ThreadLocal with key of type [org.apache.log4j.helpers.ThreadLocalMap] (value [org.apache.log4j.helpers.ThreadLocalMap@7dc1e95f]) and a value of type [java.util.Hashtable] (value [{session=*2CBFB7}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.This is exactly what we need in order to continue leak analysis.
And since we really got down to business, you need to know how to properly check if PermGen is cleaned or not. Tomcat Manager Application, which can show memory usage, including PermGen, will help us with this again.
Another feature is that clearing occurs only after reaching the maximum amount of PermGen memory, so you need to set a small value of the maximum available PermGen memory (for example,
-XX:MaxPermSize=100M ) so that after two or three restarts of the web application, the occupied memory reaches 100 %, and either a cleanup occurred, or an OutOfMemoryError fell if there were still leaks.Now let's look at how to get rid of leaks using examples.
Take the following message:
SEVERE: The web application [/drp] appears to have started a thread named [AWT-Windows] but has failed to stop it. This is very likely to create a memory leak.It tells us that the web application started and did not stop the
AWT-Windows thread, therefore, its contextClassLoader turned out to be a class loader of the web application, and the garbage collector cannot collect it. Here we can track using breakpoint with the condition on the name of the thread, who created this stream, and, having rummaged in the source code, find what opportunities it can stop, for example, set a flag or call some method, for example Thread#interrupt() . These steps will need to be performed when the web application is stopped.But you can also notice that the name of the thread looks like something systemic ... Maybe
JreMemoryLeakPreventionListener , about which we learned above, can do something with this thread? We go to the documentation and see that the listener actually has the AWTThreadProtection parameter, which for some reason is false by default. We put it in true in server.xml and make sure that Tomcat does not issue such a message.In this case, the
AWT-Windows thread was created due to captcha generation on the server using the image manipulation classes from the JDK.Ok, here we got off with a simple option in Tomcat, let's try something more complicated:
SEVERE: The web application [/drp] created a ThreadLocal with key of type [org.apache.log4j.helpers.ThreadLocalMap] (value [org.apache.log4j.helpers.ThreadLocalMap@7dc1e95f]) and a value of type [java.util.Hashtable] (value [{session=*2CBFB7}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.Here we see that someone put in
ThreadLocal variable of the class ThreadLocalMap some value and did not remove it. We are looking for where the ThreadLocalMap class is used, we find org.apache.log4j.MDC , and this class is already directly used in our web application for logging additional information. We see that the put method of the MDC class is called, and the remove method is not called. It seems that calling remove for each put in the right place should help. We fix, check - it works!After correcting all such errors, it is likely that you will get rid of
OutOfMemoryError: PermGen space , at least in my practice this was so.Analysis with VisualVM
If you are not using Tomcat, or if the correction of errors indicated by Tomcat in the log did not help, then you can continue the analysis using the profiler. I took the free VisualVM profiler included in the JDK.
First, let's start the server with one secured web application and restart it so that the leak is visible. Open VisualVM, select the desired process and make a heap dump by selecting the corresponding item in the drop-down menu.
Select the “OQL Console” tab and execute the following query:
select x from org.apache.catalina.loader.WebappClassLoader x(for other servlet implementations, the class will be different).
One of two copies remained from the first stopped web application, the garbage collector could not collect it. To determine which of them is old, click on one of them and look for the
started field. The old started will be false .
In the “References” window, all links to this class loader are shown, we need to find the one that the garbage collector cannot collect. To do this, right-click on
this and select "Show Nearest GC Root".
Great, we found some thread whose old class loader is a
contextClassloader . Click on it with the right mouse button and select “Show Instance”.
We look at the fields of the object and think what we can catch on to understand what kind of object it is, somehow find the code that creates it, catch it in the debugger, etc. In this case, the name of the thread is
AWT-Windows . We found the same problem that Tomcat wrote to us about using VisualVM only. How to solve it you already know.Total
We learned to identify, analyze and correct PermGen memory leaks. It turned out to be not so difficult, especially thanks to the built-in Tomcat tools. I can not guarantee that the above methods can get rid of all types of leaks, but I managed to get rid of leaks in several large projects.
Links
Source: https://habr.com/ru/post/222443/
All Articles