Compiling and decompiling try-with-resources, or a story about how I fixed a bug and what came of it.

Introduction

Some time ago, the backlog of the working draft was almost empty, and various kinds of research tasks surfaced. One of them sounded quite intriguing: to tie in to the project mutation testing using PITest . On Habré there is already a very detailed overview of this library (with examples and pictures). I will not retell this article in my own words, but I still recommend that you first read it.

I admit that the idea of ​​mutational testing, I fired up. With almost no extra effort, get a search tool for potentially dangerous code points - it's worth it! I immediately got down to business. At that time, the library was relatively young, and as a result, it was very raw: here you need a little mess with the maven configuration, there you can patch the plug-in for Sonar. However, after a while, I was still able to check the whole project. Result: hundreds of surviving mutations! Evolution to scale on our build server.
')
Rolling up my sleeves, I plunged into work. In some tests, there is not enough verification of stubs, in others, instead of logic, it is generally unclear what is being tested. Rule, improve, rewrite. In general, the process has begun, but the number of surviving mutations did not decrease as rapidly as desired. The reason was simple: PIT gave a huge amount of false positives on the try-with-resources block. Short searches have shown that the bug is known , but still not fixed. Well, the library code is open. Why not incline it and see what is the matter?

We understand the reasons

I threw the simplest example , a unit test to it and launched PITest. The result is in front of you: instead of one, there are eleven surviving mutations, ten of which point to a line with the “}” symbol. Calls to the close and addSupressed methods suggest that the code generated for the try-with-resources block belongs to this line. To confirm this guess, I decided to decompile the class file. For this, I used the JD-GUI , although now I would recommend the built-in decompiler IntelliJ IDEA 14 .

public static void main(String[] args) throws IOException { ByteArrayOutputStream baos = new ByteArrayOutputStream(); Throwable var2 = null; try { baos.flush(); } catch (Throwable var11) { var2 = var11; throw var11; } finally { if (baos != null) { if (var2 != null) { try { baos.close(); } catch (Throwable var10) { var2.addSuppressed(var10); } } else { baos.close(); } } } } 

The guess was confirmed, but the question remained: how did the two lines of try-with-resources turn into a dozen lines of try-catch-finally? gvsmirnov bequeathed us in any incomprehensible situation to download the source code OpenJDK . This is what I did.

All code related to the try-with-resources compilation task is located between lines 1428 and 1580 of the Lower class. Javadoc tells us that this class is designed to translate syntactic sugar: no magic, only the simplest modifications of the syntax tree. All in accordance with JLS 14.20.3 .

With the behavior of the compiler figured out. It remains to understand why the library is trying to mutate the code generated by the compiler and how it works. Having rummaged in source codes, I clarified the following. PITest only manipulates bytecode loaded into RAM. It replaces instructions for specific rules, after which it runs unit tests. ASM is used to work with bytecode.

The first idea was to intercept the line number from the visitGeneratedTryCatchBlock method's MethodVisitor class, and then just tell the library which line to ignore. Similar functionality has already been implemented for the finally block. However, I was surprised to learn that the visitGeneratedTryCatchBlock method does not exist. ASM does not distinguish the code generated by the compiler from the generated by the programmer. Ambush. I had to look into bytecode, the output and formatting of which was kindly provided by the Textifier .


The naive assumption that the try-catch-finally block is implemented at the JVM level has not been confirmed . There is no special instruction for it, only the exception table and goto between the labels. It turns out, standard means to recognize the generated block will not work. We need to look for another solution.

What if…

Before I started guessing at the coffee grounds, I decided to put the bytecode tags on the decompiled class. That's what came out of it.

 public static void main(String[] args) throws IOException { ByteArrayOutputStream baos = new ByteArrayOutputStream(); // L11 Throwable primaryExc = null; // L12 try { baos.flush(); // L3 } catch (Throwable t) { // L5 primaryExc = t; throw t; } finally { // L6 if (baos != null) { // L4 L10 if (primaryExc != null) { try { baos.close(); // L0 L7 } catch (Throwable suppressedExc) { // L2 L9 primaryExc.addSuppressed(suppressedExc); // L15 L19 } // L1 L16 L8 L20 } else { baos.close(); // L14 L18 } } // L17 } // L13 } 

There are clearly two main ways of executing the program:

 L11 L12 L3 {L4 [L0 (L2 L15 L16) L1] L14} L13
 L11 L12 L3 [L5 {L6] L10 [L7 (L9 L19 L20) L8] L18 L17}

There are labels under each other, whose code blocks match or almost match. In parentheses is the code that will be executed when the close method throws an exception. Similarly, in square - when the method flush . Two ways came from the fact that the finally block was substituted by the compiler twice. And now, to finally break down your visual parser: the labels in curly brackets refer to line 11. The PITest false positives refer to the same line.

Here is the solution! It is necessary to select a minimally repeating set of instructions. If such a set is found in the checked bytecode, and even on one line - there is a generated code for the try-with-resources block. It does not sound very ironic, but I decided to try. Below is a list of instructions that I ended up with.

 private static final List<Integer> JAVAC_CLASS_INS_SEQUENCE = Arrays.asList( ASTORE, // store throwable ALOAD, IFNULL, // closeable != null ALOAD, IFNULL, // localThrowable2 != null ALOAD, INVOKEVIRTUAL, GOTO, // closeable.close() ASTORE, // Throwable x2 ALOAD, ALOAD, INVOKEVIRTUAL, GOTO, // localThrowable2.addSuppressed(x2) ALOAD, INVOKEVIRTUAL, // closeable.close() ALOAD, ATHROW); // throw throwable 

Approximately so it can be compared to the code in the finally block.

 } finally { if (closeable != null) { // IFNULL if (localThrowable2 != null) { // IFNULL try { closeable.close(); // INVOKEVIRTUAL or INVOKEINTERFACE } catch (Throwable x2) { localThrowable2.addSuppressed(x2); // INVOKEVIRTUAL } } else { closeable.close(); // INVOKEVIRTUAL or INVOKEINTERFACE } } } // ATHROW 

“Not so difficult,” I thought after several days of hard work. I threw a few more examples ; wrote tests that use them. Everything is fine, everything works. I tried to build PITest to run it on live code: tests crashed. Not the ones I wrote; others.

Compilers are different

So, the code has moved from the “not compiled” stage to the “not working” stage. One of the tests that existed before has fallen. Rolled back - works. Inside the test, the Java7TryWithResources.class.bin file is checked , which was already in the project. After printing bytecode, I could not believe my eyes: a completely different order of instructions was used to compile try-with-resources!

Trying not to panic, I began to check all the compilers at hand. I worked with javac from Oracle JDK, javac from OpenJDK was expected to give a similar result. I tried different versions: to no avail. It was the turn of the compilers, which were not at hand. Eclipse Compiler for Java, ECJ. Compiled, printed baytkod - at first glance it looks like the one I'm looking for.


After that I decided to decompile the resulting class file. The result of the work of the decompiler refused to compile. Well, nothing, you can already work with it. Having brought the program code into accord with bytecode, I got the following.

 public static void main(String[] paramArrayOfString) throws Throwable { Throwable primaryExceptionVariable = null; // L5 Throwable caughtThrowableVariable = null; try { ByteArrayOutputStream baos = new ByteArrayOutputStream(); // L3 try { baos.flush(); // L0 } catch (Throwable t) { primaryExceptionVariable = t; // L2 throw primaryExceptionVariable; // L7 } finally { if (baos != null) { // L1 baos.close(); } } } catch (Throwable t) { caughtThrowableVariable = t; // L4 if (primaryExceptionVariable == null) { primaryExceptionVariable = caughtThrowableVariable; } else if (primaryExceptionVariable != caughtThrowableVariable) { // L8 primaryExceptionVariable.addSuppressed(caughtThrowableVariable); } throw primaryExceptionVariable; // L9 } // L6 } 

ECJ uses a completely different approach to compiling try-with-resources. Labels are noticeably smaller, code blocks are noticeably larger. Instead of a bloated table, exceptions are simply thrown up a level. In the examples it is more difficult to notice that it turns out such a doll.

What is under the hood? I went to download the source again, this time ECJ . The compilation of the try statement is hidden in the TryStatement file. This time no trees, only opcodes, only hardcore. The bytecode responsible for try-with-resources is generated between lines 500 and 604. It is clear from the history of commits that the body of the try block was simply framed with a chain of calls to create and close resources.

Since there is no finally-block substitution, then there is no duplication of code. However, due to nesting, the same actions are repeated for different exceptions. This I took advantage of. The ECJ instruction set is as follows.

 private static final List<Integer> ECJ_INS_SEQUENCE = Arrays.asList( ASTORE, // store throwable2 ALOAD, IFNONNULL, // if (throwable1 == null) ALOAD, ASTORE, GOTO, // throwable1 = throwable2; ALOAD, ALOAD, IF_ACMPEQ, // if (throwable1 != throwable2) { ALOAD, ALOAD, INVOKEVIRTUAL, // throwable1.addSuppressed(throwable2) ALOAD, ATHROW); // throw throwable1 

And this is the corresponding java-code.

 if (throwable1 == null) { // IFNONNULL throwable1 = throwable2; } else { if (throwable1 != throwable2) { // IF_ACMPEQ throwable1.addSuppressed(throwable2); // INVOKEVIRTUAL } } // ATHROW 

What about the rest of the compilers? It turned out that AspectJ generates almost the same bytecode as ECJ. For him there was no need to invent a separate sequence. I could not download a compiler from IBM (and I didn’t really want to). The rest of the compilers were ignored due to the low prevalence.

results

The attentive reader has already noticed that the instruction set for javac does not take into account one nuance. Different methods are actually used to call the class and interface methods: INVOKEVIRTUAL and INVOKEINTERFACE, respectively. The implementation described above takes into account only the first case and does not take into account the second. Well, nothing, it is not difficult to fix.

So, what happened in the end?

Firstly, the main result of the work was a patch correcting the bug mentioned at the beginning of the article. Almost all the code fit in one class (not counting the tests), which currently looks like this: TryWithResourcesMethodVisitor . I urge everyone to criticize and offer their best solutions to this problem.

Secondly, I learned what are the ways to compile the try-with-resources block. As a result, I figured out how try-catch-finally looks at the bytecode level. Well, a by-product was the translation of the article , which I mentioned above.

Thirdly, I wrote this article, where I told you everything. Perhaps now one of you will be able to increase the fundamental dragging factor using the knowledge gained.

And where is the benefit and morality, you ask? I leave their search to the reader. I note only that I enjoyed it while I was writing this article. Hope you got it from reading. See you again!

PS As a bonus, I propose to look at the early proposals for the implementation of try-with-resources from Joshua Bloch.


It looks funny.

 { final LocalVariableDeclaration ; boolean #suppressSecondaryException = false; try Block catch (final Throwable #t) { #suppressSecondaryException = true; throw #t; } finally { if (#suppressSecondaryException) try { localVar.close(); } catch(Exception #ignore) { } else localVar.close(); } }