It was a beautiful May day. My view accidentally fell on the chat guys from the last server. Their May day was not so beautiful: during the redeployment of a secondary service, the authorization service fell due to it insofar as. Tsimes of the situation is that nobody supports the falling part of the authorization service, he passed on to us by inheritance and never really failed. I was fascinated by the detective search for the reasons, and until a certain point I was a passive reader - until I saw the phrase of our administrator, filled with the acquired gray hair of his hair: “In an hour, 800+ streams flow.”

This is already interesting! On Java, streams flow at such a pace, so that this over the years not to notice - it is not so simple that I voiced it. And since in this chat I was the only Java developer, it was only a matter of time until someone said, "If you are so smart, take it and fix it." And it does not matter that you are a client, and in general you have been writing for the last three years under Android.

Why not?

Step 1: take the samples for review. We build “Thread”, “Executor” and ... we do not find anything. But we find a certain library in which all calls leave.
')
Step 2: we take the library samples and ... they are not. What a twist! How did this happen? Yes, very simple. The project consists of 300+ services. He has a rich and complex story with unexpected turns. And when transferring all these wonders, in some places without documentation, with different repositories, languages ​​and technologies, it’s technically possible not to keep track of everything, especially since everything compiles perfectly, or lies in the project as a jar.

In general, sortsy are not particularly needed for familiarization. Intellij Idea quite reasonably decompiles the code. Even with a cursory reading hair stood on end. The word "Executor" has not yet been met, but "new thread" was literally everywhere. On this we set aside the code. Right now, looking for a leak in it is no easier than a needle in a stack of needles. Take a better thread dump and see:

"Thread-782" daemon prio=10 tid=0x00007f7db4654800 nid=0x2286d9 in Object.wait() [0x00007f7b929d6000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000000b843f3c8> (a java.util.LinkedList) at java.lang.Object.wait(Object.java:503) at com.aol.saabclient.SAABConnection.AsynchMsgProcessor.run(AsynchMsgProcessor.java:83) - locked <0x00000000b843f3c8> (a java.util.LinkedList) "Thread-781" daemon prio=10 tid=0x00007f7db4651000 nid=0x2286d7 in Object.wait() [0x00007f7de37ee000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000000b843f3c8> (a java.util.LinkedList) at java.lang.Object.wait(Object.java:503) at com.aol.saabclient.SAABConnection.AsynchMsgProcessor.run(AsynchMsgProcessor.java:83) - locked <0x00000000b843f3c8> (a java.util.LinkedList) "Thread-780" daemon prio=10 tid=0x00007f7db464f000 nid=0x2286d5 in Object.wait() [0x00007f7de118a000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000000b843f3c8> (a java.util.LinkedList) at java.lang.Object.wait(Object.java:503) at com.aol.saabclient.SAABConnection.AsynchMsgProcessor.run(AsynchMsgProcessor.java:83) - locked <0x00000000b843f3c8> (a java.util.LinkedList) 

Well, since it's such a booze, let's go straight to AsyncMsgProcessor (yes, SAABConnection is a package). There we see something like a manual implementation of the blocking queue around LinkedList. It is clear, that the parsing cycle is eternal, and this may even be desirable behavior. It also becomes clear that AsyncMsgProcessor is created for each connection, but here is a static (LinkedList messages). Thus, since all AsyncMsgProcessors are sorting out the same queue, you can simply limit their number. We are looking for instantiation, and find only one. Fine! It remains to change the direct instantiation to the pool and we will be happy.

There are two ways to do this:

1. Insert the decompiled code back into the compiler and pray that the decompiler does not fix it. This is the path of the dark side, as it leads to unpredictable bugs;
2. To correct the byte code of one small method by hand. There are less chances to make a mistake, which means this is the path of a real Jedi.

Rule byte code

To parse and collect class files back, you need a more or less specific tool. I found only this one: JBE - Java Bytecode Editor . It has a big problem with editing the code: you need to count all offsets in conditional and unconditional jumps with your hands, which is, in general, a so-so perspective, even for a relatively small method. Again, because of the great chance of making a mistake, any change will be given by blood and sweat. Among the less ready for direct use there is an excellent, very powerful thing - ASM . But out of the box it is not possible to first output as text, then edit and assemble back. But you can teach.

Khachim conclusion

For text output, the classes are Textifier + TraceMethodVisitor. But from such a conclusion it would be quite problematic to collect everything back, so that the bytecode would not change (at least functionally). Therefore, a few hacks:

 Textifier textifier = new Textifier(Opcodes.ASM5) { @Override public void visitLabel(Label label) { buf.setLength(0); buf.append('#'); appendLabel(label); buf.append(":\n"); text.add(buf.toString()); } @Override public void visitLineNumber(int line, Label start) { buf.setLength(0); buf.append("// line ").append(line).append('\n'); text.add(buf.toString()); } @Override public void visitMaxs(int maxStack, int maxLocals) { buf.setLength(0); buf.append("// MAXSTACK = ").append(maxStack).append('\n'); text.add(buf.toString()); buf.setLength(0); buf.append("// MAXLOCALS = ").append(maxLocals).append('\n'); text.add(buf.toString()); } @Override public void visitLdcInsn(Object cst) { buf.setLength(0); buf.append(tab2).append("LDC "); if (cst instanceof String) { Printer.appendString(buf, (String) cst); } else if (cst instanceof org.objectweb.asm.Type) { buf.append(((org.objectweb.asm.Type) cst).getDescriptor()).append(".class"); } else if (cst instanceof Long) { buf.append(cst).append('L'); } else if (cst instanceof Float) { buf.append(cst).append('F'); } else if (cst instanceof Double) { buf.append(cst).append('D'); } else if (cst instanceof Integer) { buf.append(cst); } else { throw new IllegalArgumentException("cst " + cst); } buf.append('\n'); text.add(buf.toString()); } @Override public void visitFrame(int type, int nLocal, Object[] local, int nStack, Object[] stack) { } }; 

Hachim input

The input is harder. ASM has a MethodNode class that is a visitor. It is usually implied that the MethodNode slips into accept ClassReader, filling in from it, possibly modifying. We want to slip into it the text generated at the last step (it was there that “modifications” were to occur). We simulate the behavior of the Reader:

  for (String line : methodCode.getText().split("\n")) { int lastCommentPos = line.lastIndexOf("//"); if (lastCommentPos != -1) { line = line.substring(0, lastCommentPos); } line = line.trim(); if (line.isEmpty()) { continue; } String[] withParams = line.split("\\s+"); String command = withParams[0]; if (command.startsWith("#")) { verify(command.endsWith(":")); String substring = command.substring(1, command.length() - 1); method.visitLabel(getLabel(substring, labels)); } else if (command.equals("TRYCATCHBLOCK")) { verify(withParams.length == 5); Label start = getLabel(withParams[1], labels); Label end = getLabel(withParams[2], labels); Label handler = getLabel(withParams[3], labels); String type = withParams[4]; if (type.equals("null")) { type = null; } method.visitTryCatchBlock(start, end, handler, type); } else { Opcode opcode = OPCODES.get(command); //   ASM if (opcode == null) { throw new RuntimeException("Unknown " + command); } else { switch (opcode.type) { case OpcodeGroup.INSN: verify(withParams.length == 1); method.visitInsn(opcode.opcode); break; case OpcodeGroup.INSN_INT: verify(withParams.length == 2); method.visitIntInsn(opcode.opcode, Integer.valueOf(withParams[1])); break; case OpcodeGroup.INSN_VAR: verify(withParams.length == 2); method.visitVarInsn(opcode.opcode, Integer.valueOf(withParams[1])); break; case OpcodeGroup.INSN_TYPE: verify(withParams.length == 2); method.visitTypeInsn(opcode.opcode, withParams[1]); break; case OpcodeGroup.INSN_FIELD: verify(withParams.length == 4); verify(withParams[2].equals(":")); int dotIndex = withParams[1].indexOf('.'); String owner = withParams[1].substring(0, dotIndex); String name = withParams[1].substring(dotIndex + 1); method.visitFieldInsn(opcode.opcode, owner, name, withParams[3]); break; case OpcodeGroup.INSN_METHOD: verify(withParams.length == 3); dotIndex = withParams[1].indexOf('.'); owner = withParams[1].substring(0, dotIndex); name = withParams[1].substring(dotIndex + 1); method.visitMethodInsn(opcode.opcode, owner, name, withParams[2], opcode.opcode == INVOKEINTERFACE); break; case OpcodeGroup.INSN_JUMP: verify(withParams.length == 2); method.visitJumpInsn(opcode.opcode, getLabel(withParams[1], labels)); break; case OpcodeGroup.INSN_LDC: withParams = line.split("\\s+", 2); verify(withParams.length == 2); method.visitLdcInsn(parseLdc(withParams[1])); break; case OpcodeGroup.INSN_IINC: verify(withParams.length == 3); method.visitIincInsn(Integer.valueOf(withParams[1]), Integer.valueOf(withParams[2])); break; case OpcodeGroup.INSN_MULTIANEWARRAY: verify(withParams.length == 3); method.visitMultiANewArrayInsn(withParams[1], Integer.valueOf(withParams[2])); break; default: throw new IllegalArgumentException(); } } } } 

In the end, we did a sidecode insert. It remains to saw the same pool. I will not give the code, there is even more of everything ... But the approach is very simple: we take the modified library, put it in dependencies, write the necessary classes and recompile. After that, the threads stopped flowing, everything works, happy ending. In fact, everything, of course, was not so - it took 5-7 layouts in the test environment. That byte-code is not a byte-code, the pool is not a pool ...

And the conclusion of this story is simple: there are no impossible tasks, even the loss of source codes is not a catastrophe. And if there is no necessary tool, it can always be built with the help of the same scrap materials.

PS: Nobody has put me a beer.