Geekly Articles each Day
📜 ⬆️ ⬇️

Purely experimental techniques for porting the Stream API from Java 8 to Java 6

A year ago, I talked about how using Maven and Retrolambda to port my application using the Java 8 language tools, as well as the associated “not quite Java 8” libraries, to Android. Unfortunately, the new Java 8 APIs cannot be used due to their banal absence on an older target platform. But, since the idea itself did not leave me for a long time, it became interesting to me: is it possible to port, for example, the Stream API to an older platform and not be limited to only the capabilities of a language like lambda expressions.


Ultimately, this idea implies the following: as in the previous case, using the available tools, in particular the good old Retrolambda, rewrite the Stream API bytecode so that the code using this API can work on older versions of Java. Why Java 6? Honestly, I worked with this version of Java for a longer time, I didn’t find Java 5, and for Java 7, I rather flew past.



Also I repeat right away that all the instructions given in this article are purely experimental in nature, and hardly practical. First of all, due to the fact that you have to use the boot-classloader, which is not always acceptable or even possible. And secondly, the realization of the idea itself is frankly damp and there are a lot of inconveniences and not quite obvious pitfalls.


Instruments


So, the set of necessary tools is represented by the following main packages:



And related tools involved in the experiment:



In addition to older versions of OpenJDK, an example of porting will be done using Ant, not Maven. Though I am a supporter of the convention over configuration and have not used Ant for five or six years, for this particular task, Ant seems to me a much more convenient tool. First of all, due to simplicity, as well as due to fine-tuning, which, to tell the truth, is difficult to achieve in Maven, work speed and cross-platform (shell scripts would be even shorter, but I also often use Windows without Cygwin and similar lotions).


As proof of concept a simple example will be used on the Stream API.


package test; import java.util.stream.Stream; import static java.lang.System.out; public final class EntryPoint { private EntryPoint() { } public static void main(final String... args) { runAs("stream", () -> Stream.of(args).map(String::toUpperCase).forEach(EntryPoint::dump)); } private static void runAs(final String name, final Runnable runnable) { out.println("pre: " + name); runnable.run(); out.println("post: " + name); } private static void dump(final Object o) { out.println(">" + o); } }

A few words about how the experiment will take place. Ant-ovsky build.xml divided into many steps or stages, each of which is given its own directory during the porting process. This, at least for me, great simplifies the process of finding a solution and debugging, to track changes from step to step.


Porting process


Step 0. Init


As usual, the first thing to do in Ant is almost always to create a target directory.


 <target name="init" description="Initializes the workspace"> <mkdir dir="${targetDir}"/> </target>

Step 1. Grab


An extremely important part of the experiment is the minimum accurate list of all the classes on which the test case depends. Unfortunately, I don’t know if this can be done easier, and I spent quite a lot of time to register all the necessary classes from JRE 8 using the method of repeated restartings.


On the other hand, there is some sense in trying to tighten the entire java.util.stream package and then spend even more time pulling up other dependencies (and, most likely, processing with tools like ProGuard). But I decided to go for another simple trick: I simply copy nested and inner classes using the $** mask. This is a very significant time saving and list. Some of the classes that existed in older versions of Java will most likely also need to be copied, since they have gained new features in Java 8. This concerns, for example, the default Map.putIfAbsent(Object,Object) , which is not involved in the test, but is required for its correct operation.


 <target name="01-grab" depends="init" description="Step 01: Grab some JRE 8 classes"> <unzip src="${java.home}/lib/rt.jar" dest="${step01TargetDir}"> <patternset> <include name="java/lang/AutoCloseable.class"/> <include name="java/lang/Iterable.class"/> <include name="java/util/Arrays.class"/> <include name="java/util/AbstractMap.class"/> <include name="java/util/EnumMap.class"/> <include name="java/util/EnumMap$**.class"/> <include name="java/util/function/Consumer.class"/> <include name="java/util/function/Function.class"/> <include name="java/util/function/Supplier.class"/> <include name="java/util/Iterator.class"/> <include name="java/util/Map.class"/> <include name="java/util/Objects.class"/> <include name="java/util/Spliterator.class"/> <include name="java/util/Spliterator$**.class"/> <include name="java/util/Spliterators.class"/> <include name="java/util/Spliterators$**.class"/> <include name="java/util/stream/AbstractPipeline.class"/> <include name="java/util/stream/BaseStream.class"/> <include name="java/util/stream/ForEachOps.class"/> <include name="java/util/stream/ForEachOps$**.class"/> <include name="java/util/stream/PipelineHelper.class"/> <include name="java/util/stream/ReferencePipeline.class"/> <include name="java/util/stream/ReferencePipeline$**.class"/> <include name="java/util/stream/Sink.class"/> <include name="java/util/stream/Sink$**.class"/> <include name="java/util/stream/Stream.class"/> <include name="java/util/stream/StreamShape.class"/> <include name="java/util/stream/StreamOpFlag.class"/> <include name="java/util/stream/StreamOpFlag$**.class"/> <include name="java/util/stream/StreamSupport.class"/> <include name="java/util/stream/TerminalSink.class"/> <include name="java/util/stream/TerminalOp.class"/> </patternset> </unzip> </target>

Indeed, a very impressive list of classes, needed only for simple ones, as it first seems, map() and forEach() .


Step 2. Compile


Boring test code compilation. There is simply no place.


 <target name="02-compile" depends="01-grab" description="Step 02: Compiles the source code dependent on the grabbed JRE 8 classes"> <mkdir dir="${step02TargetDir}"/> <javac srcdir="${srcDir}" destdir="${step02TargetDir}" source="1.8" target="1.8"/> </target>

Step 3. Merge


This step may seem a bit strange, since it simply merges the result of copying classes from Java 8 rt.jar and a test case. In fact, this is necessary for the next few steps that move the Java packages for proper post-processing.


 <target name="03-merge" depends="02-compile" description="Step 03: Merge into a single JAR in order to relocate Java 8 packages properly"> <zip basedir="${step01TargetDir}" destfile="${step03TargetFile}"/> <zip basedir="${step02TargetDir}" destfile="${step03TargetFile}" update="true"/> </target>

Step 4. Shade


For Maven, there is one interesting plugin that can move packages by changing the bytecode of class files directly. I don’t know, maybe I’ve been looking badly on the Internet, is there an Ant-based counterpart, but I have no choice but to write a small Ant extension myself, which is a simple Maven-plug-in adapter with the only possibility: only package movement. No other maven-shade-plugin features.


At this stage, in order to continue using Retrolambda, you need to rename all java.* Packages to something like ~.java.* (Yes, it is “tilde” - why not?). The fact is that Retrolambda relies on the work of the java.lang.invoke.MethodHandles class, which prohibits the use of classes from java.* Packages (and sun.* , As it is in Oracle JDK / JRE). Therefore, temporary relocation of packages is simply a way to “blind” java.lang.invoke.MethodHandles .


As in step # 1, I had to specify the full list of classes separately through the include list. If this is not done and the list is omitted completely, the shade in the class files will also move those classes that are not planned to be processed. In this case, for example, java.lang.String will become ~.java.lang.String (at least, it can be clearly seen from the classes decompiled using javap ) that will break Retrolambda, which will just silently stop converting code and will not generate any class for lambda / invokedynamic . I consider it more impractical to list all classes in the exclude-list, because they are simply harder to find and would have to poke around in class-files with javap in search of an extra tilde.


 <target name="04-shade" depends="03-merge" description="Step 04: Rename java.* to ~.java.* in order to let RetroLambda work since MethodHandles require non-java packages"> <shade jar="${step03TargetFile}" uberJar="${step04TargetFile}"> <relocation pattern="java" shadedPattern="~.java"> <include value="java.lang.AutoCloseable"/> <include value="java.lang.Iterable"/> <include value="java.util.Arrays"/> <include value="java.util.AbstractMap"/> <include value="java.util.EnumMap"/> <include value="java.util.EnumMap$**"/> <include value="java.util.function.Consumer"/> <include value="java.util.function.Function"/> <include value="java.util.function.Supplier"/> <include value="java.util.Iterator"/> <include value="java.util.Map"/> <include value="java.util.Objects"/> <include value="java.util.Spliterator"/> <include value="java.util.Spliterator$**"/> <include value="java.util.Spliterators"/> <include value="java.util.Spliterators$**"/> <include value="java.util.stream.AbstractPipeline"/> <include value="java.util.stream.BaseStream"/> <include value="java.util.stream.ForEachOps"/> <include value="java.util.stream.ForEachOps$**"/> <include value="java.util.stream.PipelineHelper"/> <include value="java.util.stream.ReferencePipeline"/> <include value="java.util.stream.ReferencePipeline$**"/> <include value="java.util.stream.Sink"/> <include value="java.util.stream.Sink$**"/> <include value="java.util.stream.Stream"/> <include value="java.util.stream.StreamShape"/> <include value="java.util.stream.StreamOpFlag"/> <include value="java.util.stream.StreamOpFlag$**"/> <include value="java.util.stream.StreamSupport"/> <include value="java.util.stream.TerminalSink"/> <include value="java.util.stream.TerminalOp"/> </relocation> </shade> </target>

A small digression. Theoretically, duplication of the list in Ant can be solved with elements that support refid , but this will not work for several reasons:



So all these shortcomings are solved, but obviously not within the framework of this article.


Step 5. Unzip


The next step unpacks the JAR file with relocated packages, since Retrolambda can only work with directories.


 <target name="05-unzip" depends="04-shade" description="Step 05: Unpacking shaded JAR in order to let Retrolamda work"> <unzip src="${step04TargetFile}" dest="${step05TargetDir}"/> </target>

Step 6. Retrolambda


The very heart of the experiment: the conversion of bytecode version 52 (Java 8) to version 50 (Java 6). And because of the tricks used above, Retrolambda (or, therefore, JDK 8) calmly and without any questions will instruct the classes. You also need to include support for default methods, because the set of new functions in Java 8 is built on them. Since JRE 7 and below cannot work with such methods, Retrolambda simply copies the implementation of such a method for each class in which it has not been redefined (which, by the way, means that you need to use Retrolambda only for the bundle “final application and its libraries” otherwise it is likely that you may encounter a problem when the implementation of the default method is simply absent).


 <target name="06-retrolambda" depends="05-unzip" description="Step 06: Perform downgrade from Java 8 to Java 6 bytecode"> <java jar="${retrolambdaJar}" fork="true" failonerror="true"> <sysProperty key="retrolambda.bytecodeVersion" value="50"/> <sysProperty key="retrolambda.classpath" value="${step05TargetDir}"/> <sysProperty key="retrolambda.defaultMethods" value="true"/> <sysProperty key="retrolambda.inputDir" value="${step05TargetDir}"/> <sysProperty key="retrolambda.outputDir" value="${step06TargetDir}"/> </java> </target>

Step 7. Zip


Putting the instrumented version back into one file to launch the shade plugin in the opposite direction:


 <target name="07-zip" depends="06-retrolambda" description="Step 07: Pack the downgraded classes back before unshading"> <zip basedir="${step06TargetDir}" destfile="${step07TargetFile}"/> </target>

Step 8. Unshade


Fortunately, only two parameters are enough for a shade plug-in to move in the opposite direction. Upon completion of this step, the packages in the application will be aligned back, and all that was ~.java.* Will again become java.* .


 <target name="08-unshade" depends="07-zip" description="Step 08: Relocate the ~.java package back to the java package"> <shade jar="${step07TargetFile}" uberJar="${step08TargetFile}"> <relocation pattern="~.java" shadedPattern="java"/> </shade> </target>

Step 9. Unpack


In this step, the classes are simply unpacked to build two separate JAR files. Again, nothing interesting.


 <target name="09-unpack" depends="08-unshade" description="Step 09: Unpack the unshaded JAR in order to create two separate JAR files"> <unzip src="${step08TargetFile}" dest="${step09TargetDir}"/> </target>

Steps 10 and 11. Pack


Putting all the classes together, but separately - the “new runtime” and the test application itself. And again - a very trivial and uninteresting step.


 <target name="10-pack" depends="09-unpack" description="Step 10: Pack the downgraded Java 8 runtime classes"> <zip basedir="${step09TargetDir}" destfile="${step10TargetFile}"> <include name="java/**"/> </zip> </target> <target name="11-pack" depends="09-unpack" description="Step 11: Pack the downgraded application classes"> <zip basedir="${step09TargetDir}" destfile="${step11TargetFile}"> <include name="test/**"/> </zip> </target>

Result testing


That's all. In the target directory is a tiny port of a small aspect from the real Stream API, and it can run on Java 6! To do this, create another rule for Ant:


 <target name="run-as-java-6" description="Runs the target artifact in Java 6"> <fail unless="env.JDK_6_HOME" message="JDK_6_HOME not set"/> <java jvm="${env.JDK_6_HOME}/bin/java" classpath="${step11TargetFile}" classname="${mainClass}" fork="true" failonerror="true"> <jvmarg value="-Xbootclasspath/p:${step10TargetFile}"/> <arg value="foo"/> <arg value="bar"/> <arg value="baz"/> </java> </target>

And here you just need to pay special attention to the use of not quite standard -Xbootclasspath/p . In short, its essence is as follows: it allows the JVM to specify from where to load the base classes in the first place. In this case, the other classes from the original rt.jar will be lazily loaded from $JAVA_HOME/jre/lib/rt.jar as needed. You can -verbose:class this by using the -verbose:class key when starting the JVM.


Running the example itself also requires the JDK_6_HOME environment variable pointing to JDK 6 or JRE 6. Now when calling run-as-java-6 result of successful porting will be output to the standard output:


 PRE: stream >FOO >BAR >BAZ POST: stream

Works? Yes!


Conclusion


Getting used to writing code for Java 8, I want this code to work on older versions of Java. Especially if there is a rather old and weighty code base. And if on the Internet you can often see the question of whether there is even an opportunity to work with the Stream API on older versions of Java, they will always say no. Well, almost not. And they will be right. Of course, alternative libraries with similar functionality are available that work on older JREs. I personally like Google Guava most of all, and I often use it when Java 8 is not enough.


An experimental hack is an experimental hack, and I doubt that further demonstration there is a lot of sense to go further. But, for the purpose of research and the spirit of experimentation, why not? You can learn more about the experiment on GitHub .


Unsolved and unresolved issues


Besides the problem with refid in Ant, several questions remain open to me personally:


Does this example work on other JVM implementations?

Runs on the Oracle JVM, but the Oracle license prohibits the deployment of applications replacing part of rt.jar using -Xbootclasspath .


Is it possible to create a list of dependency classes automatically, without resorting to manual iteration?

I personally do not know the automatic methods of such analysis. You can try to pull off the entire java.util.stream.* Package, but I think there will be more problems.


Is it possible to run this example on Dalvik VM?

This refers to Android. I tried to skip the results through dx and run the Dalvik VM with -Xbootclasspath directly on the real device, but Dalvik persistently ignores such a request. I suspect that the reason for this is that the applications for Dalvik VM are forged by Zygote , which obviously does not suspect such intentions. You can read more about why this can not be done and what it is fraught with, on StackOverflow . And if I managed to run dalvikvm with -Xbootclasspath , I suspect it would take some kind of launcher for the application itself, which this boot classpath would replace. Such a scenario, apparently, is not possible.


What about GWT?

And this is a completely different story and a different approach. Just the other day, the long-awaited release of GWT 2.8.0 (unfortunately, version 2.7.0 two years ago) took place, in which lambda and other features for source code written in Java 8 were fully implemented. However, that was all before the release in SNAPSHOT Versions You can’t mess with bytecode in GWT, because GWT only works with source code. To port the Stream API to the client side, I think, it’s just to collect some of the source code from JDK 8, first passing it through a preprocessor that converts the source code into a view that is digestible for GWT (RxJava porting example).


')

Source: https://habr.com/ru/post/313888/

More articles:


All Articles