Parse lambda expressions in java

From the translator: LambdaMetafactory is perhaps one of the most underrated mechanisms of Java 8. We discovered it recently, but we already appreciated its capabilities. In version 7.0 of the CUBA framework, performance is improved by eliminating reflective calls in favor of generating lambda expressions. One of the applications of this mechanism in our framework is to bind application event handlers by annotations, a common task, an analogue of EventListener from Spring. We believe that knowledge of how LambdaFactory works can be useful in many Java applications, and we hasten to share this translation with you.

In this article, we will show some little-known tricks when working with lambda expressions in Java 8 and the limitations of these expressions. The target audience of the article is senior Java developers, researchers and toolkit developers. Only public Java API without com.sun.* And other internal classes will be used, so the code is portable between different JVM implementations.

Short preface

Lambda expressions appeared in Java 8 as a way to implement anonymous methods and,
in some cases, as an alternative to anonymous classes. At the bytecode level, the lambda expression is replaced by the invokedynamic instruction. This instruction is used to create the implementation of the functional interface and its only method delegates a call to the actual method that contains the code defined in the body of the lambda expression.

For example, we have the following code:

 void printElements(List<String> strings){ strings.forEach(item -> System.out.println("Item = %s", item)); } 

This code will be converted by the Java compiler into something like:

 private static void lambda_forEach(String item) { // Java  System.out.println("Item = %s", item); } private static CallSite bootstrapLambda(Lookup lookup, String name, MethodType type) { // //lookup =  VM //name = "lambda_forEach",  VM //type = String -> void MethodHandle lambdaImplementation = lookup.findStatic(lookup.lookupClass(), name, type); return LambdaMetafactory.metafactory(lookup, "accept", MethodType.methodType(Consumer.class), //  - MethodType.methodType(void.class, Object.class), //  Consumer.accept    lambdaImplementation, //     - type); } void printElements(List<String> strings) { Consumer<String> lambda = invokedynamic# bootstrapLambda, #lambda_forEach strings.forEach(lambda); } 

The invokedynamic instruction can be roughly represented as such Java code:

 private static CallSite cs; void printElements(List<String> strings) { Consumer<String> lambda; //begin invokedynamic if (cs == null) cs = bootstrapLambda(MethodHandles.lookup(), "lambda_forEach", MethodType.methodType(void.class, String.class)); lambda = (Consumer<String>)cs.getTarget().invokeExact(); //end invokedynamic strings.forEach(lambda); } 

As you can see, the LambdaMetafactory is used to create a CallSite that provides a factory method that returns a target method handler. This method returns the implementation of the functional interface using invokeExact . If lambda expressions have captured variables, then invokeExact accepts these variables as actual parameters.

In Oracle JRE 8, the metafactory dynamically generates a Java class using ObjectWeb Asm, which creates the implementation class of the functional interface. Additional fields can be added to the created class if the lambda expression captures external variables. This is similar to anonymous Java classes, but there are the following differences:

• An anonymous class generated by a Java compiler.
• The class for implementing lambda expressions is created by the JVM at runtime.

The implementation of the metafactory depends on the JVM vendor and on the version

Of course, the invokedynamic is not only used for lambda expressions in Java. It is mainly used when running dynamic languages ​​in a JVM environment. The Nashorn engine for JavaScript execution, which is embedded in Java, makes heavy use of this instruction.

Next, we focus on the LambdaMetafactory class and its capabilities. Following
The section of this article assumes that you understand how metafactory methods work and what MethodHandle

Tricks with lambda expressions

In this section, we show how to build dynamic constructions of lambda for use in daily tasks.

Checked exceptions and lambdas

It is no secret that all functional interfaces that are in Java do not support checked exceptions. The advantages of the checked exceptions over the usual ones are a very long-standing (and still hot) dispute.

And what if you need to use code with checked exceptions inside lambda expressions in combination with Java Streams? For example, you need to convert a list of strings into a list of URLs like this:

 Arrays.asList("http://localhost/", "https://github.com").stream() .map(URL::new) .collect(Collectors.toList()) 

In the URL constructor (String), a checked exception is declared, so it cannot be used directly as a reference to a method in the Functiion class.

You say: "No, maybe if you use this trick":

 public static <T> T uncheckCall(Callable<T> callable) { try { return callable.call(); } catch (Exception e) { return sneakyThrow(e); } } private static <E extends Throwable, T> T sneakyThrow0(Throwable t) throws E { throw (E)t; } public static <T> T sneakyThrow(Throwable e) { return Util.<RuntimeException, T>sneakyThrow0(e); } //   //return s.filter(a -> uncheckCall(a::isActive)) // .map(Account::getNumber) // .collect(toSet()); 

This is a dirty hack. And that's why:

• A try-catch block is used.
• The exception is thrown again.
• The dirty use of type erasure in java.

The problem can be solved in a more "legal" way, using knowledge of the following facts:

• Checked exceptions are recognized only at the Java compiler level.
• The throws section is just metadata for a method without a semantic value at the JVM level.
• Checked and normal exceptions are not distinguishable at the bytecode level in the JVM.

The solution is to wrap the Callable.call method in a method without the throws section:

 static <V> V callUnchecked(Callable<V> callable){ return callable.call(); } 

This code will not compile because the Callable.call method Callable.call checked exception exceptions in the throws section. But we can remove this section using a dynamically constructed lambda expression.

First we need to declare a functional interface in which there is no throws section.
but who can delegate the call to Callable.call :

 @FunctionalInterface interface SilentInvoker { MethodType SIGNATURE = MethodType.methodType(Object.class, Callable.class);//  INVOKE <V> V invoke(final Callable<V> callable); } 

The second step is to create an implementation of this interface using the LambdaMetafactory and delegate the call to the SilentInvoker.invoke method to the Callable.call method. As mentioned earlier, the throws section is ignored at the bytecode level, so the SilentInvoker.invoke method can call the Callable.call method without declaring exceptions:

 private static final SilentInvoker SILENT_INVOKER; final MethodHandles.Lookup lookup = MethodHandles.lookup(); final CallSite site = LambdaMetafactory.metafactory(lookup, "invoke", MethodType.methodType(SilentInvoker.class), SilentInvoker.SIGNATURE, lookup.findVirtual(Callable.class, "call", MethodType.methodType(Object.class)), SilentInvoker.SIGNATURE); SILENT_INVOKER = (SilentInvoker) site.getTarget().invokeExact(); 

The third is to write a helper method that calls Callable.call without declaring exceptions:

 public static <V> V callUnchecked(final Callable<V> callable) /*no throws*/ { return SILENT_INVOKER.invoke(callable); } 

Now you can rewrite stream without any problems with checked exceptions:

 Arrays.asList("http://localhost/", "https://dzone.com").stream() .map(url -> callUnchecked(() -> new URL(url))) .collect(Collectors.toList()); 

This code will compile without problems, because there is no declaration of checked exceptions in callUnchecked . Moreover, calling this method can be inline using monomorphic inline caching , because it is only one class in the entire JVM that implements the SilentOnvoker interface SilentOnvoker

If the Callable.call implementation throws an exception at runtime, it will be intercepted by the calling function without any problems:

 try{ callUnchecked(() -> new URL("Invalid URL")); } catch (final Exception e){ System.out.println(e); } 

Despite the possibilities of this method, you should always remember the following recommendation:

Hide checked exceptions with callUnchecked only if you are sure that the code you are calling will not throw out any exceptions.

The following example shows an example of this approach:

 callUnchecked(() -> new URL("https://dzone.com")); // URL        MalformedURLException 

The full implementation of this method is here , it is part of the open source project SNAMP .

We work with Getters and Setters

This section will be useful to those who write serialization / deserialization for various data formats, such as JSON, Thrift, etc. Moreover, it can be quite useful if your code relies heavily on reflection for Getters and Setters in JavaBeans.

A getter declared in a JavaBean is a method called getXXX with no parameters and a return data type other than void . A setter declared in a JavaBean is a method named setXXX , with one parameter and returning void . These two notations can be represented as functional interfaces:

• Getter can be represented by the Function class, in which the argument is the value of this .
• Setter can be represented by the BiConsumer class, in which the first argument is this and the second is the value that is passed to Setter.

Now we will create two methods that can convert any getter or setter into these
functional interfaces. Never mind that both interfaces are generics. After erasing types
the actual data type will be Object . Automatic casting of the return type and arguments can be done using the LambdaMetafactory . In addition, the Guava library will help with caching lambda expressions for identical getters and setters.

First step: you need to create a cache for getters and setters. The Method class from the Reflection API represents a real getter or setter and is used as the key.
The cache value is a dynamically constructed functional interface for a specific getter or setter.

 private static final Cache<Method, Function> GETTERS = CacheBuilder.newBuilder().weakValues().build(); private static final Cache<Method, BiConsumer> SETTERS = CacheBuilder.newBuilder().weakValues().build(); 

Second, create factory methods that create an instance of a functional interface based on getter or setter references.

 private static Function createGetter(final MethodHandles.Lookup lookup, final MethodHandle getter) throws Exception{ final CallSite site = LambdaMetafactory.metafactory(lookup, "apply", MethodType.methodType(Function.class), MethodType.methodType(Object.class, Object.class), //signature of method Function.apply after type erasure getter, getter.type()); //actual signature of getter try { return (Function) site.getTarget().invokeExact(); } catch (final Exception e) { throw e; } catch (final Throwable e) { throw new Error(e); } } private static BiConsumer createSetter(final MethodHandles.Lookup lookup, final MethodHandle setter) throws Exception { final CallSite site = LambdaMetafactory.metafactory(lookup, "accept", MethodType.methodType(BiConsumer.class), MethodType.methodType(void.class, Object.class, Object.class), //signature of method BiConsumer.accept after type erasure setter, setter.type()); //actual signature of setter try { return (BiConsumer) site.getTarget().invokeExact(); } catch (final Exception e) { throw e; } catch (final Throwable e) { throw new Error(e); } } 

Automatic type casting between Object type arguments in functional interfaces (after erasing types) and actual argument types and return values ​​is achieved using the difference between samMethodType and instantiatedMethodType (the third and fifth arguments of the metafactory method, respectively). The type of the created instance of the method is the specialization of the method that provides the implementation of the lambda expression.

Third, create a facade for these factories with caching support:

 public static Function reflectGetter(final MethodHandles.Lookup lookup, final Method getter) throws ReflectiveOperationException { try { return GETTERS.get(getter, () -> createGetter(lookup, lookup.unreflect(getter))); } catch (final ExecutionException e) { throw new ReflectiveOperationException(e.getCause()); } } public static BiConsumer reflectSetter(final MethodHandles.Lookup lookup, final Method setter) throws ReflectiveOperationException { try { return SETTERS.get(setter, () -> createSetter(lookup, lookup.unreflect(setter))); } catch (final ExecutionException e) { throw new ReflectiveOperationException(e.getCause()); } } 

Method information obtained from an instance of the Method class using the Java Reflection API can be easily converted to MethodHandle . Bear in mind that the class instance methods always have a hidden first argument used to pass this into this method. Static methods have no such parameter. For example, the real signature of the Integer.intValue() method looks like an int intValue(Integer this) . This trick is used in our implementation of functional wrappers for getters and setters.

And now - time to test the code:

 final Date d = new Date(); final BiConsumer<Date, Long> timeSetter = reflectSetter(MethodHandles.lookup(), Date.class.getDeclaredMethod("setTime", long.class)); timeSetter.accept(d, 42L); //the same as d.setTime(42L); final Function<Date, Long> timeGetter = reflectGetter(MethodHandles.lookup(), Date.class.getDeclaredMethod("getTime")); System.out.println(timeGetter.apply(d)); //the same as d.getTime() //output is 42 

This approach with cached getters and setters can be effectively used in libraries for serialization / deserialization (such as Jackson), which getters and setters use during serialization and deserialization.

Calling functional interfaces with dynamically generated implementations using LambdaMetaFactory significantly faster than calls through the Java Reflection API

The full code can be found here , it is part of the SNAMP library.

Limitations and bugs

In this section, we will look at some of the bugs and limitations associated with lambda expressions in the Java compiler and the JVM. All of these restrictions can be replicated in OpenJDK and Oracle JDK with javac version 1.8.0_131 for Windows and Linux.

Creating lambda expressions from method handlers

As you know, a lambda expression can be constructed dynamically using LambdaMetaFactory . To do this, you need to define a handler — the MethodHandle class, which points to the implementation of a single method that is defined in the functional interface. Let's take a look at this simple example:

 final class TestClass { String value = ""; public String getValue() { return value; } public void setValue(final String value) { this.value = value; } } final TestClass obj = new TestClass(); obj.setValue("Hello, world!"); final MethodHandles.Lookup lookup = MethodHandles.lookup(); final CallSite site = LambdaMetafactory.metafactory(lookup, "get", MethodType.methodType(Supplier.class, TestClass.class), MethodType.methodType(Object.class), lookup.findVirtual(TestClass.class, "getValue", MethodType.methodType(String.class)), MethodType.methodType(String.class)); final Supplier<String> getter = (Supplier<String>) site.getTarget().invokeExact(obj); System.out.println(getter.get()); 

This code is equivalent to:

 final TestClass obj = new TestClass(); obj.setValue("Hello, world!"); final Supplier<String> elementGetter = () -> obj.getValue(); System.out.println(elementGetter.get()); 

But what if we replace the method handler that points to getValue to the handler that represents the getter of the field:

 final CallSite site = LambdaMetafactory.metafactory(lookup, "get", MethodType.methodType(Supplier.class, TestClass.class), MethodType.methodType(Object.class), lookup.findGetter(TestClass.class, "value", String.class), //field getter instead of method handle to getValue MethodType.methodType(String.class)); 

This code should, expectedly, work because the findGetter returns a handler that points to the getter of the field and has the correct signature. But, if you run this code, you will see the following exception:

 java.lang.invoke.LambdaConversionException: Unsupported MethodHandle kind: getField 

Interestingly, getter for the field works fine if we use MethodHandleProxies :

 final Supplier<String> getter = MethodHandleProxies .asInterfaceInstance(Supplier.class, lookup.findGetter(TestClass.class, "value", String.class) .bindTo(obj)); 

It should be noted that MethodHandleProxies is not a very good way to dynamically create lambda expressions, because this class simply wraps MethodHandle into a proxy class and delegates the call to InvocationHandler.invoke to MethodHandle.invokeWithArguments . This approach uses Java Reflection and is very slow.

As shown earlier, not all method handlers can be used to create lambda expressions during code execution.

Only a few types of method handlers can be used to dynamically create lambda expressions.

Here they are:

• REF_invokeInterface: can be created using Lookup.findVirtual for interface methods
• REF_invokeVirtual: can be created using Lookup.findVirtual for virtual class methods
• REF_invokeStatic: created using Lookup.findStatic for static methods
• REF_newInvokeSpecial: can be created using Lookup.findConstructor for constructors
• REF_invokeSpecial: can be created using Lookup.findSpecial
  for private methods and early binding with virtual class methods

The remaining types of handlers will cause a LambdaConversionException error.

Generic exceptions

This bug is related to the Java compiler and the ability to declare generic exceptions in the throws section. The following code example demonstrates this behavior:

 interface ExtendedCallable<V, E extends Exception> extends Callable<V>{ @Override V call() throws E; } final ExtendedCallable<URL, MalformedURLException> urlFactory = () -> new URL("http://localhost"); urlFactory.call(); 

This code must compile because the URL class constructor throws a MalformedURLException . But it does not compile. You receive the following error message:

 Error:(46, 73) java: call() in <anonymous Test$CODEgt; cannot implement call() in ExtendedCallable overridden method does not throw java.lang.Exception 

But, if we replace the lambda expression with an anonymous class, then the code will compile:

 final ExtendedCallable<URL, MalformedURLException> urlFactory = new ExtendedCallable<URL, MalformedURLException>() { @Override public URL call() throws MalformedURLException { return new URL("http://localhost"); } }; urlFactory.call(); 

Therefore:

Type inference for generic exceptions does not work correctly in combination with lambda expressions

Limitations of the type of parameterization

You can construct a generic object with several type constraints using the & sign: <T extends A & B & C & ... Z> .
This method of determining generic parameters is rarely used, but in a certain way affects lambda expressions in Java due to some limitations:

• Each type constraint other than the first must be an interface.
• The pure version of the class with such generic only takes into account the first type constraint from the list.

The second constraint leads to different code behavior at compile time and at run time, when binding to the lambda expression occurs. This difference can be demonstrated using the following code:

 final class MutableInteger extends Number implements IntSupplier, IntConsumer { //mutable container of int value private int value; public MutableInteger(final int v) { value = v; } @Override public int intValue() { return value; } @Override public long longValue() { return value; } @Override public float floatValue() { return value; } @Override public double doubleValue() { return value; } @Override public int getAsInt() { return intValue(); } @Override public void accept(final int value) { this.value = value; } } static <T extends Number & IntSupplier> OptionalInt findMinValue(final Collection <T> values) { return values.stream().mapToInt(IntSupplier::getAsInt).min(); } final List <MutableInteger> values = Arrays.asList(new MutableInteger(10), new MutableInteger(20)); final int mv = findMinValue(values).orElse(Integer.MIN_VALUE); System.out.println(mv); 

This code is absolutely correct and successfully compiled. The MutableInteger class satisfies the constraints of a generic type T:

• MutableInteger inherited from Number .
• MutableInteger implements IntSupplier .

But the code will fall with the exception at runtime:

 java.lang.BootstrapMethodError: call site initialization exception at java.lang.invoke.CallSite.makeSite(CallSite.java:341) at java.lang.invoke.MethodHandleNatives.linkCallSiteImpl(MethodHandleNatives.java:307) at java.lang.invoke.MethodHandleNatives.linkCallSite(MethodHandleNatives.java:297) at Test.minValue(Test.java:77) Caused by: java.lang.invoke.LambdaConversionException: Invalid receiver type class java.lang.Number; not a subtype of implementation type interface java.util.function.IntSupplier at java.lang.invoke.AbstractValidatingLambdaMetafactory.validateMetafactoryArgs(AbstractValidatingLambdaMetafactory.java:233) at java.lang.invoke.LambdaMetafactory.metafactory(LambdaMetafactory.java:303) at java.lang.invoke.CallSite.makeSite(CallSite.java:302) 

This happens because the JavaStream pipeline captures only the pure type, which, in our case, is the Number class and does not implement the IntSupplier interface. This problem can be fixed by explicitly declaring the type of the parameter in a separate method used as a reference to the method:

 private static int getInt(final IntSupplier i){ return i.getAsInt(); } private static <T extends Number & IntSupplier> OptionalInt findMinValue(final Collection<T> values){ return values.stream().mapToInt(UtilsTest::getInt).min(); } 

This example demonstrates incorrect type inference in the compiler and runtime.

Handling several generic parameter type constraints in conjunction with the use of lambda expressions during compilation and execution is inconsistent