Jigsaw project in Java 9. A modular future that cannot be avoided

We hear about the Java modularization project year after year. Every time we wait, follow the announcements. We are told that Jigsaw is almost ready, but every time they postpone the release. Maybe this is not such a simple project, as many see it? It is possible that changes in it will affect the development of the root? Maybe the modular system is just the tip of the iceberg? We asked Ivan Krylov to answer questions related to the Jigsaw project.

Ivan has been developing Java virtual machines and compilers for over 10 years, and is currently developing the compiler infrastructure in the Zing virtual machine in Azul Systems. Regular speaker at JUG.ru and other European Java conferences.
')

What is Jigsaw, and how will it affect the Java world?

- Ivan, good afternoon. Please tell us what impact the Jigsaw project will have on the Java ecosystem?

- Kind. To explain my view on Java and modularity, it’s worth talking a little about my business. I have been doing commercial programming for about twenty years. The last eleven of these are the development of virtual machines. Worked in the HotSpot Runtime team at Sun. Then the company was bought by Oracle. Then there were a few changes in activity, and for the last three and a half years I have been working at Azul Systems, where I have been developing virtual machines.

All this I have told you to understand that my view on functionality is the view of the JVM developer, a person who is quite far from the world of the "bloody enterprise", from the problems faced by developers of a high level of abstraction.

Returning to the question of the impact of Jigsaw on the Java ecosystem, I would say that initially this influence is not as great as it might seem. Let's start with the fact that if a developer does not want to switch to the paradigm of modules, he may not move. Old ways of interacting with code and scopes in general will work. Problems will begin when the developer tries to transfer his code to the modules, and another code has not yet been transferred to the modular infrastructure.

Other problems will start when using the OSGi mechanism, when they want to switch to the Jigsaw dependency mechanism, it may be difficult to maintain two ways of writing these dependencies, but these are different special cases. In general, for a certain number of developers, the Jigsaw project will pass quite imperceptibly, therefore, to evaluate globally for everyone, probably there will not be one universal answer.

- In Java 8, lambda expressions and the Stream API are key innovations. What significant changes can Jigsaw compare to?

- Jigsaw project modules do not have a big impact on writing code and on the functional sequence of the code. Jigsaw is responsible for how the interaction between the various components is written, how we wrap the code in units of abstraction. In Java 9, no new APIs are planned that would be revolutionary or greatly change how developers write their code today.

There are certain evolutionary changes: for example, Optional appeared in Java 8, a few methods will be added to Optional in Java 9, such as converter to Stream , or method, but these are incremental changes. We felt the need to add - added. Large pleasant changes in Process API , which will allow you to find out the pid of the virtual machine, getting the process tree, process management. The Process API will allow you to get rid of a number of hacks that needed to be wrapped: for example, shell calls with parsing output or other methods will leave another necessity to get into native. Such incremental changes are scattered across the class libraries' code, but there is not one code in the API or one powerful syntactic change.

All semantic changes are collected in the Milling Project Coin . The most noticeable is probably the appearance of private (default) interface methods. Imagine a situation where two private interface methods contain common code. Previously, such code could only be rendered in a public method, but such a method may not make sense from the point of view of the public API offered by this interface. Now such a method can be declared private. This method will not be visible as part of the API of this interface, but its implementation can be safely used inside these default methods. This change will make the API that you propose using the interface and, in particular, offering default implementations of the methods of this interface, cleaner.

- The project of modularization has existed for a long time, about nine years, the release dates were postponed more than once. At the JavaOne 2015 conference, this topic was honored by several speakers (Alan Bateman, Alex Buckley, Mark Reinhold). Does such a stir around show the importance of the project?

- The Jigsaw project pulled some changes. I consider Jigsaw as some two parts, the first is a change in the language of the visibility model, the module declaration, and the syntax. The second part is how the module model was applied directly to the class libraries in jdk. Some of the classes moved, others became deprecated or were no longer visible outside.

The big debate was about the mechanism of reflection. Previously, he allowed to reach out to any private class and method. There were possible field changes on the fly through reflection and their privacy. That is, I would say that exploits were available that were used by a large number of libraries. Ultimately, this would be reflected in users. The language architects you listed went to the community to talk about what changes will be, why they will happen, why you need to make these painful changes. The pain is that a certain number of libraries automatically became immediately incompatible. Translation of the code to a more compatible version takes time for them, which, as usual, is accompanied by noise, for example, like this: “switching to the ninth version breaks everything”. These explanations were necessary.

A large number of reports about Jigsaw last year were on JavaOne, almost the same number on Devoxx in Antwerp. This will be the same reports, but there will be interesting from outsiders, for example, the guys from IBM tell us whether there is life after the transition to modules. I will be looking forward to watching videos from JavaOne 2016.

- New concept - module. How to correctly interpret the “component” parts of a module: module name, exports and requires? I would like to elaborate on “requires” and understand what we can actually “require”, for example, “requires local” or “requires optional”?

- Let's start with a simple book example, a situation where you do not have existing code, you just write it from scratch. You have classes and you have placed them in different packages, decided that they should not have a close one-to-one connection, and they are in different modules. When some class from the first module needs classes from the second module, it says that I need, that is, requires , this module is two , indicating which package it needs. The whole hierarchy is quite strict, if the package manages abstractions at the class level, the module manages the level hierarchy of the package, this requires expression .

Validation occurs in both cases: at compile time and at execution time, as often happens with other elements of the Java language. The JVM cannot guarantee that all checks have been passed, let's say, honestly, at the compilation level, bytecode can come from anywhere, so in many respects the checks are duplicated in javac and at the JVM Runtime level. Requires checks that the packages necessary for the operation of this module are available, these packages are located in some module two or module three . If, in turn, some other modules are needed, then such a transitive chain is built in order to check literally at the very beginning of work that all the necessary packages from the corresponding modules are available to us.

Then, exports is the mechanism of visibility. He says that this module provides such packages to other modules. You can specify that all at once is a conditional "*", which shows that all other modules, all those who want to use this module, can use packages. This means that not all classes are available to users, but only those that are publicly declared, i.e. those that are escaped are made available to other modules through this mechanism. You can specify export to when you want to export directly to the specified module and to no one else, and this static declaration mechanism.

In addition, there are certain runtime flags that allow you to change these scopes at the time of the call for an already compiled module, so that you do not need to reassemble the module. If he did not have enough exports for successful work, then you can add them on the command line, but this is seen as an intermediate stage, because, in theory, all declarations should be in a module. Those. A module is a kind of complete unit with a declaration of what is needed and what is not for this module and what it provides to other modules for use.

In principle, jdk itself is rebuilt so that the entire jdk library set represents a set of modules with clear dependencies. The result is that we can build a subset jre in which there will be only those modules that we need. In this regard, the jlink utility has appeared, which allows us to assemble a Run-Time Image only from those modules that we need. This is a definite development of the idea that appeared in Java 8 with profiles. In jdk 8 it was possible to build jdk under different profiles with different numbers of classes and different amounts of functionality, but this is, let's say, a static assembly, without checking what the application really needs. And jlink will allow us to build some portable Runtime, which will consist of a virtual machine, a java.base module, and a number of modules needed by the application.

In one presentation, the authors of the book about modules in java 9 Paul Bakker and Sander Mak gave various examples. We assembled a variant of the program of a simple text analyzer with the necessary runtime - we got a distribution kit of about 20 megabytes. When the swing windows were added to the application, new modules were added, the size of this distribution grew to 50 megabytes. The borders of the JDK libraries are smeared out here, because in addition to the modules that are inside jdk and that will be supplied with jdk, you can also add modules from third-party libraries, i.e. such distributions will be completely differently arranged. This will allow a new program to be distributed, it is not required that the customer go and download the new version of jdk, but deliver it directly with Runtime compiled.

- Can we forget about the problem of " Dependency hell " forever?

- Quite often in real production systems you can see a very long line of the classpath, which lists a large number of jar, or directories, where the compiled code is located. It often happens that the classes declare the same classes of the same packages, and then Runtime takes the one that previously appeared in the classpath. In this regard, many people have noted such a problem, when the behavior of the program changed from changing places of jar in the class-path, this is solved with the help of modules in the bud.

This happens as follows: if we have classes laid out in packages, then a particular package can declare a single module. Therefore, if, for example, we have two modules in two jar and both of them include package elements, then such a situation is impossible, because it is assumed that the package now belongs to one of the modules, because Each module declares the package uniquely. However, Jigsaw does not solve the problem of version control, i.e. the correspondence of versions in the understanding in which we are accustomed to see in package managers, for example, in Linux.

The task of the version control modules initially stood in earlier versions, but then it was removed, the only thing that can be done is to write the version in metadata. You can extract this metadata, but the class loading mechanism itself doesn’t do any additional checks; it is at the mercy of other package managers, such as OSGi, that can be put on top. This is due to computational difficulties. This is quite a difficult task that the classloader didn’t want to give up, so it was given to external tools.

- Different types of modules will appear: Named Modules, The Unnamed Module, Automatic Modules. Can you talk briefly about them? What are their differences and what does each bring to the overall structure?

- If you write a module from scratch, by all the rules, that is, put it in the top-level directory with the module name, add the file module-info.java , which contains the module declaration, the name that should coincide with the directory name, with listing exports and requires . Such a module is a regular named module.

If you live the old fashioned way and you have classes that you just used to throw through the classpath, they will also end up in a module called the Unnamed Module. Therefore, for the modules that you did not name, there is a nameless last resort, since now the nine will not be able to work without modules.

There is a certain intermediate stage, when you have, for example, a jar file, there are some packages in it, you put it not in the classpath, but in the modulepath, then Runtime starts to think of it as a certain module whose name coincides with the name jar -file, and with it you can continue to do similar things, as with the Named module, and refer to it.

For example, if you are developing your own named module. You depend on some kind of library that has not yet been converted to a modular structure, then you can put the jar with this library in the modulepath. It will become an automatic module. Now you can refer to module-info.java : requires and then the name of the module that was assigned to it automatically.

Finally, the last change that is occurring in the last weeks, and it is not quite clear whether this proposal will be adopted or not. I said a little earlier that there is a certain problem with reflection, if before it was possible to get through to anything, now there is protection at the virtual machine level, and a new concept appears, which is currently under development, and may fall into the standard, although this is not guaranteed - a weak module.

A Weak module is an intermediate state in which the code in this module can, through reflection, reach classes that are in named modules, and even those that are not exported correctly, but at the same time such a module cannot export its functionality to the outside. This was done specifically to go through this transition period, when we will have to get through reflection, reach the internal code, which was not intended for this purpose.

I think that over time the use of weak modules will come to naught, maybe they will remain for debugging purposes. Like Automatic modules, it is an intermediate mechanism that allows you to go through the migration path from unmodulated code to modularized code.

OSGi vs Jigsaw: there will be no holivar

“Why hasn't OSGi been approved for Java modularization?” You agree with the words of Mark Reinhold : “The OSGi module layer is not operative at compile time; it only addresses the modularity during packaging, deployment, and execution. It’s since it’s built on the Platform itself. There is a Penrose project for OSGi / Jigsaw interaction . (The OSGi modular layer does not act at compile time; it refers to modularity only during archiving, deployment, or execution. In addition, as it is (OSGi), it is useful for libraries and application modules, but since it is created strictly based on the Java SE Platform and cannot be used to modularize its platform (note the author’s translation).

- I specifically studied OSGi, and I had a definite opinion on why OSGi didn’t become a Jigsaw. The first answer on the surface, it was marked by Mark Reinhold: the mechanism itself is written on top of the Java SE specification, if we want to supply Java SE with fragments, that is, modules, the OSGi mechanism will not help us, we need something at a lower level. In general, OSGi works through the class loading mechanism, that is, one common OSGi environment classloader first appears, then OSGi appears on its own classloader for each bundle, and due to this, some distinction is made between the visibility of classes.

The Jigsaw mechanism does the same functionality, but already at the virtual machine level, which, in turn, is significant enough for the performance of the virtual machine itself. It's no secret that inside a virtual machine a class is addressed through a sequence: classloader and then the package and the name of the class, through this pair. And if the package and the name of the class are concrete string literals that correspond to it, then the classloader is just a reference to the class loading'a object. Thus, in the case of persist, when we finished the virtual machine and then try to recover this data during the next boot, this reference loses any meaning, the classloader does not have its own string literal. This imposes restrictions on the optimization of work in such a ahead of time style. Therefore, the Jigsaw developers decided to abandon the basing of this visibility mechanism on the class loading mechanism, as it was done in OSGi. OSGi did this because it was the only thing that was available to them from what the virtual machine provides, they could not interfere with the class resolution mechanism.

- It turns out that the actual implementation of OSGi and Jigsaw at different levels.

- Yes, quite right. Jigsaw implemented at a lower level. Fortunately or misfortune. Probably, to some misfortune for those who have already managed to use OSGi. Now you have to somehow interact, maintain two parallel dependency descriptions, or use a project like Penrose that allows you to synchronize. The description is set using a JSON script, from which you can generate a modular description, a Jigsaw dependency description, and an OSGi description. We'll have to cross these two worlds, and for those users who use OSGi now, life will become a bit more difficult.

- What changes will occur in the work of classloaders?

“The changes aren't really that big.” , , Bootstrap classloader, Extension classloader Application classloader, instance of URL classloader. Java 9 , Extension classloader Platform classloader , Extension , jdk, . Java 9 , , Bootstrap classloader, Platform classloader. , Application classloader, instance of URL classloader, instance . , , Gradle, , instance of URL classloader — . Gradle Jigsaw .

, , , Gradle, . , Application classloader'. .

, , , — .

, — , , , .. , Whole Program Optimization, , .. , jit- . , , Oracle ahead of time jmod, , . , , , .

: ,

— ?

— , Jigsaw , . , , - requires . , , , , «Hello World». -: requires exports. , , , , , , .

, , . IDE, , , .

, private -, . javac , , . , , , , class resolution, , , , . Those. , .

— , IDE «Jigsaw». , . ?

— , , , , module , StackOverflow Google , .

IDE: , IDEA, «», , Jigsaw. , IDE, , IDE , .. «».

— Jigsaw — «». , ?

— , OSGi, , , , , . , , , .

Java 8 jdep , jdk, Java 9 jar module-info.java , , , requires . Jdep module-info.java, , , .. , , jdep .

, , , .

---

Java 9 Joker 2016 Sander Mak, Java 9 Modularity O'Reilly, Java 9 Modularity in Action .

«» JVM Java 9 , , Joker 2016:

• JIT code life cycle
• From Java to Assembly: Down the Rabbit Hole
• Java on Elbrus
• HotSpot Internals: Safepoints, NullPointers and StackOverflows
• Close JMM Degree Contacts
• G1 GC, G1 GC
• Native code, off-heap data and java

– .