Conflict Resolution in Transitive Dependencies - Good, Bad, Angry

Instead of the preface

Next Saturday, we will be performing in St. Petersburg on JUG.ru with EvgenyBorisov . There will be a lot of fun trash and interesting information (sometimes you can’t figure out where the border is), and one of my speeches will be devoted to the WTF nude of modular program development. I will tell a few horror films, one of which will be about how everyone is trying to quickly, flexibly and correctly describe the dependencies in the project, and what usually comes out of it. Interesting? Then welcome to hell!


Rather, of course, “Good, Convenient and WTF”.

A little bit of theory ...

What is a dependency manager and why is it needed?

Any modern build tool (especially in the JVM world) includes (or has an easy-to-connect) dependency manager (aka dependency manager). The most famous are, of course, Apache Maven , Apache Ivy (dependency manager for Apache Ant), and Gradle .
One of the main functions of the build tools in the JVM world is to create classpaths. They are used during the build process in many places - to compile code, to run tests, to build archives (war, ear, distributives and installers) and even to set up projects in the IDE.
To facilitate the process of finding online, downloading, storing and configuring dependencies and there are dependency managers. You declare that you need, for example, commons-lang , and voila, you have it.
')

What are transitive dependencies?

Transitive dependency is the artifact on which the direct dependence of the project depends. Imagine the following situation:

Our project A depends on two artifacts - E and B At this direct dependencies end and begin transitive ( C, D ). As a result, we get dependency chains, artifacts in which can be repeated ( D , in our example)

Why do we need transitive dependencies?

Obviously not to compile. But, for the rest, perhaps, they are needed - these artifacts must be in the archives builds, they must be in the classpath to run tests, and the paths to them must be given through the API for integration with the IDE.

How can a conflict arise?

If we look at the diagram above, we will see the same conflict. The classpath of project A must contain both artifact D version 1 (E depends on it) and artifact D version 2 ( C depends on it)!

Why is that bad?

The JVM (and javac) defines the uniqueness of a class by its name (and classloader, but in our simple example, all classes are loaded with one classloader). In the case when there are two classes with the same name in the classpath, only the first one will be loaded. If we assume that there are classes with the same name in D1 and D2 (you will agree, most likely, it is), then the class from the jar, which will be registered in the second generated classpath, will simply not be loaded. Which one will be the first? It depends on the logic of the dependency manager and, in general, is unknown.
As you understand, this is a conflict:

What to do?

Who is to blame is clear (Java, not those whom you thought about), but what can be done?
There are several strategies for resolving conflicts in transitive dependencies (some of them are logical, others are absurd), but, naturally, there is no silver bullet. Let's look at some of them:

• Latest . The “Newest” strategy implies backward compatibility. If D2 is fully compatible with D1, then leaving only the newer artifact (D2) in the classpath will get the correct operation of C (after all, it is written under D2), but also the correct operation of E (after all, if D2 is backward compatible, then it works exactly same as D1, under which is written E). This strategy is of two subspecies - the newest version, and the newest by date. Most often, they will work the same way (except in cases where there is none).
  In the case of our example, using the latest in the classpath will be D2.
• Fail (aka Strict). With this strategy, the build will fail when the dependency manager detects a conflict. Naturally, the safest, but also the most time-consuming strategy.
  In the case of our example, when using fail, the build will fail.
• All (aka No-conflict). “The one and the other, and it is possible without bread” means that both D1 and D2 from our example will be in the classpath (in arbitrary order). Hell? Hell! But in the case of using classpath isolation technologies (by loading different modules with different classloaders), it may well be not only useful, but also necessary.
  In the case of our example, when using all in the classpath, both D1 and D2 will appear.
• Nearest . The “Nearest” strategy is a completely and completely magnificent WTF, which I will gladly discuss below. Stay tuned.
• Custom . In this case, the dependency manager will ask you what the master will do. This, of course, is “manual control”, but can sometimes be quite useful. Here is an example of pseudocode on pseudo-gruvi:
  

   coflictManager = {artifact, versionA, versionB -> //,        Apache,    if(artifact.org.startsWith ('org.apache')){ [versionA, versionB].max() } else { fail() } } 

  
  In the case of our example, when using this custom implementation, assuming that the org of D1 and D2 starts with 'org.apache', then the classpath will have D2, otherwise, the build will fall.
  

Who is that much

Now let's see which of the Der Grosse Troika mentioned above that he can.

Apache ivy

In terms of conflict managers, Ivy is beautiful. They are connected, both are the latest, as well as fail and all go in the box. With custom, too, everything is beautiful. You can fully implement your logic, or you can use a half-finished product and only come up with a suitable regex. By default, the latest (version) works.

Gradle

In the first versions of Gradle (up to 0.6, if memory serves me) Ivy was used as a dependency manager. Accordingly, all of the above was true for Gradle too, but the guys from Gradleware wrote their dependency manager (mainly because of problems with the local Ivy storage with a parallel build, one of the main advantages of Gradle). In the process of releasing their manager, such “minor” features as a replacement for the conflict manager were pushed far into the roadmap, and for a long time Gradle existed only with the latest. Do not like the latest - disable transitive dependencies, and go ahead, list everything manually. But today everything is in order. Starting from 1.0, there is fail , and from 1.4, there is also custom .

Apache maven

Well, for the sake of the next picture, the whole post was conceived.
Which version of D will you find in the classpath? D1? D2? both? none? assembly will fall?

As you have probably already guessed, D1 gets into the classpath (which is very likely to lead to problems, because all the code in C written for new functionality that does not exist in D1 will simply fall). This is the most wonderful WTF that I promised you. Maven works with the unique nearest strategy, which selects in the classpath that artifact that is closer to the root of the project (A) in the project tree.

How so? What kind of nonsense?

The root of the problem lies in the difficulty of eliminating dependency in Maven. If, for example, you want to use D2, not D1, then, for good, you should say to Maven: Dear Maven, never use D1. Just for example, in Gradle we would write this:

 configurations {all*.exclude group: 'mygroup', module: 'D', version: '1'} 

The problem is that expressing it in Maven is impossible. (An experienced, and therefore attentive and thoughtful user of Maven will exclaim here, "But what about the enforcer-plugin ?!" And it will be wrong ). You can say specifically to module E: “did you think you have a dependency on D? So, it is not. This is a good way out, there is no more conflict, D2 in the classpath, win. But this solution is not scalable at all. What if dozens of artifacts depend on D1? At different levels of transitivity?

Well, and where does the nearest?

The problem of lack of global exclude was solved in Maven in a very “interesting” way. It was decided that if you declared in your project A a dependency with a certain version, then only this version will fall into the classpath. That is, in practice, this is the ultimated nearest - it cannot be closer than to A (this is root), so the conflict is resolved, we don’t need to look for all the places where we need to exclude D. On the way, however, we got very strange and hardly predictable behavior in those when A does not declare D directly (see our example), but what is, that is.

Interestingly enough, the idea that “what the user himself declared is law” is used in Gradle too, and this does not prevent them from using sane strategies like latest and fail for everything else.

Update: several people in the comments reminded that the enforcer-plugin has the functionality of fail. This partially solves the problem. Remains: 1. wild nearest by default. 2. solutions to the problem - to register all conflicting dependencies in your project (tolerably), or infinite exclude (hell).

And if the same depth?

This wonderful question (what to do if in our example B depended on D2) did not occur to the guys from Maven for two and a half years (from the 2.0 release in October 2005 to the version 2.0.9 in April 2008) and which artifact will be in the classpath was just vague . In Maven 2.0.9, a willful decision was made - it will be the first!

How does this help us? Right, no way. Because we generally do not know which of them will be the first, because transitive dependencies do not manifest themselves until a conflict occurs (or until we begin to investigate this riddle). Thank you guys!

Instead of an epilogue

Maven's WTF nude is, of course, not limited to the miraculous creation of the alternative mind — the nearest strategy. But today, I think, that's enough. Holivary in the comments are welcome in every way (if anything, I’ll follow the Gradle), and all St. Petersburg people come to JUG on Saturday, the 31st in PetroCongress to continue the banquet.