How we did the build

Introduction

When a project grows, acquires a lot of modules and dependencies, setting up the project manually can take an unreasonable amount of time. In addition, all the technical participants in the project, from developers, testers, to administrators, are engaged in setting up, and therefore time wasting.

In addition, it is necessary to constantly update the development and test systems, and even nothing to confuse. There can not do without the practice of continuous integration .


This article shows the experience of implementing assemblies using maven, so basic knowledge of maven is necessary. For a quick start it will be enough to study the site http://www.apache-maven.ru , for a more detailed development of the topic - you should refer to the original source http://maven.apache.org
')
The purpose of the article is not to teach using Mayven, but to show the experience of implementation with positive and negative solutions. With this knowledge, it is much easier and faster to implement maven assemblies in your team.

Why not ant?

When warning such questions, I’ll say that we use ant , not for the assembly itself, but for technical functions, see below.

Another important advantage of maven is the ability to collect only the code with which you work, everything else is collected at night on a separate server.

Environment

First of all, we should talk about the structure of our projects and approaches to development. This is very important, due to the fact that the project has been developing for many years and has a structure that does not correspond to maven-webapp-artifact

And so, we have a lot of modules stored in one SVN repository. Each module has a main development branch (trunk), as well as tags (tags) and branches (branches) as needed. Most modules contain both java-classes and jsp-pages, scripts, graphics, etc. Most modules are independent web applications and can operate independently. Examples of modules: core, soap, workflow, addons, common ... I counted about 100 of them.

A project most often consists of a separate module in which specific resources and logic are placed in java-classes. Sometimes, there is no such module. The final assembly of the project consists of a set of modules in the form of a single web-application.

The necessary libraries for the module are stored in the lib folder directly in the repository, for web resources there is its own sx-war folder. As well as the standard src and test folders.



Naturally, almost all modules depend on the core, and from each other, of necessity.

There is also a database with meta-information. Modules interact on the basis of it, all work with the user is built. For example, a publication block is also a unit of meta-information and is stored in a database. That is, in addition to the data itself, the database stores settings for working with this data. They are configured in the console, which is also built on the publication blocks. It is possible to transfer meta-information using service packs.

Ideally, each module has its own common package with meta-information. The core database is the lightest database and can be expanded with any module. The kernel is almost application-server, only without the possibility of dynamic loading / unloading of multifunctional modules. Not all modules have separated meta, but we strive for this :)

BC

We have such a phrase. It means that “this” was a very long time ago, even before the start of our work in the company. I'll tell you how the assembly setup went before me. I also had to spend a lot of time setting it up for the first time.

And so you got us a job. The second day is coming. The first went to view screencasts on subsystems and training work with meta-information. You need to configure the assembly. It is good if the knowledge base for the project contains information on the necessary modules for the project. Often this information is not relevant (forget to update or update from time to time). Information about the dependencies of the modules themselves is not found, and the one that is, is also becoming obsolete.

The trunk versions of all necessary modules are extracted from SVN, a project is created in the IDE. And then the most difficult thing: set up the import of all necessary libraries and module connections. It is still complicated when it is necessary to exclude some versions of libraries from one modules and use others. Set up the assembly of all modules in one folder. Next, configure the web application server (tomcat is used).

And now the project is set up, then you need to configure the second, third, version of the first (use branches to work on errors). Developer becomes uncomfortable.

The dependencies have changed, the assemblies of all have broken. And administrators use ant to create production assemblies - they also break everything. Often there are problems due to different assemblies of the administrator and developer.

First try

We had no special people responsible for the assembly, as there was no good knowledge of maven. Therefore, the first attempt to introduce maven on projects failed.

The parent pom.xml was placed in a separate repository folder. After that, in a separate project folder, svn-externals were configured and all the modules necessary for the project were included, plus the parent pom.xml in a separate folder:



The classical approach using the hierarchy of modules in a single SVN structure did not suit us, because each module is used in dozens of projects. And not just in one. And any manipulations with parent pom on projects are undesirable. In addition, fixing the version of any module will automatically affect all projects.

All necessary libraries were not searched on http://search.maven.org , because there are a lot of them and all of them are impersonal in the repository (libraries without versions are stored in the lib folder. For example, axis.jar, jtds.jar, velocity.jar). They did it simply: all the libraries were uploaded to the Nexus using a script. GroupId and ArtifactId were set by sx. <Library name> (sx.axis).

Here is what a typical pom.xml module looked like:


It can be seen that it is necessary to change the list of modules for each project.

At the time of assembly, each module was assembled into a separate war, and then unpacked into a common deployment using maven-overlay .

Reasons for failure

Completely crossed out the advantages of the maven approach using svn-externals. This property does not allow to unambiguously fix the code of all modules without manually editing the svn properties of a folder. This becomes practically unreal if there are 10-15 modules and a patch is released for a certain version. And so wanted to use the maven release plugin .

pros

• Everything is going "here and now." The developer does not depend on the update period for SNAPSHOT dependencies (builds from the main development branch);
• The developer can make changes to several modules at once (a dubious item, because the encapsulation of modules is broken);
• The whole set of necessary modules is visible;

Minuses

• Complete assembly of all modules is a lengthy procedure;
• It is impossible to properly fix the version of the code;
• All libraries are not taken from the public repository, adding a new one is a separate procedure;
• It is possible jar-hell, when someone decides to use the library from the public repository and it already exists in the corporate (in the patch will get both versions, and even three!). This happens because the GroupId: ArtifactId: Version changes;
• The code seems to be stored in the same SVN folder, and the checkout must be completely different (with svn-externals). Not transparent - for aesthetes;

Second try

The second attempt to implement assemblies was developed in several iterations.

Iteration 1

First of all, it was necessary to obtain a mechanism for fixing the code (release version) of both individual modules and the entire product. For this, we got rid of using svn-externals. Each module turned out to be autonomous and assembled separately. In the final assembly fell all the dependencies that the module has.

The parent pom.xml was also modified in accordance with the tasks. All modules use it as a base:

Iteration 2

In all modules of the project a version release plugin was implemented. For this, blocks were added to each module:


There were thoughts to render this configuration in the base pom.xml, and use the ArtifactId-Version property, but not in all projects this value coincides with the name of the folder in the repository. Therefore, the idea was postponed until better times. The first version was just the base version for all pom.xml modules.

In the base pom.xml added a new profile: production. It uploads the build to corporate FTP and may further extend to projects.

Iteration 3

After an unpleasant collision with jar-hell , it was decided to put the libraries in order in all assemblies. And it was not easy, because only in the core there were more than 100 libraries and almost all without versions. Each one had to be found in the public repository, the “calculate” version. Libraries opened with “surprises” with internal builds and patches, modified versions. We carefully placed them in the maven-repository with preserving the coordinates of the artifact, but with the _custom postscript in Version.

We use Nexus as a repository. On it the gateway to public repositories with caching was configured. A single entry point has appeared: a corporate maven repository. Before that there were more than three of them: libraries, releases, daily builds and other public repositories. All required authorization. Three similar repositories in each pom.xml, and even the access parameters in settings.xml caused constant confusion.

Attempted to leave a link to the repository only in the base pom.xml. This is quite realistic, but when you first build any module, Maiwen must load the base module from the repository, and its coordinates are unknown. You can do it manually, after that everything works fine. Refused because of the opacity of this action.

Iteration 4

We set up nightly assemblies of all base and project modules in the Hudson continuous integration system. There were no problems with them, because the modules are now completely independent. It makes no difference whether this is a public library or project module. Maven-based builds are run after every code change in SVN and successful artifacts get into the Nexus repository.

Over time, we set up a cascade assembly of modules in tasks, as well as deployments for development assemblies and assemblies for testing. The warmup is performed using rather simple ant-scripts. They extract the latest module assembly ( dependency: get ) from Nexus, unpack it on the server and restart the application server (the scripts themselves can be found at the end of the article).

pros

• You can work with each module separately;
• Fixing the version of all module dependencies, you can fix the entire code of the project;
• Continuous integration eliminates the routine work and allows you to quickly transfer functionality to the production environment and to the customer;
• The release of the module version is fully controllable and quite simple;
• Now you can develop with only one specific module. Indeed, the total deployment of several modules can affect the final behavior of the code;
• Thanks to Hudson, it's always clear who broke the build;
• There is no need to set up projects at all; the IDE does everything itself. You just need to run the build (and the mvn package is enough);
• It is always clear on which code the module depends. And before, the trunk was always going with its own mistakes, broken commits, etc .;

Minuses

• The IDE does not have a common project with all modules, so making changes to several modules at once requires separate work with each module. This is certainly not very convenient, but conceptually correct, because any module must be self-sufficient;
• It is not possible to release all versions of components in a cascade when the main product is released
• ...

Develop a solution

In the process, the assembly made changes. For those who are still assembling “in the old-fashioned way”, a knowledge base page has been created that contains dependencies of the modules. This information is updated on the basis of the pom.xml module every night with a separate task in Hudson (do not forget about DRY ).

To speed up the assembly of modules, unnecessary work during assembly was eliminated. For example, we build a soap module that depends on the core. The core code falls into the final soap assembly. Next is going to the project, which depends on the core and soap. soap is not used separately, so the core is excluded from its assembly, and it is included in the project. Thereby we accelerate assembly. For the ability to build soap with the inclusion of a dependent core, the contact text profile has been added, which runs when necessary.

You may notice that all modules are found in pom.xml twice: one record is used during compilation, the second means that the module falls into the patch. In order not to replace the module versions throughout pom.xml, all versions of the corporate modules were moved to separate properties and are located at the very beginning of the file. This reduces the likelihood of error when switching to another version:



ant-scripts for working with maven artifacts

Conclusion

So continuous integration was introduced in our project. Missing the need for manual code updates on the servers. You can always see the results of the assembly, which are sent via RSS. A large screen monitoring of the assembly has not yet been delivered.

New developers do not spend a lot of time setting up assemblies. And it only pleases.

The release of code versions has been improved, now this process is almost completely automated and does not take much time. It has become much easier to accompany previously released versions.

New developments fall on the development contexts within 24 hours, which means they get to the customer faster. Difficult perceptible errors of integration also began to manifest themselves, and therefore, to correct, faster.

UPDATE: Useful article on implementation experience from commenting: http://bazhenov.me/blog/2011/04/09/build.html