Migrating data with mongoDB and Spring Boot

Each developer sooner or later encounters the need to migrate data in a database. On our project we use mongoDB as a database. We approached data migration in various ways:

• write js scripts and run directly in the database

• used Mongobee - a tool for automatic migrations

  
  

  Mongobee worked fine until we were faced with a situation where we wanted to add a new field with a unique index. Suppose we have a class:

  
  

  @Document @Data public class TestDocument { @Id private String id; private String text; } 

  
  

  Now we add a new field:

  
  

   @Document @Data public class TestDocument { @Id private String id; private String text; @Indexed(unique = true) private String text2; } 

  
  

  We wrote a migration that should fill in the text2 field in all documents with unique values:

  
  

   @ChangeLog public class TestDocumentChangelog { @ChangeSet(order = 1, id = "change1", author = "Stepan") public void changeset(MongoTemplate template) { template.findAll(Document.class, "testDocument").forEach(document -> { document.put("text2", UUID.randomUUID().toString()); template.save(document, "testDocument"); }); } } 

  
  

  But when we launched the application, we saw that Spring launched the Mongo Data components earlier than our migration started, and therefore an automatic creation of an index over the text2 field resulted in an error, since this field was not yet filled in one document.
  After this incident, we decided to abandon Mongobee and try to write our own tool, which would also make it easy to write migrations, but in addition would have such functions as:

  
  
• Run before Mongo Data
• Transaction support added in MongoDB 4.0
• Embedding dependencies in the classes ChangeLog

The result was a library called Mongration, which supports all the functionality described above.

Spring Boot life cycle support

The first function is implemented using auto configuration, which starts after creating MongoClient and before scanning repositories:

 @Configuration @AutoConfigureAfter(MongoAutoConfiguration.class) @AutoConfigureBefore(MongoDataAutoConfiguration.class) @Import(MongrationConfiguration.class) public class MongrationAutoConfiguration { } 

But the problem with auto configuration is that if you use the @EnableMongoRepositories annotation to activate repositories, Mongo Data components are initialized before our auto configuration. To avoid this problem, you must use the @EnableMongration annotation with @EnableMongoRepositories :

 @Configuration @EnableMongoRepositories @EnableMongration public class MongoRepositoriesConfiguration { } 

The @EnableMongration does nothing more than launching the same configuration, only allows you to start it earlier:

 @Target(ElementType.TYPE) @Retention(RetentionPolicy.RUNTIME) @Import(MongrationConfiguration.class) public @interface EnableMongration { } 

Transaction support

To support transactions, you need to configure Replica Set MongoDB. You must also declare a MongoTransationManager bin (if Mongration does not find this bin in context, then it will create it yourself). Mongration allows you to use transactions in two ways:

• Using @Transactional
  

   @Transactional @ChangeSet(order = 1, id = "change1", author = "Stepan") public void changeset(MongoTemplate template) { template.findAll(Document.class, "testDocument").forEach(document -> { document.put("text2", UUID.randomUUID().toString()); template.save(document, "testDocument"); }); } 

• Using TransactionTemplate
  

   @ChangeSet(order = 1, id = "change1", author = "Stepan") public void migration(MongoTemplate template, TransactionTemplate txTemplate) { template.createCollection("entity"); txTemplate.execute(new TransactionCallbackWithoutResult() { @Override protected void doInTransactionWithoutResult(TransactionStatus status) { template.save(new Document("index", "1").append("text", "1"), "entity"); template.save(new Document("index", "2").append("text", "2"), "entity"); template.save(new Document("index", "3").append("text", "3"), "entity"); } }); } 

  
  

The second method is useful in that it allows using both DDL operations that cannot be started in a transaction, and DML operations launched in a transaction in a single migration.

Embedding dependencies in ChangeLog classes

This functionality is possible due to the fact that each ChangeLog class itself is a regular bean:

 @Target(ElementType.TYPE) @Retention(RetentionPolicy.RUNTIME) @Component public @interface ChangeLog { } 

It is worth noting that you cannot inject bins that have dependencies on Mongo Data components, because they are not yet initialized at the time of the migrations.

The source code for the library is available on Github .