Developing microservices using Scala, Spray, MongoDB, Docker and Ansible
The purpose of this article is to show a possible approach for building microservices using Scala , RESTful JSON, Spray and Akka . We will use MongoDB as the database. As a result of our work, we will pack our project in a Docker container, and Vagrant and Ansible will allow us to manage the application configuration.
In this article you will not find details about the Scala language and other technologies that will be used in the project. In it, you will not find a manual that will answer all your questions. The purpose of the article is to show a technique that can be used in the development of microservices. In fact, most of this article is not tied to a specific technology. Docker has a wider scope of use than microservices. Ansible allows you to quickly deploy any desired environment, and Vagrant is a great tool for creating virtual machines.
So, let's start creating the “Book Service” with the following methods:
We will use Ubuntu as a server. The easiest way to create one is to use Vagrant. If you have not installed it yet, please download it and install it. You will also need Git to clone the repository with the source code. The rest of the article does not require you to manually install additional packages.
')
Let's start repository cloning
Next, you need to create a Ubuntu server using Vagrant with the following settings:
We defined the box (OS) as Ubuntu, synced_folder means that everything inside the ./vagrant directory on the host machine will be available inside the virtual machine. The rest of the work on installing applications and preparing the environment we will impose on Ansible, which will be installed using bootstrap.sh. There are two virtual machines in Vagrantfile: dev and prod. Each of them will use Ansible, so make sure it is installed correctly.
The classic way to work with Ansible is to divide the configuration into roles. In our case, we have 4 roles that are in ansible / roles. The first role includes the installation of Scala and SBT. One more installs docker. The third installs the MongoDB container. The last role (books) will be used to deploy the application to the combat virtual machine.
As an example, declare a mongodb role as follows:
This role ensures that the folder exists and the mongodb container is running. The ansible / dev.yml playbook links these roles together:
Every time we start a playbook, all tasks from the scala, docker and mongodb roles are executed.
One of the charms of Ansible is that it performs tasks only when needed. If you run it a second time, he will check that everything is in place and will not do anything. However, if you delete the / data / db folder, Ansible will notice the loss and create it again.
Time to run a virtual machine! The first launch will be a bit long since Vagrant needs to download the Ubuntu distribution, install the packages and download the MongoDB Docker image. Subsequent launches will be noticeably faster.
The vagrant up command creates a new virtual machine or starts one of the existing ones. With vagrant ssh, we go to the newly created machine. Finally, ll / vagrant displays a list of files and directories as proof that our local files are available inside the virtual machine.
It's all. Our development server with Scala, SBT and MongoDB is ready for work. Let's start developing our service.
I like Scala, it is a very powerful language, and akka is my favorite framework for building message-driven JVM applications. Despite the fact that Akka came from Scala, nothing prevents to use it in Java.
Spray is a simple but powerful tool for building REST / HTTP services. It is asynchronous due to the use of Akka-actors and has a wonderful DSL for describing HTTP routes.
In TDD fashion, we write tests before implementation. Here is an example of tests that checks the route on which the list of books is given.
This is a simple test, which, I hope, will show the direction in which you need to move in order to develop tests for APIs built on Spray. The first thing we check is the server, it should return the code 200 (OK) for this request. The second thing we pay attention to is that after adding a book to the database, it returns correctly. You can view the full source code with tests in ServiceSpec.scala
How are these checks implemented? The code that allows this is shown below:
We defined the route / api / v1 / books, the GET method and the answer inside the complete () statement. In our case, we received a list of all books from the database and converted it to the BookReduced case class. All source code including all methods (GET, PUT, DELETE) can be found in ServiceActor.scala
Both tests and implementation are given as an example, in practice they are usually more difficult. But Spray copes with this really great.
During development, you can run tests in quick mode.
Each time the source code changes, all affected tests will be run again. Usually, I have a terminal window open with test results; this gives me the opportunity to receive instant feedback on the quality of the code I'm working on.
Our application, like any other, needs testing, building and deploying.
Let's create a Docker container with a service. You can specify the required settings in the Dockerfile.
We compiled the JAR (passing the tests is part of the build phase), collected the Docker container and sent it to the Docker Hub. If you plan to reproduce these steps again, please create an account at hub.docker.com and change “vfarcic” to your login.
The container that we created contains everything you need to run our service. It is based on Ubuntu, contains a JDK7 with MongoDB and a compiled JAR file. This container can be run on any machine with Docker installed. It does not require installation of additional dependencies on the server, the container is self-sufficient and can be run anywhere.
Let's deploy (run) the container that we created on another virtual machine. This is very similar to deploying an application in a production-based environment.
To create a production-virtual machine, with our service you need to run the following commands:
The first command stops the dev-virtual machine. Each machine requires 2GB of RAM, and if you have enough free RAM, you can skip this step. The second command starts the production-machine, with the expanded service.
After some waiting time, the virtual machine will be created, Ansible will install and start the playbook prod.yml. It will install Docker and launch the vfarcic / books-service compiled in the previous step and sent to the Docker Hub. During operation, it will use port 8080 and have a shared / data / db folder with the host system.
Let's try what we did. First, let's try to send a PUT request to add test data:
Let's check that the service returned us the correct data:
We can delete a book:
Verify that the deleted book no longer exists:
In conclusion, we will try to extract a specific copy of the book:
We tried a quick way to develop, build and deploy microservice. Docker simplifies deployment and does not require additional dependencies. Each service requiring JDK and MongoDB does not require installed applications on the target machine. This is all part of a container that is running as a Docker process.
The idea of ​​microservices has existed for a long time, and until recently has not received due attention, due to application compatibility problems required for the simultaneous operation of hundreds and thousands of different instances of services. The advantages that arose through the use of microservices (separation, reduced development time, scalability, etc.) were not as significant as compared with the problems that they had while providing the right environment and deployment. Docker and tools such as Ansible help significantly reduce effort. With the avoidance of this problem, the most diverse microservices become in vogue because of the advantages that they carry.
Spray is an excellent choice for microservice. Docker-containers contain everything that is needed for the application to work, and nothing superfluous. Using large web servers such as JBoss and WebSphere may be unreasonable for a small service. Even small servers like Tomcat are usually not necessary. Play! - This is a great framework for building the RESTFul API, but it contains many things that we do not use. Spray, on the other hand, does only one thing — it provides asynchronous routing for the RESTFul API, and it does it perfectly.
We can continue to expand the functionality of the service. For example, we can add a registration and authentication module.
However, this will bring us one step closer to a monolithic application. In the world of microservices, new services should be new applications, and in the case of Docker, separate containers, each of which listens to its port and responds to requests addressed to it.
When building microservices, you need to try to create them in such a way that they only do one or a small amount of work. Complexity is solved by combining them together, rather than building a large monolithic application.
In this article you will not find details about the Scala language and other technologies that will be used in the project. In it, you will not find a manual that will answer all your questions. The purpose of the article is to show a technique that can be used in the development of microservices. In fact, most of this article is not tied to a specific technology. Docker has a wider scope of use than microservices. Ansible allows you to quickly deploy any desired environment, and Vagrant is a great tool for creating virtual machines.
So, let's start creating the “Book Service” with the following methods:
- Get all the books
- Get information about the book
- Update existing book
- Delete existing book
Environment
We will use Ubuntu as a server. The easiest way to create one is to use Vagrant. If you have not installed it yet, please download it and install it. You will also need Git to clone the repository with the source code. The rest of the article does not require you to manually install additional packages.
')
Let's start repository cloning
git clone https://github.com/vfarcic/books-service.git cd books-service Next, you need to create a Ubuntu server using Vagrant with the following settings:
# -*- mode: ruby -*- # vi: set ft=ruby : VAGRANTFILE_API_VERSION = "2" Vagrant.configure(VAGRANTFILE_API_VERSION) do |config| config.vm.box = "ubuntu/trusty64" config.vm.synced_folder ".", "/vagrant" config.vm.provision "shell", path: "bootstrap.sh" config.vm.provider "virtualbox" do |v| v.memory = 2048 end config.vm.define :dev do |dev| dev.vm.provision :shell, inline: 'ansible-playbook /vagrant/ansible/dev.yml -c local' end config.vm.define :prod do |prod| prod.vm.provision :shell, inline: 'ansible-playbook /vagrant/ansible/prod.yml -c local' end end
We defined the box (OS) as Ubuntu, synced_folder means that everything inside the ./vagrant directory on the host machine will be available inside the virtual machine. The rest of the work on installing applications and preparing the environment we will impose on Ansible, which will be installed using bootstrap.sh. There are two virtual machines in Vagrantfile: dev and prod. Each of them will use Ansible, so make sure it is installed correctly.
The classic way to work with Ansible is to divide the configuration into roles. In our case, we have 4 roles that are in ansible / roles. The first role includes the installation of Scala and SBT. One more installs docker. The third installs the MongoDB container. The last role (books) will be used to deploy the application to the combat virtual machine.
As an example, declare a mongodb role as follows:
- name: Directory is present file: path=/data/db state=directory tags: [mongodb] - name: Container is running docker: name=mongodb image=dockerfile/mongodb ports=27017:27017 volumes=/data/db:/data/db tags: [mongodb]
This role ensures that the folder exists and the mongodb container is running. The ansible / dev.yml playbook links these roles together:
- hosts: localhost remote_user: vagrant sudo: yes roles: - scala - docker - mongodb
Every time we start a playbook, all tasks from the scala, docker and mongodb roles are executed.
One of the charms of Ansible is that it performs tasks only when needed. If you run it a second time, he will check that everything is in place and will not do anything. However, if you delete the / data / db folder, Ansible will notice the loss and create it again.
Time to run a virtual machine! The first launch will be a bit long since Vagrant needs to download the Ubuntu distribution, install the packages and download the MongoDB Docker image. Subsequent launches will be noticeably faster.
vagrant up dev vagrant ssh dev ll /vagrant
The vagrant up command creates a new virtual machine or starts one of the existing ones. With vagrant ssh, we go to the newly created machine. Finally, ll / vagrant displays a list of files and directories as proof that our local files are available inside the virtual machine.
It's all. Our development server with Scala, SBT and MongoDB is ready for work. Let's start developing our service.
"Book Service"
I like Scala, it is a very powerful language, and akka is my favorite framework for building message-driven JVM applications. Despite the fact that Akka came from Scala, nothing prevents to use it in Java.
Spray is a simple but powerful tool for building REST / HTTP services. It is asynchronous due to the use of Akka-actors and has a wonderful DSL for describing HTTP routes.
In TDD fashion, we write tests before implementation. Here is an example of tests that checks the route on which the list of books is given.
"GET /api/v1/books" should { "return OK" in { Get("/api/v1/books") ~> route ~> check { response.status must equalTo(OK) } } "return all books" in { val expected = insertBooks(3).map { book => BookReduced(book._id, book.title, book.author) } Get("/api/v1/books") ~> route ~> check { response.entity must not equalTo None val books = responseAs[List[BookReduced]] books must haveSize(expected.size) books must equalTo(expected) } } }
This is a simple test, which, I hope, will show the direction in which you need to move in order to develop tests for APIs built on Spray. The first thing we check is the server, it should return the code 200 (OK) for this request. The second thing we pay attention to is that after adding a book to the database, it returns correctly. You can view the full source code with tests in ServiceSpec.scala
How are these checks implemented? The code that allows this is shown below:
val route = pathPrefix("api" / "v1" / "books") { get { complete( collection.find().toList.map(grater[BookReduced].asObject(_)) ) } }
We defined the route / api / v1 / books, the GET method and the answer inside the complete () statement. In our case, we received a list of all books from the database and converted it to the BookReduced case class. All source code including all methods (GET, PUT, DELETE) can be found in ServiceActor.scala
Both tests and implementation are given as an example, in practice they are usually more difficult. But Spray copes with this really great.
During development, you can run tests in quick mode.
#[ ] cd /vagrant sbt ~test-quick
Each time the source code changes, all affected tests will be run again. Usually, I have a terminal window open with test results; this gives me the opportunity to receive instant feedback on the quality of the code I'm working on.
Testing, building and deploying
Our application, like any other, needs testing, building and deploying.
Let's create a Docker container with a service. You can specify the required settings in the Dockerfile.
#[ ] cd /vagrant sbt assembly sudo docker build -t vfarcic/books-service . sudo docker push vfarcic/books-service
We compiled the JAR (passing the tests is part of the build phase), collected the Docker container and sent it to the Docker Hub. If you plan to reproduce these steps again, please create an account at hub.docker.com and change “vfarcic” to your login.
The container that we created contains everything you need to run our service. It is based on Ubuntu, contains a JDK7 with MongoDB and a compiled JAR file. This container can be run on any machine with Docker installed. It does not require installation of additional dependencies on the server, the container is self-sufficient and can be run anywhere.
Let's deploy (run) the container that we created on another virtual machine. This is very similar to deploying an application in a production-based environment.
To create a production-virtual machine, with our service you need to run the following commands:
#[ ] vagrant halt dev vagrant up prod
The first command stops the dev-virtual machine. Each machine requires 2GB of RAM, and if you have enough free RAM, you can skip this step. The second command starts the production-machine, with the expanded service.
After some waiting time, the virtual machine will be created, Ansible will install and start the playbook prod.yml. It will install Docker and launch the vfarcic / books-service compiled in the previous step and sent to the Docker Hub. During operation, it will use port 8080 and have a shared / data / db folder with the host system.
Let's try what we did. First, let's try to send a PUT request to add test data:
curl -H 'Content-Type: application/json' -X PUT -d '{"_id": 1, "title": "My First Book", "author": "John Doe", "description": "Not a very good book"}' http://localhost:8080/api/v1/books curl -H 'Content-Type: application/json' -X PUT -d '{"_id": 2, "title": "My Second Book", "author": "John Doe", "description": "Not a bad as the first book"}' http://localhost:8080/api/v1/books curl -H 'Content-Type: application/json' -X PUT -d '{"_id": 3, "title": "My Third Book", "author": "John Doe", "description": "Failed writers club"}' http://localhost:8080/api/v1/books
Let's check that the service returned us the correct data:
curl -H 'Content-Type: application/json' http://localhost:8080/api/v1/books
We can delete a book:
curl -H 'Content-Type: application/json' -X DELETE http://localhost:8080/api/v1/books/_id/3
Verify that the deleted book no longer exists:
curl -H 'Content-Type: application/json' http://localhost:8080/api/v1/books
In conclusion, we will try to extract a specific copy of the book:
curl -H 'Content-Type: application/json' http://localhost:8080/api/v1/books/_id/1
We tried a quick way to develop, build and deploy microservice. Docker simplifies deployment and does not require additional dependencies. Each service requiring JDK and MongoDB does not require installed applications on the target machine. This is all part of a container that is running as a Docker process.
Results
The idea of ​​microservices has existed for a long time, and until recently has not received due attention, due to application compatibility problems required for the simultaneous operation of hundreds and thousands of different instances of services. The advantages that arose through the use of microservices (separation, reduced development time, scalability, etc.) were not as significant as compared with the problems that they had while providing the right environment and deployment. Docker and tools such as Ansible help significantly reduce effort. With the avoidance of this problem, the most diverse microservices become in vogue because of the advantages that they carry.
Spray is an excellent choice for microservice. Docker-containers contain everything that is needed for the application to work, and nothing superfluous. Using large web servers such as JBoss and WebSphere may be unreasonable for a small service. Even small servers like Tomcat are usually not necessary. Play! - This is a great framework for building the RESTFul API, but it contains many things that we do not use. Spray, on the other hand, does only one thing — it provides asynchronous routing for the RESTFul API, and it does it perfectly.
We can continue to expand the functionality of the service. For example, we can add a registration and authentication module.
However, this will bring us one step closer to a monolithic application. In the world of microservices, new services should be new applications, and in the case of Docker, separate containers, each of which listens to its port and responds to requests addressed to it.
When building microservices, you need to try to create them in such a way that they only do one or a small amount of work. Complexity is solved by combining them together, rather than building a large monolithic application.
Source: https://habr.com/ru/post/250043/
All Articles