The girl on the scooter. Illustration of freepik , Nomad logo by HashiCorp

Kubernetes is a 300-pound gorilla for orchestrating containers. It works in some of the largest container systems in the world, but is expensive.

Especially expensive for small teams that have to spend a lot of time on support and a steep learning curve. For our team of four, this is too much overhead. Therefore, we began to look for alternatives - and fell in love with Nomad .

What do you want

Our team supports a number of typical services for monitoring and analyzing performance: API endpoints for metrics written in Go, Prometheus export, log parsers such as Logstash and Gollum , as well as databases such as InfluxDB or Elasticsearch. Each of these services runs in its own container. We need a simple system to keep it all working.
')
We started with a list of requirements for container orchestration:

• Running a set of services on many machines.
  
• Browse running services.
  
• Communication between services.
  
• Automatic restart if the service crashes.
  
• Infrastructure maintenance by a small team.

In addition, the following things will be pleasant, but not mandatory additions:

• Tagging machines by their capabilities (for example, tagging machines with fast disks for heavy I / O services).
  
• The ability to run services independently of the orchestrator (for example, during development).
  
• Common place for sharing configurations and secrets.
  
• End point for metrics and logs.

Why Kubernetes does not suit us

When creating a prototype with Kubernetes, we noticed that we began to add more and more complex layers of logic, which we relied on unconditionally.

As an example, Kubernetes supports embedded service configurations via ConfigMaps . You can quickly get confused, especially when you merge several configuration files or add additional services to the pod. Kubernetes (or helm in this case) allows you to implement dynamically external configurations for the separation of interests. But this leads to a hard hidden connection between your project and Kubernetes. However, Helm and ConfigMaps are additional options, so you do not need to use them. You can simply copy the configuration to the Docker image. Nevertheless, it is tempting to go this way and build unnecessary abstractions, which you may regret later.

In addition, the Kubernetes ecosystem is developing rapidly. It takes a lot of time and energy to stay up to date with best practices and the latest tools. Kubectl, minikube, kubeadm, helm, tiller, kops, oc - the list goes on and on. At the beginning of work, not all of these tools are needed, but you do not know what you need, so you need to be aware of everything. Because of this, the learning curve is pretty steep.

When to use Kubernetes

In our company, many use Kubernetes and are quite satisfied with this. These instances are managed by Google or Amazon, which have enough support resources.

Kubernetes comes with amazing features that make large-scale container orchestration more manageable:

• Detailed rights management .
  
• Custom controllers add logic to the cluster. These are just programs that talk to the Kubernetes API.
  
• Autoscale ! Kubernetes is able to scale services on demand, using service metrics and not requiring manual intervention.

The question is, do you really need all these features? It is impossible to rely only on abstractions; you have to find out what's going on under the hood .

Our team provides most of the services remotely (due to the close connection with the main infrastructure), so we did not want to raise our own cluster of Kubernetes. We just wanted to provide services.

Batteries not included

Nomad is 20% orchestration, which gives 80% of what is needed. All he does is manage the deployments. Nomad takes care of deployments, restarts containers in case of errors ... and that's it.

The whole point of Nomad is that it does the least : no detailed rights management or advanced network policies , so specifically conceived. These components are either provided by or not provided at all.

I think that Nomad has found the perfect compromise between ease of use and utility. It is good for small, independent services. If you need more control, you will have to raise them yourself or use a different approach. Nomad is just an orchestra.

The best thing about Nomad is that it is easy to replace . Binding to the vendor is practically absent, since its functions are easily integrated into any other system that manages the services. It works just like a regular binary on every machine in a cluster, that's all!

Loosely connected Nomad ecosystem

The real power of Nomad in its ecosystem. It integrates very well with other - completely optional - products, such as Consul (key-value storage) or Vault (secret processing). Inside the Nomad file there are sections for extracting data from these services:

template { data = <<EOH LOG_LEVEL="{{key "service/geo-api/log-verbosity"}}" API_KEY="{{with secret "secret/geo-api-key"}}{{.Data.value}}{{end}}" EOH destination = "secrets/file.env" env = true } 

Here we read the service/geo-api/log-verbosity from Consul and in the process of work we present it as the environment variable LOG_LEVEL . We also present the secret/geo-api-key from Vault as API_KEY . Simple but powerful!

Due to its simplicity, Nomad is easily expanded using other services through the API. For example, job tags are supported. We mark all services with metrics with the trv-metrics tag. Thus, Prometheus easily finds these services through Consul and periodically checks the end point /metrics for new data. The same can be done, for example, for logs using Loki .

There are many other examples of extensibility:

• Run a Jenkins job using a hook, and Consul tracks a repeat deployment of a Nomad job when a service’s configuration changes.
  
• Ceph adds a distributed file system to Nomad.
  
• fabio for load balancing.

All this allows you to organically develop the infrastructure without special reference to the vendor.

Honest warning

No system is perfect. I do not advise you to immediately implement the newest functions in production. Of course, there are errors and missing functions, but the same applies to Kubernetes.

Compared to Kubernetes, the Nomad community is not that big. Kubernetes already has about 75,000 commits and 2,000 contributors, while Nomad has about 14,000 commits and 300 contributors. Nomad will be hard to keep up with and keep up with Kubernetes in speed, but maybe this is not necessary! This is a more specialized system, and a smaller community also means that your pull-request is more likely to be noticed and accepted than Kubernetes.

Summary

Conclusion: do not use Kubernetes just because everyone does it. Carefully evaluate your requirements and check which tool is more profitable.

If you plan to deploy a mass of homogeneous services on a large-scale infrastructure, then Kubernetes is a good option. Just remember the added complexity and running costs. Some costs can be avoided using a managed Kubernetes environment such as Google Kubernetes Engine or Amazon EKS .

If you're just looking for a reliable orchestrator, simple to support and expandable, why not try Nomad? You might be surprised how far it takes you.

If Kubernetes compare with the machine, the Nomad will be a scooter. Sometimes you need one thing, and sometimes another. Both have a right to exist.