As I wrote a Docker container launch audit on Go
Universal containerization captures the world. This epidemic has not bypassed me either, and now, for the last six months, I have been doing what is now called the buzzword DevOps. We decided to use Docker in projects that I do, because it makes the process of deploying applications obscenely simple, and literally forces you to follow another trend that is no less fashionable today - microservice architecture, which promotes the rapid reproduction of these containers based on it. At some point, you realize that it would be nice to collect statistics on their life and death in an unsafe environment. And as a bonus, explore the tools that you use in your work, write something not in the main programming language, and just do something optional, but useful.
In the article I will tell you how for three evenings and a piece of night a project was developed for auditing and collecting statistics on the life cycle of containers.
A quick search in Google did not lead to finding a ready-made solution, so we will do it ourselves.
What do you need:
The first task is solved by the registrator . This is a solution from the guys from GliderLabs, which allows you to automatically register containers in configuration storage systems, such as onsul or Netflix Eurika. Unfortunately, the latter are sharpened for a completely different task: to say what services are now available, and where are the containers that implement them.
If we consider each event (the launch or death of the container) as a record of a certain log, with which we can do everything we need, then to store these records you can take ElasticSearch, and for viewing and analysis in real time - Kibana.
')
It remains for us to decide the second point, namely, to make a link between the registrar and the elastic.
Any entertainment begins with a fork, so feel free to click a button on the GitHub for the repository (https://github.com/gliderlabs/registrator). We clone to our local machine and look at the contents:
The scheme is simple. In registrator.go, a Docker client is created that listens to the socket, and, if any event occurs (start, stop or death of the container), sends the container identifier and the event associated with it to bridge. Inside the bridge, an adapter (module) is created, which was specified when the application was started, to which detailed information about the container is already being transmitted for further processing. Thus, it is enough to add a new module that will send data to ElasticSearch.
Before writing the code, let's try building and running a project. In the Makefile there is a task, in which a new Docker image is created and launched:
The consul hints to us that this is the default master system, without which the application will not work. Put it in the Docker container in standalone mode:
Then run the registrar assembly:
If everything went well (unfortunately, luck is such a thing), then we will see something like this:
As you can see, we had 4 containers. One of them had no ports, the other had no port 443 published, and so on. To check that the services have actually been added, you can use the dig utility.
It is necessary to add -80 to the name of the container, since nginx exposes several ports to the outside, and from the point of view of Consul, these are different services.
So, we launched the recorder, which means that it's time to start writing code.
Adapters in the project for different backends are implemented as separate modules. In general, Go module is a very interesting thing. This can be either a local folder or a project on GitHub, there is almost no difference in the connection.
Add a new folder to the project root: / elastic and place the file with our future implementation in it: elastic.go.
Give the default name for our module
We import the third-party packages we need:
To handle events, you need to implement an interface
The adapter is registered through the init () method, which is executed when the module is loaded:
When creating an adapter, you must create an instance of the client to ElasticSearch:
Use the isRunning () method to verify that the instance is still alive.
Let the container record have the following structure:
Implement container registration method:
We dump all the service information in json.
We get the current time. Go uses funny notation to determine the date format.
Create a new entry for the log:
And send it to a specially created index
The Deregister function completely repeats the previous one, only with a different action.
All that remains is to change the consul on elastic to the Makefile, and set the module in modules.go.
Launch ElasticSearch
In order for Kibana to work correctly with the index, you need to add a slightly redesigned template from logstash:
Launch Kibana
We start the recorder:
We start the container with nginx for testing the solution
In Kibana, you need to configure a new index of containers, after which you can see a record of running nginx.
The final implementation file is here .
Everyone is good at our solution, but for his work we need to keep a separate self-written index, and still remember to roll the correct template with mapping. So that people do not bother with such questions, there are log aggregators, who not only know how to collect information from a huge number of sources, but also do all the dirty work for us in terms of bringing logs to a single format. We will take logstash for our experiments.
By tradition, we want to run logstash in a container. The official Docker image for logstash comes without source files, which in my opinion is somewhat strange ( as the careful reader noted grossws , the link to the Dockerfile is still present ). The second most popular and the only, by the way, image found on github-e for some reason launches within itself ElasticSearch and Kibana, which contradicts the idea of ​​“one container - one process”. There, of course, there is an opportunity to send in a magic combination of flags, but he still climbed at me to take some keys from the author’s site. On DockerHub, there were a dozen more containers from unknown persons, so it’s best to assemble the container ourselves for our needs. All we need is such a Dockerfile:
The image will be very simple and will start only if there is an external configuration file, which is quite normal for our entertainment tasks. Collect the image and fill it with the Docker Hub:
Create the configuration file /mnt/logstash/conf/logstash.conf with the following contents:
type => “audit” will make all our logs have a common value in the type field, which will allow us to distinguish them from other logs by this discriminator. The remaining settings are pretty obvious. Run the newly baked container:
and check that the logs will be written if we send json via tcp.
We are already doing the second module, so it is worth putting the implementation into a separate project, which we call auditor . First of all, we need to wind the already existing "meat" from the registrar. Therefore, we take our fork and brazenly copy the code into our project.
We check that everything is still going on by running the command: make dev.
We note that in the regitrator.go file, the bridge module is connected as an external dependency, so you can safely delete this folder. Check again that everything works.
Modify Dockerfile.dev:
Similarly, we are changing the release Dockefile. We remove the extra task and change the name of the container in the Makefile:
Add a new module / logstash and a file logstash.go to our project. Take a ready-made client for logstash, which is dumb as a cork, and in fact is just a wrapper over the standard net library: github.com/heatxsink/go-logstash .
This time the structure of the container will differ slightly from the previous version:
This is due to the fact that now we just need to serialize the object in json and send it as a string in logstash, which itself will deal with all the fields in the message.
Just like last time we register our factory:
And create a new instance of the adapter:
Here we had to use the net.SplitHostPort (urls) utilization method, which can isolate the host and port from the string, because the client accepts them separately, and they come together at uri.Host.
The numeric representation of the port can be obtained using the method of converting a string to a number: intPort, _: = strconv.Atoi (port). The underscore is needed because the function returns two parameters, the second of which is an error, which we can not handle.
The implementation of the Ping method is pretty simple:
In fact, we check that we can connect via tcp to logstash. In the Connect function, reconnection will occur only if the current one can no longer be used.
It remains to implement the registration method:
I think that the code is clear enough and does not require comments, except for one. Calling Connect before Writeln ensures that a working connection is received.
The Deregister Method is a complete copy of the method above.
We change the dockerfile.dev in the elastic launch line to logstash, run and check for the presence of records in ElasticSearch:
Let's commit our changes to GitHub and go to build an image for DockerHub. On hub.docker.com , go to your page and click the button + Add Repository. When I was going to create an image for logstash, I chose the Repository sub-item, which allows you to manually upload your images, but there is another way - Automated Build. By clicking on it, Docker Hub will offer to connect your account to it on GitHub or BitBucket. After that, it remains only to choose your repository, the desired branch, and change the image names, if it is very necessary. Everything else, including the transfer of the description from README.MD will take over the Docker Hub.
After a short wait, here it is - the finished image .
Now you can test it by executing a simple command:
Ps. The project is not used in production, and from my critical point of view it requires finishing, but everyone who reads an article can try it and, if desired, improve it.
In the article I will tell you how for three evenings and a piece of night a project was developed for auditing and collecting statistics on the life cycle of containers.
First half
A quick search in Google did not lead to finding a ready-made solution, so we will do it ourselves.
What do you need:
- monitoring the start and stop of an individual container taken
- sending event messages to some storage
- convenient tool for viewing events and their subsequent analysis
The first task is solved by the registrator . This is a solution from the guys from GliderLabs, which allows you to automatically register containers in configuration storage systems, such as onsul or Netflix Eurika. Unfortunately, the latter are sharpened for a completely different task: to say what services are now available, and where are the containers that implement them.
If we consider each event (the launch or death of the container) as a record of a certain log, with which we can do everything we need, then to store these records you can take ElasticSearch, and for viewing and analysis in real time - Kibana.
')
It remains for us to decide the second point, namely, to make a link between the registrar and the elastic.
How is the recorder
Any entertainment begins with a fork, so feel free to click a button on the GitHub for the repository (https://github.com/gliderlabs/registrator). We clone to our local machine and look at the contents:
registrator.go // modules.go // (consul, etcd ..) Dockerfile // docker- Dockerfile.dev // dev- /bridge // /consul // consul The scheme is simple. In registrator.go, a Docker client is created that listens to the socket, and, if any event occurs (start, stop or death of the container), sends the container identifier and the event associated with it to bridge. Inside the bridge, an adapter (module) is created, which was specified when the application was started, to which detailed information about the container is already being transmitted for further processing. Thus, it is enough to add a new module that will send data to ElasticSearch.
make dev
Before writing the code, let's try building and running a project. In the Makefile there is a task, in which a new Docker image is created and launched:
dev: docker build -f Dockerfile.dev -t $(NAME):dev . docker run --rm --net host \ -v /var/run/docker.sock:/tmp/docker.sock \ $(NAME):dev /bin/registrator consul: The consul hints to us that this is the default master system, without which the application will not work. Put it in the Docker container in standalone mode:
$ docker run -p 8400:8400 -p 8500:8500 -p 53:53/udp \ -h node1 progrium/consul -server -bootstrap Then run the registrar assembly:
make dev If everything went well (unfortunately, luck is such a thing), then we will see something like this:
2015/04/04 19:55:48 Starting registrator dev ... 2015/04/04 19:55:48 Using elastic adapter: consul:// 2015/04/04 19:55:48 Listening for Docker events ... 2015/04/04 19:55:48 Syncing services on 4 containers 2015/04/04 19:55:48 ignored: cedfd1ae9f68 no published ports 2015/04/04 19:55:48 added: b4455d0f7d50 ubuntu:kibana:80 2015/04/04 19:55:48 added: 3d598d184eb6 ubuntu:nginx:80 2015/04/04 19:55:48 ignored: 3d598d184eb6 port 443 not published on host 2015/04/04 19:55:48 added: bcad15ac5759 ubuntu:determined_goldstine:9200 2015/04/04 19:55:48 added: bcad15ac5759 ubuntu:determined_goldstine:9300 As you can see, we had 4 containers. One of them had no ports, the other had no port 443 published, and so on. To check that the services have actually been added, you can use the dig utility.
dig @localhost nginx-80.service.consul It is necessary to add -80 to the name of the container, since nginx exposes several ports to the outside, and from the point of view of Consul, these are different services.
So, we launched the recorder, which means that it's time to start writing code.
Go go go
Adapters in the project for different backends are implemented as separate modules. In general, Go module is a very interesting thing. This can be either a local folder or a project on GitHub, there is almost no difference in the connection.
Add a new folder to the project root: / elastic and place the file with our future implementation in it: elastic.go.
Give the default name for our module
package elastic We import the third-party packages we need:
import ( "net/url" "errors" "encoding/json" "time" "github.com/gliderlabs/registrator/bridge" elasticapi "github.com/olivere/elastic" ) To handle events, you need to implement an interface
type RegistryAdapter interface { Ping() error // Register(service *Service) error Deregister(service *Service) error Refresh(service *Service) error // :) } The adapter is registered through the init () method, which is executed when the module is loaded:
func init() { bridge.Register(new(Factory), "elastic") } When creating an adapter, you must create an instance of the client to ElasticSearch:
func (f *Factory) New(uri *url.URL) bridge.RegistryAdapter { urls := "http://127.0.0.1:9200" if uri.Host != "" { urls = "http://"+uri.Host } client, err := elasticapi.NewClient(elasticapi.SetURL(urls)) if err != nil { log.Fatal("elastic: ", uri.Scheme) } return &ElasticAdapter{client: client} } type ElasticAdapter struct { client *elasticapi.Client } Use the isRunning () method to verify that the instance is still alive.
func (r *ElasticAdapter) Ping() error { status := r.client.IsRunning() if !status { return errors.New("client is not Running") } return nil } Let the container record have the following structure:
type Container struct { Name string `json:"container_name"` Action string `json:"action"` //start and stop Message string `json:"message"` Timestamp string `json:"@timestamp"` } Implement container registration method:
func (r *ElasticAdapter) Register(service *bridge.Service) error We dump all the service information in json.
serviceAsJson, err := json.Marshal(service) if err != nil { return err } We get the current time. Go uses funny notation to determine the date format.
timestamp := time.Now().Local().Format("2006-01-02T15:04:05.000Z07:00") Create a new entry for the log:
container := Container { Name: service.Name, Action: "start", Message: string(serviceAsJson), Timestamp: timestamp } And send it to a specially created index
_, err = r.client.Index(). Index("containers"). Type("audit"). BodyJson(container). Timestamp(timestamp). Do() if err != nil { return err } The Deregister function completely repeats the previous one, only with a different action.
All that remains is to change the consul on elastic to the Makefile, and set the module in modules.go.
All together now
Launch ElasticSearch
docker run -d --name elastic -p 9200:9200 \ -p 9300:9300 dockerfile/elasticsearch In order for Kibana to work correctly with the index, you need to add a slightly redesigned template from logstash:
{ "template" : "containers*", "settings" : { "index.refresh_interval" : "5s" }, "mappings" : { "_default_" : { "_all" : {"enabled" : true}, "dynamic_templates" : [ { "string_fields" : { "match" : "*", "match_mapping_type" : "string", "mapping" : { "type" : "string", "index" : "analyzed", "omit_norms" : true, "fields" : { "raw" : {"type": "string", "index" : "not_analyzed", "ignore_above" : 256} } } } } ], "_ttl": { "enabled": true, "default": "1d" }, "properties" : { "@version": { "type": "string", "index": "not_analyzed" }, "geoip" : { "type" : "object", "dynamic": true, "path": "full", "properties" : { "location" : { "type" : "geo_point" } } } } } } } Launch Kibana
docker run -d -p 8080:80 -e KIBANA_SECURE=false \ --name kibana --link elastic:es \ balsamiq/docker-kibana We start the recorder:
make dev We start the container with nginx for testing the solution
docker run -d --name nginx -p 80:80 nginx In Kibana, you need to configure a new index of containers, after which you can see a record of running nginx.
The final implementation file is here .
Logstash bursts into bar
Everyone is good at our solution, but for his work we need to keep a separate self-written index, and still remember to roll the correct template with mapping. So that people do not bother with such questions, there are log aggregators, who not only know how to collect information from a huge number of sources, but also do all the dirty work for us in terms of bringing logs to a single format. We will take logstash for our experiments.
By tradition, we want to run logstash in a container. The official Docker image for logstash comes without source files, which in my opinion is somewhat strange ( as the careful reader noted grossws , the link to the Dockerfile is still present ). The second most popular and the only, by the way, image found on github-e for some reason launches within itself ElasticSearch and Kibana, which contradicts the idea of ​​“one container - one process”. There, of course, there is an opportunity to send in a magic combination of flags, but he still climbed at me to take some keys from the author’s site. On DockerHub, there were a dozen more containers from unknown persons, so it’s best to assemble the container ourselves for our needs. All we need is such a Dockerfile:
FROM dockerfile/java:oracle-java8 MAINTAINER aatarasoff@gmail.com RUN echo 'deb http://packages.elasticsearch.org/logstash/1.5/debian stable main' | sudo tee /etc/apt/sources.list.d/logstash.list && \ apt-get -y update && \ apt-get -y --force-yes install logstash EXPOSE 5959 VOLUME ["/opt/conf", "/opt/certs", "/opt/logs"] ENTRYPOINT exec /opt/logstash/bin/logstash agent -f /opt/conf/logstash.conf The image will be very simple and will start only if there is an external configuration file, which is quite normal for our entertainment tasks. Collect the image and fill it with the Docker Hub:
docker build -t aatarasoff/logstash . docker push aatarasoff/logstash Create the configuration file /mnt/logstash/conf/logstash.conf with the following contents:
input { tcp { type => "audit" port => 5959 codec => json } } output { elasticsearch { embedded => false host => "10.211.55.8" port => "9200" protocol => "http" } } type => “audit” will make all our logs have a common value in the type field, which will allow us to distinguish them from other logs by this discriminator. The remaining settings are pretty obvious. Run the newly baked container:
docker run -d -p 5959:5959 -v /mnt/logstash/conf:/opt/conf \ --name logstash aatarasoff/logstash and check that the logs will be written if we send json via tcp.
Implementation number 2
We are already doing the second module, so it is worth putting the implementation into a separate project, which we call auditor . First of all, we need to wind the already existing "meat" from the registrar. Therefore, we take our fork and brazenly copy the code into our project.
We check that everything is still going on by running the command: make dev.
We note that in the regitrator.go file, the bridge module is connected as an external dependency, so you can safely delete this folder. Check again that everything works.
Modify Dockerfile.dev:
FROM gliderlabs/alpine:3.1 CMD ["/bin/auditor"] ENV GOPATH /go RUN apk-install go git mercurial COPY . /go/src/github.com/aatarasoff/auditor RUN cd /go/src/github.com/aatarasoff/auditor \ && go get -v && go build -ldflags "-X main.Version dev" -o /bin/auditor Similarly, we are changing the release Dockefile. We remove the extra task and change the name of the container in the Makefile:
NAME=auditor VERSION=$(shell cat VERSION) dev: docker build -f Dockerfile.dev -t $(NAME):dev . docker run --rm --net host \ -v /var/run/docker.sock:/tmp/docker.sock \ $(NAME):dev /bin/auditor elastic: build: mkdir -p build docker build -t $(NAME):$(VERSION) . docker save $(NAME):$(VERSION) | gzip -9 > build/$(NAME)_$(VERSION).tgz Add a new module / logstash and a file logstash.go to our project. Take a ready-made client for logstash, which is dumb as a cork, and in fact is just a wrapper over the standard net library: github.com/heatxsink/go-logstash .
This time the structure of the container will differ slightly from the previous version:
type Container struct { Name string `json:"container_name"` Action string `json:"action"` Service *bridge.Service `json:"info"` } This is due to the fact that now we just need to serialize the object in json and send it as a string in logstash, which itself will deal with all the fields in the message.
Just like last time we register our factory:
func init() { bridge.Register(new(Factory), "logstash") } And create a new instance of the adapter:
func (f *Factory) New(uri *url.URL) bridge.RegistryAdapter { urls := "127.0.0.1:5959" if uri.Host != "" { urls = uri.Host } host, port, err := net.SplitHostPort(urls) if err != nil { log.Fatal("logstash: ", "split error") } intPort, _ := strconv.Atoi(port) client := logstashapi.New(host, intPort, 5000) return &LogstashAdapter{client: client} } type LogstashAdapter struct { client *logstashapi.Logstash } Here we had to use the net.SplitHostPort (urls) utilization method, which can isolate the host and port from the string, because the client accepts them separately, and they come together at uri.Host.
The numeric representation of the port can be obtained using the method of converting a string to a number: intPort, _: = strconv.Atoi (port). The underscore is needed because the function returns two parameters, the second of which is an error, which we can not handle.
The implementation of the Ping method is pretty simple:
func (r *LogstashAdapter) Ping() error { _, err := r.client.Connect() if err != nil { return err } return nil } In fact, we check that we can connect via tcp to logstash. In the Connect function, reconnection will occur only if the current one can no longer be used.
It remains to implement the registration method:
func (r *LogstashAdapter) Register(service *bridge.Service) error { container := Container{Name: service.Name, Action: "start", Service: service} asJson, err := json.Marshal(container) if err != nil { return err } _, err = r.client.Connect() if err != nil { return err } err = r.client.Writeln(string(asJson)) if err != nil { return err } return nil } I think that the code is clear enough and does not require comments, except for one. Calling Connect before Writeln ensures that a working connection is received.
The Deregister Method is a complete copy of the method above.
We change the dockerfile.dev in the elastic launch line to logstash, run and check for the presence of records in ElasticSearch:
curl 'http://localhost:9200/_search?pretty' ... share your happiness with others
Let's commit our changes to GitHub and go to build an image for DockerHub. On hub.docker.com , go to your page and click the button + Add Repository. When I was going to create an image for logstash, I chose the Repository sub-item, which allows you to manually upload your images, but there is another way - Automated Build. By clicking on it, Docker Hub will offer to connect your account to it on GitHub or BitBucket. After that, it remains only to choose your repository, the desired branch, and change the image names, if it is very necessary. Everything else, including the transfer of the description from README.MD will take over the Docker Hub.
After a short wait, here it is - the finished image .
Now you can test it by executing a simple command:
docker run -d --net=host \ -v /var/run/docker.sock:/tmp/docker.sock \ --name auditor aatarasoff/auditor logstash:// Ps. The project is not used in production, and from my critical point of view it requires finishing, but everyone who reads an article can try it and, if desired, improve it.
Source: https://habr.com/ru/post/257913/
All Articles