Application Monitoring with Prometheus
Good evening!
This week we are launching the fourth in a row “DevOps: practices and tools” course, so by tradition a small interesting article is for you.
Go
')
In this practical guide, we will look at how to integrate Prometheus monitoring into an existing application. Monitoring the application can give an idea of how and when the application is used. Moreover, potential problems can be foreseen.
Key points:
Value of Exporters
Prometheus is a time series database with a UI and a very complex query language (PromQL). Prometheus can parse metrics, counters, indicators and histograms over HTTP using plaintext and a more efficient protocol.
Glossary:
Embedding endpoints / metrics into an existing application is called instrumentation. And when endpoint / metrics is part of an autonomous process, the exporter (Expoter).
NodeExporter
One of the most popular exporters is NodeExporter. When NodeExporter is running on a host, it provides details of the I / O load, memory, disk, and CPU. It can be run as a Docker container, but it requires a large number of additional flags, so it is recommended to run it directly on the host being monitored.
Built-in exporter
Prometheus provides its own set of metrics - which, in essence, is dogfooding (using your own products for testing purposes). It is already built in and usually enabled by default.
Community Exporters
Interesting exporters written for the Docker Summit in Berlin are the Docker Hub and GitHub exporters by Edward Marshall. These exporters collect Docker Hub and GitHub site metrics by periodically querying the API and passing these values.
Ed used the Python client libraries to create the exporter, but other language bindings are available.
One of the first exporters I wrote was to monitor the Bitcoin mining statistics — how many dollars I earned, how many solutions (hashes) my hardware processes per second.
My Bitcoin exporter is written in Node.js and uses the Histogram and Radial Sensor type metrics.
Other exporters are highlighted in the documentation. For example, MySQL, Mongo, Redis, NATS, Nginx, and JenkinsCI.
Creating your exporter
And then everything becomes much more interesting - you can monitor almost anything. Imagine that we have a shop in Shopify (that is: a pocket online store) in which we monitor sales and order status.
Here are some ideas for metrics you can follow:
And if this kind of data is already available, you can simply monitor lower-level metrics:
Let's look at a simple Prometheus recipe with Docker, and then back to writing your own instrumentation for a test application.
Prometheus with Docker
Prometheus is written in Golang and can be used as a standalone statically compiled binary file without dependencies. The project also packs a binary file with a reasonable configuration in a Docker container.
Run Prometheus in Docker
Define the Docker Compose, which allows command lines to remain simple and repeatable:
Expand the stack file:
Swarm mode is required for deployment of stack files, so run docker swarm init, if you haven’t done so before.
Go to http: // localhost: 9090 / for a look at the interface.
In the screenshot above, you can see the number of go_routines used, recorded by Prometheus itself. To see the raw metrics that Prometheus collects about itself, open a browser and go to http: // localhost: 9090 / metrics
The go routine is a lightweight version of the stream that is used in Golang to ensure competitiveness.
You probably do not need to monitor Prometheus. But the fact of having such an opportunity means that you can immediately get acquainted with the metrics.
Application Instrumentation
Look at the instrumentation of your application.
There are two approaches to instrumentation, but both include the need to open or implement an HTTP / s endpoint. The default endpoint is / metrics, but you can migrate it to your prometheus.yml file. Prometheus will use the endpoint to collect metrics at regular intervals of 5s or 30s.
Do I need to edit the application code?
You can make / metrics endpoint part of the existing application code. This is feasible if you already have the details and data to interact with payment levels or databases. Minus - you need to include a new library, endpoint and dependency in your product or project.
What is the other option?
You can write a separate process that acts as an interlayer for accessing information from your application or environment. In the case of the Docker Hub exporter , data was read from an external API that could not be managed. Therefore, if it is difficult for you to obtain permission to modify an existing application, this method may work.
The advantage of a separate process is that you do not have to redeploy the application if you need to update what you are monitoring.
Endpoint implementation
Prometheus can read data from two “exposure” formats. Go to localhost : 9090 / metrics and look at the result of the previous go_routines example.
Use client library
There are many libraries for working with metrics, and most of them can output text and the more efficient binary format (Protobuf) mentioned above.
Prometheus Exposure Formats
The project supports language bundles for Golang, Java, Python and Ruby, but there are many others in the public domain.
The full list can be found here:
Prometheus Libraries
Should I write my own?
The plaintext format is so simple that you can easily embed a protocol by following the Prometheus exposure formats .
And before you create your own, make sure that you definitely will not be able to use existing trusted client libraries.
Instrumenting Golang Applications
Create a simple application and instrument it directly using the Golang Prometheus library.
Usage example:
You need to write a web service to provide SHA-256 hashes on demand. For this you need to know a few things:
Fortunately, you can get all of the above using a histogram metric, but the simplest example would be a counter (always increasing or staying the same) of a radial sensor (similar to a counter, but it can either rise or fall).
You can see on Metric Types
The full code, including the Dockerfile, is available on GitHub: alexellis / hash-browns
Here is an example of the “server.go” file:
Here we register our application path and metrics.
After that, you can record the time spent by calling histogram.Observe (seconds).
You can generate a hash:
The last step is to edit prometheus.yml and start parsing the new application code. After that you will find the metrics in the drop-down list and be able to visualize the values.
Union
We need to change the configuration of Prometheus, for this we will use a little trick to extract the default config from the official Docker image:
Now fix the file prometheus.yml, created in the current directory. In the scrape_config section, add the following:
Docker Swarm allows services to communicate with each other via the built-in DNS, so the target is simply a label (“hashbrowns”) from the layout file.
Now create a new docker-compose.yml file:
Deploy the stack file:
Note: Calling the service by name will work as long as you have only one replica. If you want to scale the service, you'll have to figure out how to let Prometheus find the replicas separately.
Here’s how it looks in the Prometheus interface:
We give answers in the format of PromQL using the metrics that we have collected above. It took me some time to get used to PromQL , but I was lucky that Julius wrote a very detailed article available here .
The PromQl expression written below gives the ascending rate for our hash function within 1 minute. Frames can be reduced, but they must cover at least two samples.
To find out the average for the last 5 minutes, write:
Completion
Yes, I have a Docker image (alexellis2 / prometheus-armhf: 1.5.2), available on the Docker Hub for Prometheus, and the AlertManager project (alexellis2 / alertmanager-armhf: 0.5.1).
THE END
As always, we are waiting for questions and comments here or at the Open Day .
This week we are launching the fourth in a row “DevOps: practices and tools” course, so by tradition a small interesting article is for you.
Go
')
In this practical guide, we will look at how to integrate Prometheus monitoring into an existing application. Monitoring the application can give an idea of how and when the application is used. Moreover, potential problems can be foreseen.
Key points:
- Learn how to monitor applications and servers with Prometheus;
- Run Prometheus in Docker;
- Monitoring and Instrumenting a Golang Sample Application;
- Deploy Prometheus with Docker Stacks;
- A brief tutorial with PromQL examples.
Value of Exporters
Prometheus is a time series database with a UI and a very complex query language (PromQL). Prometheus can parse metrics, counters, indicators and histograms over HTTP using plaintext and a more efficient protocol.
Glossary:
Embedding endpoints / metrics into an existing application is called instrumentation. And when endpoint / metrics is part of an autonomous process, the exporter (Expoter).
NodeExporter
One of the most popular exporters is NodeExporter. When NodeExporter is running on a host, it provides details of the I / O load, memory, disk, and CPU. It can be run as a Docker container, but it requires a large number of additional flags, so it is recommended to run it directly on the host being monitored.
Built-in exporter
Prometheus provides its own set of metrics - which, in essence, is dogfooding (using your own products for testing purposes). It is already built in and usually enabled by default.
Community Exporters
Interesting exporters written for the Docker Summit in Berlin are the Docker Hub and GitHub exporters by Edward Marshall. These exporters collect Docker Hub and GitHub site metrics by periodically querying the API and passing these values.
Ed used the Python client libraries to create the exporter, but other language bindings are available.
- Docker Hub Exporter , written in Python
One of the first exporters I wrote was to monitor the Bitcoin mining statistics — how many dollars I earned, how many solutions (hashes) my hardware processes per second.
My Bitcoin exporter is written in Node.js and uses the Histogram and Radial Sensor type metrics.
- Bitcoin Mining Exporter on Node.js
Other exporters are highlighted in the documentation. For example, MySQL, Mongo, Redis, NATS, Nginx, and JenkinsCI.
Creating your exporter
And then everything becomes much more interesting - you can monitor almost anything. Imagine that we have a shop in Shopify (that is: a pocket online store) in which we monitor sales and order status.
Here are some ideas for metrics you can follow:
- best selling product;
- best selling category;
- total number of orders for a certain time;
- waiting time between payment and sending the order;
- total number of reviews;
- average rating in reviews;
- total number of registrations.
And if this kind of data is already available, you can simply monitor lower-level metrics:
- the number of transactions processed;
- payment gateway response time;
- HTTP errors, such as 403 and 404;
- geolocation of buyers using IP addresses.
Let's look at a simple Prometheus recipe with Docker, and then back to writing your own instrumentation for a test application.
Prometheus with Docker
Prometheus is written in Golang and can be used as a standalone statically compiled binary file without dependencies. The project also packs a binary file with a reasonable configuration in a Docker container.
Run Prometheus in Docker
Define the Docker Compose, which allows command lines to remain simple and repeatable:
version: "3" services: prometheus: image: quay.io/prometheus/prometheus:latest ports: - 9090:9090 Expand the stack file:
Swarm mode is required for deployment of stack files, so run docker swarm init, if you haven’t done so before.
$ docker swarm init $ docker stack deploy monitoring --compose-file=./docker-compose.yml Go to http: // localhost: 9090 / for a look at the interface.
In the screenshot above, you can see the number of go_routines used, recorded by Prometheus itself. To see the raw metrics that Prometheus collects about itself, open a browser and go to http: // localhost: 9090 / metrics
The go routine is a lightweight version of the stream that is used in Golang to ensure competitiveness.
You probably do not need to monitor Prometheus. But the fact of having such an opportunity means that you can immediately get acquainted with the metrics.
Application Instrumentation
Look at the instrumentation of your application.
There are two approaches to instrumentation, but both include the need to open or implement an HTTP / s endpoint. The default endpoint is / metrics, but you can migrate it to your prometheus.yml file. Prometheus will use the endpoint to collect metrics at regular intervals of 5s or 30s.
Do I need to edit the application code?
You can make / metrics endpoint part of the existing application code. This is feasible if you already have the details and data to interact with payment levels or databases. Minus - you need to include a new library, endpoint and dependency in your product or project.
What is the other option?
You can write a separate process that acts as an interlayer for accessing information from your application or environment. In the case of the Docker Hub exporter , data was read from an external API that could not be managed. Therefore, if it is difficult for you to obtain permission to modify an existing application, this method may work.
The advantage of a separate process is that you do not have to redeploy the application if you need to update what you are monitoring.
Endpoint implementation
Prometheus can read data from two “exposure” formats. Go to localhost : 9090 / metrics and look at the result of the previous go_routines example.
# HELP go_goroutines Number of goroutines that currently exist. # TYPE go_goroutines gauge go_goroutines 92 Use client library
There are many libraries for working with metrics, and most of them can output text and the more efficient binary format (Protobuf) mentioned above.
Prometheus Exposure Formats
The project supports language bundles for Golang, Java, Python and Ruby, but there are many others in the public domain.
The full list can be found here:
Prometheus Libraries
Should I write my own?
The plaintext format is so simple that you can easily embed a protocol by following the Prometheus exposure formats .
And before you create your own, make sure that you definitely will not be able to use existing trusted client libraries.
Instrumenting Golang Applications
Create a simple application and instrument it directly using the Golang Prometheus library.
Usage example:
You need to write a web service to provide SHA-256 hashes on demand. For this you need to know a few things:
- How many requests are received for hashes;
- What is the average time it takes to calculate a hash;
- How many 400 errors (bad request) we get.
Fortunately, you can get all of the above using a histogram metric, but the simplest example would be a counter (always increasing or staying the same) of a radial sensor (similar to a counter, but it can either rise or fall).
You can see on Metric Types
The full code, including the Dockerfile, is available on GitHub: alexellis / hash-browns
Here is an example of the “server.go” file:
func main() { histogram := prometheus.NewHistogramVec(prometheus.HistogramOpts{ Name: "hash_duration_seconds", Help: "Time taken to create hashes", }, []string{"code"}) r := mux.NewRouter() r.Handle("/metrics", prometheusHandler()) r.Handle("/hash", hashHandler(histogram)) prometheus.Register(histogram) s := &http.Server{ Addr: ":8080", ReadTimeout: 8 * time.Second, WriteTimeout: 8 * time.Second, MaxHeaderBytes: 1 << 20, Handler: r, } log.Fatal(s.ListenAndServe()) } Here we register our application path and metrics.
After that, you can record the time spent by calling histogram.Observe (seconds).
start := time.Now() // Do something duration := time.Since(start) code := 200 // some HTTP Code histogram.WithLabelValues(fmt.Sprintf("%d", code)).Observe(duration.Seconds()) You can generate a hash:
$ curl localhost:8080/hash -d "my_input_value_here" 49b8c4256d603a68ee9bcd95f8e11eed784189bd40c354950014b6d7f7263d6c C , curl localhost:8080/metrics: # HELP hash_seconds Time taken to create hashes # TYPE hash_seconds histogram hash_seconds_bucket{code="200",le="1"} 2 hash_seconds_bucket{code="200",le="2.5"} 2 hash_seconds_bucket{code="200",le="5"} 2 hash_seconds_bucket{code="200",le="10"} 2 hash_seconds_bucket{code="200",le="+Inf"} 2 hash_seconds_sum{code="200"} 9.370800000000002e-05 hash_seconds_count{code="200"} 2 The last step is to edit prometheus.yml and start parsing the new application code. After that you will find the metrics in the drop-down list and be able to visualize the values.
Union
We need to change the configuration of Prometheus, for this we will use a little trick to extract the default config from the official Docker image:
$ docker run --entrypoint='' -ti quay.io/prometheus/prometheus:latest /bin/cat /etc/prometheus/prometheus.yml > prometheus.yml Now fix the file prometheus.yml, created in the current directory. In the scrape_config section, add the following:
- job_name: 'hashbrowns' # metrics_path defaults to '/metrics' # scheme defaults to 'http'. static_configs: - targets: ['hashbrowns:8080'] Docker Swarm allows services to communicate with each other via the built-in DNS, so the target is simply a label (“hashbrowns”) from the layout file.
Now create a new docker-compose.yml file:
version: "3" services: prometheus: image: quay.io/prometheus/prometheus:latest ports: - 9090:9090 volumes: - "./prometheus.yml:/etc/prometheus/prometheus.yml" hashbrowns: image: alexellis2/hash-browns ports: - 8080:8080 Deploy the stack file:
$ docker stack deploy tutorial --compose-file=./docker-compose.yml Note: Calling the service by name will work as long as you have only one replica. If you want to scale the service, you'll have to figure out how to let Prometheus find the replicas separately.
Here’s how it looks in the Prometheus interface:
We give answers in the format of PromQL using the metrics that we have collected above. It took me some time to get used to PromQL , but I was lucky that Julius wrote a very detailed article available here .
- Answer to the question: How many requests are received for hashes;
The PromQl expression written below gives the ascending rate for our hash function within 1 minute. Frames can be reduced, but they must cover at least two samples.
rate(hash_seconds_count[1m]) - The answer to the question: What is the average time it takes to calculate the hash;
To find out the average for the last 5 minutes, write:
rate(hash_seconds_sum[5m]) / rate(hash_seconds_count[5m]) - The answer to the question: How many errors 400 (bad request) we get.
rate(hash_seconds_count{code="400"}[5m]) Completion
- Does it work on an ARM processor or a Raspberry Pi?
Yes, I have a Docker image (alexellis2 / prometheus-armhf: 1.5.2), available on the Docker Hub for Prometheus, and the AlertManager project (alexellis2 / alertmanager-armhf: 0.5.1).
THE END
As always, we are waiting for questions and comments here or at the Open Day .
Source: https://habr.com/ru/post/358588/
All Articles