ExpvarMon - console monitoring of services on Go
For Go-programs, there is a convenient standard expvar package, allowing one line to connect the output of debag information in JSON format. And in order to monitor the current state as quickly and clearly as possible, the Expvarmon console program was written, requiring a minimum of configuration for displaying metrics and debugging information for your Go-services.
Functions:
')
In the standard Go library there is a very useful package expvar , which allows you to add a single output line of debug information in json-format to the address / debug / vars. By default, memory usage and garbage collector (GC) data are displayed, and any of their own metrics / counters / variables are easily added. Usually, this data is collected by a separate process, which puts into some time-series database, and then it turns into convenient and beautiful dashboards.
But often, even during development or debugging sessions, you just want to monitor the status of the program, be sure that resources are used adequately to the load, there are no memory leaks or gorutin, and so on. Raising a whole monitoring infrastructure for this is often costly and unreasonable, and raw json is not very presentable.
For such cases, a program was written over the weekend for monitoring expvar variables right in the terminal, which requires almost zero configuration and does not use any third-party databases and resources. The program uses the excellent TermUI package from the terminal gizak lover (see its homepage !).
Installing a program like any other program on Go is extremely simple:
I hope you have $ GOPATH / bin registered in $ PATH.
If you are not familiar with the expvar package, then a brief explanation and instruction.
The init () function of this package says the following:
The first line registers the handler for handling the URL "/ debug / vars" for the standard http.DefaultServeMux from the standard net / http package. If you don’t know how net / http works, this may be a separate article, but now you only need to know that if your program starts a standard http-server (say, http.ListenAndServe (": 1234", nil) ), then it will automatically have a GET request handler at / debug / vars. And the response of this request will by default contain approximately the following JSON:
This is the JSON representation of the two variables defined by the next two lines — the command line and the current runtime.Memstats values. The latter contains a lot of details about the current memory usage and the work of the garbage collector, most of which are too detailed. Typically, the values ​​used for monitoring are Alloc, Sys, HeapAlloc — actually used memory requested from the OS, used memory in the heap, respectively.
Since init () calls automatically when importing a package, it’s enough to add one line in the program:
This program is extremely simple - it reads this JSON for the specified service or services at a specified interval, and displays it in a convenient form for monitoring, while showing sparkline-graphs for numerical values. All it needs to run is the port or “host: port” of the service (s) you want to monitor. For example:
You can specify both one and 30+ ports / services - how long do you have the size of the terminal.
The program can also monitor itself:
The default interval is 5 seconds, but you can specify less or more. Keep in mind that a too short interval is not recommended, since even the memstats update affects the garbage collector and increases the pauses. If your application runs under a heavy load, a very short interval (100ms) can affect productivity.
By default, the following variables are monitored:
The values ​​of variables are taken in the form in which they are in JSON, with a record through the dot. The “mem:”, “duration:”, “str:” modifiers are optional, and affect the formatting / display. If you specify variables without a modifier, they will be displayed as is. The modifier “mem:” will convert the values ​​to the “KB, MB, etc” representation, and the “duration:” will convert the int64 value to time.Duration (“ns, ms, s, m, etc”). The “str:” modifier simply says that the value is not digital, and there is no need to draw a sparkline graph for this variable.
For example:
Again, you can specify as one variable, and a couple of dozen, as far as you have the size of the terminal.
The program has two modes - for one service and for several services. In the first case, the graphs are displayed for all variables, in the second - only for the first two, in the order in which they are indicated. For graphs, the maximum values ​​that were observed during the monitoring session are displayed.
Expvarmon displays icons with services that have fallen and that are currently lying. Unfortunately, if the polling interval is longer than the service crash / restart time, the program will not catch the service crash.
It is also important to understand that the data is not recorded or stored anywhere. The graphs show the latest values ​​- and the scale of the graphs depends on the interval and the size of the terminal. There is no zoom, no history, no time search. This solution is for a simpler task - instant monitoring of current values.
Thanks to the capabilities of TermUI, the program dynamically changes the size of all widgets when changing the font size or the terminal window.
Personally, in the standard variable list, expvar lacks two things - the number of running gorutin and service uptime. Here is a demo wrapper that will export three additional variables. Just connect it to your program, one import.
Many web services on Go are written using additional web frameworks or more advanced http routers. expvar out of the box will not work with them. For it, you will need to start the standard http.ListenAndServer () on another port. And this is even better, since opening out / debug / vars is highly discouraged when it comes to public web services.
If you are using standard net / http, but you want expvar to be on a different port, the easiest way is to do so. "Main" ServeMux run as follows:
a / debug / vars hang on standard
Github: github.com/divan/expvarmon
Expvar docs: golang.org/pkg/expvar
runtime.Memstats: golang.org/pkg/runtime/#MemStats
Functions:
- single- and multi-services modes
- monitoring local and remote programs
- arbitrary number of services and variables
- support for memory, time intervals, bool and arbitrary numbers / strings
- sparkline graphics
- display of maximum values
- display fallen / restarted services
- auto-resize when changing font or window sizes
')
Introduction
In the standard Go library there is a very useful package expvar , which allows you to add a single output line of debug information in json-format to the address / debug / vars. By default, memory usage and garbage collector (GC) data are displayed, and any of their own metrics / counters / variables are easily added. Usually, this data is collected by a separate process, which puts into some time-series database, and then it turns into convenient and beautiful dashboards.
But often, even during development or debugging sessions, you just want to monitor the status of the program, be sure that resources are used adequately to the load, there are no memory leaks or gorutin, and so on. Raising a whole monitoring infrastructure for this is often costly and unreasonable, and raw json is not very presentable.
For such cases, a program was written over the weekend for monitoring expvar variables right in the terminal, which requires almost zero configuration and does not use any third-party databases and resources. The program uses the excellent TermUI package from the terminal gizak lover (see its homepage !).
Installation
Installing a program like any other program on Go is extremely simple:
go get github.com/divan/expvarmon I hope you have $ GOPATH / bin registered in $ PATH.
Using
expvar
Short explanation
import _ "expvar"
Long explanation
If you are not familiar with the expvar package, then a brief explanation and instruction.
The init () function of this package says the following:
func init() { http.HandleFunc("/debug/vars", expvarHandler) Publish("cmdline", Func(cmdline)) Publish("memstats", Func(memstats)) }
The first line registers the handler for handling the URL "/ debug / vars" for the standard http.DefaultServeMux from the standard net / http package. If you don’t know how net / http works, this may be a separate article, but now you only need to know that if your program starts a standard http-server (say, http.ListenAndServe (": 1234", nil) ), then it will automatically have a GET request handler at / debug / vars. And the response of this request will by default contain approximately the following JSON:
$ curl -s http://localhost:1234/debug/vars | json_pp | head { "cmdline" : [ "./expvar.demo", "-bind=:1234", ], "memstats" : { "NumGC" : 5, "Alloc" : 114016, "DebugGC" : false, "HeapObjects" : 519, "HeapSys" : 868352, "StackInuse" : 180224,
This is the JSON representation of the two variables defined by the next two lines — the command line and the current runtime.Memstats values. The latter contains a lot of details about the current memory usage and the work of the garbage collector, most of which are too detailed. Typically, the values ​​used for monitoring are Alloc, Sys, HeapAlloc — actually used memory requested from the OS, used memory in the heap, respectively.
Since init () calls automatically when importing a package, it’s enough to add one line in the program:
import _ "expvar"
expvarmon
This program is extremely simple - it reads this JSON for the specified service or services at a specified interval, and displays it in a convenient form for monitoring, while showing sparkline-graphs for numerical values. All it needs to run is the port or “host: port” of the service (s) you want to monitor. For example:
expvarmon -ports="80" expvarmon -ports="23000-23010,80" expvarmon -ports="80,remoteapp.corp.local:80-82"
You can specify both one and 30+ ports / services - how long do you have the size of the terminal.
The program can also monitor itself:
expvarmon -self
The default interval is 5 seconds, but you can specify less or more. Keep in mind that a too short interval is not recommended, since even the memstats update affects the garbage collector and increases the pauses. If your application runs under a heavy load, a very short interval (100ms) can affect productivity.
expvarmon -self -i 5m expvarmon -self -i 30s
By default, the following variables are monitored:
- mem: memstats.Alloc
- mem: memstats.Sys
- mem: memstats.HeapAlloc
- mem: memstats.HeapInuse
- memstats.EnableGC
- memstats.NumGC
- duration: memstats.PauseTotalNs
The values ​​of variables are taken in the form in which they are in JSON, with a record through the dot. The “mem:”, “duration:”, “str:” modifiers are optional, and affect the formatting / display. If you specify variables without a modifier, they will be displayed as is. The modifier “mem:” will convert the values ​​to the “KB, MB, etc” representation, and the “duration:” will convert the int64 value to time.Duration (“ns, ms, s, m, etc”). The “str:” modifier simply says that the value is not digital, and there is no need to draw a sparkline graph for this variable.
For example:
expvarmon -ports="80" -vars="mem:memstats.Alloc,duration:Response.Mean,Goroutines,str:Uptime"
Again, you can specify as one variable, and a couple of dozen, as far as you have the size of the terminal.
The program has two modes - for one service and for several services. In the first case, the graphs are displayed for all variables, in the second - only for the first two, in the order in which they are indicated. For graphs, the maximum values ​​that were observed during the monitoring session are displayed.
Additionally
Expvarmon displays icons with services that have fallen and that are currently lying. Unfortunately, if the polling interval is longer than the service crash / restart time, the program will not catch the service crash.
It is also important to understand that the data is not recorded or stored anywhere. The graphs show the latest values ​​- and the scale of the graphs depends on the interval and the size of the terminal. There is no zoom, no history, no time search. This solution is for a simpler task - instant monitoring of current values.
Thanks to the capabilities of TermUI, the program dynamically changes the size of all widgets when changing the font size or the terminal window.
Additional variables
Personally, in the standard variable list, expvar lacks two things - the number of running gorutin and service uptime. Here is a demo wrapper that will export three additional variables. Just connect it to your program, one import.
package myexpvars import ( "expvar" "math/rand" "runtime" "time" ) var ( startTime = time.Now().UTC() ) // goroutines is an expvar.Func compliant wrapper for runtime.NumGoroutine function. func goroutines() interface{} { return runtime.NumGoroutine() } // uptime is an expvar.Func compliant wrapper for uptime info. func uptime() interface{} { uptime := time.Since(startTime) return int64(uptime) } // responseTime fake var. func responseTime() interface{} { resp := time.Duration(rand.Intn(1000)) * time.Millisecond return int64(resp) } func init() { expvar.Publish("Goroutines", expvar.Func(goroutines)) expvar.Publish("Uptime", expvar.Func(uptime)) expvar.Publish("MeanResponse", expvar.Func(responseTime)) }
What to do if you use a third-party http-router, instead of the standard
Many web services on Go are written using additional web frameworks or more advanced http routers. expvar out of the box will not work with them. For it, you will need to start the standard http.ListenAndServer () on another port. And this is even better, since opening out / debug / vars is highly discouraged when it comes to public web services.
If you are using standard net / http, but you want expvar to be on a different port, the easiest way is to do so. "Main" ServeMux run as follows:
mux := http.NewServerMux() server := &http.Server{Addr: “:80”, Handler: mux} server.ListenAndServe()
a / debug / vars hang on standard
http.ListenAndServe(":1234", nil)
Screenshots
Links
Github: github.com/divan/expvarmon
Expvar docs: golang.org/pkg/expvar
runtime.Memstats: golang.org/pkg/runtime/#MemStats
Source: https://habr.com/ru/post/257593/
All Articles