Extend collectd functionality by adding your own bindings

Hello% habraUser%
This article will discuss how to expand the collectd statistics collection system by adding new python bindings.
This article is an addition to this article.

Formulation of the problem

It is necessary to collect data using collectd from the ustats module for nginx. You need to collect HTTP 499, HTTP 500, HTTP 503 and TCP errors and display them on the same graph.

The solution of the problem

After studying the methods of collecting collectd statistics, it became clear that collecting data in the following ways:

• curl-json
• exec plugin
• python / perl / java plugin

I decided to write my plugin in python to understand the work of collectd itself.

The algorithm of the plugin is as follows

• Pick up data
• Parse them
• Put in collectd.

We take data:

Any module for collectd must contain
import collectd
We also load the modules we need
import json, urllib2
In the url variable we will store the link where the ustats data is located in the json format, which we will receive from the collectd config file.
url = None
Get data from ustats

 def fetch_data(url): response = urllib2.urlopen(urllib2.Request(url)) data = json.loads(response.read()) return data 

')
We are taking data from the config:

 def configure_callback(conf): global url for c in conf.children: if c.key == 'UstatsURL': url = c.values[0] elif c.key == 'Verbose': VERBOSE_LOGGING = bool(c.values[0]) else: collectd.warning ('ustats_info plugin: Unknown config key: %s.' % c.key) log_verbose('Configured with url=%s' % (url)) 

Next, we will have the plugin “reader”, respectively, it will be necessary to deal with the dispatch () function, which will put the data in collectd.
dispatch ([type] [, values] [, plugin_instance] [, type_instance] [, plugin] [, host] [, time] [, interval]) -> None.
type - data type
values - Actual values ​​that are sent to collectd. Data types are described in types.db.
plugin_instance - instance of the plugin
type_instance - type instance
plugin - the name of the plugin, by default python
interval - The time interval in seconds, between the first and second sending of the same data type. Must be a positive integer. If the number is negative, then the interval is the default (10 seconds)

Function to send data to collectd

 def dispatch_value(plugin_instance, info, key, type, type_instance=None): if not type_instance: type_instance = key value = int(info) log_verbose('Sending value: %s=%s' % (type_instance, value)) val = collectd.Values(plugin='ustats_info') val.plugin_instance = plugin_instance val.type = type val.type_instance = type_instance val.values = [value] val.dispatch() 

collectd stores data according to this scheme:
host "/" plugin ["-" plugin instance] "/" type ["-" type instance]
Here we have:
plugin_instance = upstream
type = backend
type_instance = errors
At the output we get about the following:
host / plugin-upstream / backend-tcperror.rrd

Now the main function to be executed collectd in a loop

 def read_callback(): global old_data log_verbose('Read callback called') data = fetch_data(url) last_backend = None if not data: collectd.error('ustats plugin: No data received') return if old_data == None: old_data = data upstreams = [] for key in data.keys(): upstreams.append(key) for upstream in upstreams: for index in range(len(data[upstream])): if data[upstream][index] and data[upstream][index] !=1 and data[upstream][index][0] != last_backend: dispatch_value(upstream, getValue(data[upstream][index][4], old_data[upstream][index][4]), 'http_499_errors', data[upstream][index][0].partition(":")[0]) dispatch_value(upstream, getValue(data[upstream][index][5], old_data[upstream][index][5]), 'http_500_errors', data[upstream][index][0].partition(":")[0]) dispatch_value(upstream, getValue(data[upstream][index][6], old_data[upstream][index][6]), 'http_503_errors', data[upstream][index][0].partition(":")[0]) dispatch_value(upstream, getValue(data[upstream][index][7], old_data[upstream][index][7]), 'tcp_errors', data[upstream][index][0].partition(":")[0]) dispatch_value(upstream, getValue(data[upstream][index][13], old_data[upstream][index][13]), 'total_errors', data[upstream][index][0].partition(":")[0]) last_backend = data[upstream][index][0] old_data = data 

After registering the reader function and configuration function

 collectd.register_config(configure_callback) collectd.register_read(read_callback) 

The plugin is ready, but we used types that are not known collectd, add to my_types.db
backend value:GAUGE:0:65535
describe the plugin in collectd.conf

 <Plugin python > ModulePath "/usr/lib/collectd/plugins/python" #   ,    Import "ustats_info" #   <Module ustats_info> UstatsURL "http://localhost/ustats?json" Verbose true </Module> </Plugin> 

Conclusion

The graphics are of this type.

The full text of the script on GitHub
I hope that the article will help someone to expand the collection of statistics collectd