Geekly Articles each Day
📜 ⬆️ ⬇️

Update timezone in logstash

Hello! For a couple of months already, our company has successfully used in production a bundle of logstash-elasticsearch-kibana for collecting and processing a sufficiently large amount of logs. Looking at the kibana after the transfer of the clock it was found that all the logs come with a time lag of 1 hour. Under the cut I want to share a solution to the problem with timezones in a bunch of logstash-elasticsearch-kibana and the finished build of logstash with updated timezones.

I will not particularly delve into how the change of timezone influenced the final view of the logs, but in fact in kibana we received events created at the current time as events an hour earlier. It turned out quite a funny picture, given that not all rsyslogd restarted automatically and picked up the new timezone, a couple of servers continued to work on the old time zones.

Of course, you can solve the problem with the logstash configuration crutch, but I decided to fully update the timezone in logstash.


In fig. schedule of events at the time of transition to logstash with the correct timezones.
At Logstash, three sources of time zones are possible at once - one in java , the second in the ruby tzinfo gem , the third is compiled into jruby (the joda-time artifact and joda-timezones).

  1. Let's start with a simple one - update tzdata in java:


    To do this, use the utility from Oracle tzupdater , you can download from the Oracle website .
    ')
    The update takes place literally in one command:

    java -jar tzupdater.jar -V tzupdater version 1.4.7-b01 JRE time zone data version: tzdata2014c Embedded time zone data version: tzdata2014g java -jar tzupdater.jar -u java -jar tzupdater.jar -V tzupdater version 1.4.7-b01 JRE time zone data version: tzdata2014g Embedded time zone data version: tzdata2014g

  2. Build jruby with new tzdata:


    1. First, let's collect the joda-time artifact with new timezones:
      Before building, install the maven and ant packages:
       apt-get install maven ant
      Download the latest available joda-timezones package, replace the tzdata version in it, and specify the more recent joda-time (2.5 dated October 3, 2014) in dependencies:
       mkdir joda-time cd joda-time wget http://search.maven.org/remotecontent?filepath=org/jruby/joda-timezones/2013d/joda-timezones-2013d.pom -O pom.xml sed -i 's/2013d/2014g/' pom.xml sed -i 's/<version>2.2/<version>2.5/' pom.xml mvn package mvn instal
      Now, maven has received the current version of joda-timezones.
    2. Download the source version of jruby, which is used in logstash (for logstash 1.4, this is jruby-1.7.11):
       wget https://github.com/jruby/jruby/archive/1.7.11.tar.gz tar -xvf 1.7.11.tar.gz cd jruby-1.7.11/
      Replace the artifact version in ./core/pom.xml:
       sed -i 's/<tzdata.version>2013d/<tzdata.version>2014g/' ./core/pom.xml sed -i 's/<tzdata.jar.version>2013d/<tzdata.jar.version>2014g/' ./core/pom.xml
      In ./pom.xml:
       sed -i 's/<joda.time.version>2.3/<joda.time.version>2.5/' ./pom.xml
      Putting jruby:
       ~/jruby-1.7.11# mvn -Pcomplete
      After the build, check which version of tz will be answered by jruby, at the same time we will compare the time with the real one (at the end I start the http-server, which would be more convenient to collect logstash):
       ~/jruby-1.7.11# java -jar ./maven/jruby-complete/target/jruby-complete-1.7.11.jar -rrbconfig -e 'p RbConfig::CONFIG["tzdata.version"]' "2014g" ~/jruby-1.7.11# java -jar ./maven/jruby-complete/target/jruby-complete-1.7.11.jar -e 'p Time.now' 2014-10-29 14:58:07 +0500 ~/jruby-1.7.11# cd ./maven/jruby-complete/target/;python -m SimpleHTTPServer

  3. We collect logstash:


    Download unpack:
     wget https://github.com/elasticsearch/logstash/archive/v1.4.2.tar.gz tar -xvf v1.4.2.tar.gz mv logstash-1.4.2 logstash-contrib;cd logstash-contrib
    Make changes:
     diff --git a/Makefile b/Makefile index 0ec3da1..7fcca1a 100644 --- a/Makefile +++ b/Makefile @@ -7,7 +7,7 @@ ELASTICSEARCH_VERSION=1.1.1 WITH_JRUBY=java -jar $(shell pwd)/$(JRUBY) -S JRUBY=vendor/jar/jruby-complete-$(JRUBY_VERSION).jar -JRUBY_URL=http://jruby.org.s3.amazonaws.com/downloads/$(JRUBY_VERSION)/jruby-complete-$(JRUBY_VERSION).jar +JRUBY_URL=http://127.0.0.1:8000/jruby-complete-$(JRUBY_VERSION).jar JRUBY_CMD=bin/logstash env java -jar $(JRUBY) ELASTICSEARCH_URL=http://download.elasticsearch.org/elasticsearch/elasticsearch diff --git a/logstash.gemspec b/logstash.gemspec index 4917d83..6ba8ae4 100644 --- a/logstash.gemspec +++ b/logstash.gemspec @@ -23,6 +23,7 @@ Gem::Specification.new do |gem| gem.add_runtime_dependency "stud" #(Apache 2.0 license) gem.add_runtime_dependency "clamp" # for command line args/flags (MIT license) gem.add_runtime_dependency "i18n", [">=0.6.6"] #(MIT license) + gem.add_runtime_dependency "tzinfo", [">=1.2.2"]#(MIT license) # Web dependencies gem.add_runtime_dependency "ftw", ["~> 0.0.39"] #(Apache 2.0 license) diff --git a/tools/Gemfile.jruby-1.9.lock b/tools/Gemfile.jruby-1.9.lock index dc11fd5..41e4362 100644 --- a/tools/Gemfile.jruby-1.9.lock +++ b/tools/Gemfile.jruby-1.9.lock @@ -169,7 +165,7 @@ GEM http_parser.rb (~> 0.5.0) json (~> 1.8) simple_oauth (~> 0.2.0) - tzinfo (1.1.0) + tzinfo (1.2.2) thread_safe (~> 0.1) user_agent_parser (2.1.2) uuidtools (2.1.4)

    And run the assembly:
     make tarball

    Upon completion of the assembly we get ./build/logstash-1.4.2.tar.gz - ready logstash with updated tzdata!


PS All manipulations were done on ubuntu 14.04 with Oracle Java (TM) Development Kit (JDK) 7 installed (build 1.7.0_72-b14)

The finished build of logstash 1.4.3 with updated timezone can be downloaded from yandex-disk or mail.ru.

Source: https://habr.com/ru/post/241957/

More articles:


All Articles