Log server Elasticsearch + Logstash + Kibana4 + Beats (windows / linux). Installation and Setup

Retrieving server infrastructure data.
Transmits information about the processor, memory usage. For each process, information about parents, pid, status, etc. is displayed. Also Topbeat allows you to view information about the file system - the state of the disks, the amount of free space, etc.

Installation (on client):

 cd ~/ELK/releases/beats/topbeat/ sudo curl -L -O https://download.elastic.co/beats/topbeat/topbeat_1.1.0_amd64.deb sudo dpkg -i topbeat_1.1.0_amd64.deb 

You need to add Topbeat index templates to the server so that Elasticsearch can correctly analyze input information:

 sudo curl -XPUT 'http://localhost:9200/_template/topbeat' -d@/etc/topbeat/topbeat.template.json 

With a successful download, we should see:

{"acknowledged":true}

The topbeat.template.json file is created when you install Topbeat and has a default location of /etc/topbeat/topbeat.template.json. Therefore, if for some reason we will not install Beats clients on the ELK server, we will need to copy this template from the client to the server, or create this file on the server and copy its contents there (from the client). And then download it curl -XPUT 'eel_server_server: 9200 / _template / topbeat' -d@/PATH/topbeat.template.json .
But we will assume that the Bits are installed on the server and have the following location /etc/topbeat/topbeat.template.json .

We edit the config (on the client):

 sudo vi /etc/topbeat/topbeat.yml 

In the output block, you need to comment on the appeal to elasticsearch, because we will use logstash:

 ### Elasticsearch as output #elasticsearch: #hosts: ["localhost:9200"] 

Uncomment the block with Logstash, specify its IP address and port:

  logstash: hosts: ["ip_elk-server.ss.lu:5044"] 

Important: do not use tabs to move the cursor in the config! Only spaces. Otherwise, you get an error:

 Loading config file error: YAML config parsing failed on /etc/topbeat /topbeat.yml: yaml: line 14: found character that cannot start any token. Exiting. 

If the Logstash server is located on the external network, then the forward server of the port must be configured on the remote server's firewall, in this case 5044 (tcp / udp).
Additional logging options are well described in configs.

We start the service:

 sudo /etc/ini.d/topbeat start 

Open the Kibana interface and observe the incoming information:

Transmits information from dynamic files to the server, which we will specify:

Setting:

 cd ~/ELK/releases/beats/filebeat/ sudo curl -L -O https://download.elastic.co/beats/filebeat/filebeat_1.1.0_amd64.deb sudo dpkg -i filebeat_1.1.0_amd64.deb 

Add indexes on the server (by analogy with how we set up Topbeat. Ie, if there is no template on the server, we create it):

 sudo curl -XPUT 'http://localhost:9200/_template/filebeat?pretty' -d@/etc/filebeat/filebeat.template.json 

Open the config:

 sudo vi /etc/filebeat/filebeat.yml 

Specify from which files we will take information (by default, all files from / var / log are with the .log extension):

 prospectors: paths: - /var/log/*.log 

We specify what we need on this client, for example:

 paths: # - /var/log/*.log - /var/log/elasticsearch/*.log - /var/log/syslog - /var/log/nginx/*.log # - c:\programdata\elasticsearch\logs\* 

Remember about the lack of tabs in the code!

We will also use logstash to handle indexes:

 ### Elasticsearch as output #elasticsearch: # Array of hosts to connect to. # Scheme and port can be left out and will be set to the default (http and 9200) # In case you specify and additional path, the scheme is required: http://localhost:9200/path # IPv6 addresses should always be defined as: https://[2001:db8::1]:9200 #hosts: ["localhost:9200"] ... ### Logstash as output logstash: # The Logstash hosts hosts: ["ip_elk-server.ss.lu:5044"] 

Run:

 sudo /etc/init.d/filebeat start 

We look the information from Filebeat:

Very useful tool. Analyzes traffic between servers. Instantly detects errors. Analyzes protocols DNS, HTTP, MySQL, PostgreSQL, GLC, Memcache and others.

It is configured in the same way as Topbeat / Filebeat:

 sudo apt-get install libpcap0.8 sudo curl -L -O https://download.elastic.co/beats/packetbeat/packetbeat_1.1.0_amd64.deb sudo dpkg -i packetbeat_1.1.0_amd64.deb 

Edit the cofig (comment on Elasticsearch and set up Logstash)
output:

  #elasticsearch: #hosts: ["localhost:9200"] logstash: hosts: ["ip_elk-server.ss.lu:5044""] 

Go to the server and add an index for Packetbeat:

 sudo curl -XPUT 'http://localhost:9200/_template/packetbeat' -d@/etc/packetbeat/packetbeat.template.json 

Run:

 sudo /etc/ini.d/packetbeat start 

Download www.elastic.co/downloads/beats/winlogbeat . Unpack in C: \ and rename to Winlogbeat. Start PowerShell from admin and install the service:

 PS C:\Users\Administrator> cd 'C:\Winlogbeat' PS C:\Winlogbeat> .\install-service-winlogbeat.ps1 

If we see a message stating that scripts are disabled on the system by default (and it will be so), then we simply create a policy for Winlogbeat:

 PowerShell.exe -ExecutionPolicy UnRestricted -File .\install-service-winlogbeat.ps1 

 Security warning Run only scripts that you trust. While scripts from the internet can be useful, this script can potentially harm your computer. If you trust this script, use the Unblock-File cmdlet to allow the script to run without this warning message. Do you want to run C:\Program Files\Winlogbeat\install-service-winlogbeat.ps1? [D] Do not run [R] Run once [S] Suspend [?] Help (default is "D"): R Status Name DisplayName ------ ---- ----------- Stopped winlogbeat winlogbeat 

Before starting the service, we rule in the config - C: \ Winlogbeat \ winlogbeat.yml.

 output: #elasticsearch: # hosts: localhost:9200 logstash: hosts: ["ip_elk-server.ss.lu:5044"] 

The event_logs block lists the main system logs that need to be transported to Logstash:

 winlogbeat: registry_file: C:/ProgramData/winlogbeat/.winlogbeat.yml event_logs: - name: Application - name: Security - name: System logging: to_files: true files: path: C:/winlogbeat/winlogbeat/Logs level: info 

In event_logs, you can add other magazines, a list of which can be viewed as:

 PS C:\Users\Administrator> Get-EventLog * 

If the system is higher than Vista, then you can specify the channels:

 PS C:\Users\Administrator> Get-WinEvent -ListLog * | Format-List -Property LogName 

Next we need to upload the indexes for winlogbeat to the server as we did for topbeat, filebeat, packetbeat. This can be done remotely:

 PS C:\Winlogbeat> Invoke-WebRequest -Method Put -InFile winlogbeat.template.json -Uri http://IP_address_elk-server:9200/_template/winlogbeat?pretty 

There are problems with this method, you can do the following:

Create an index file on the server winlogbeat.template.json
sudo vi ~ / ELK / releases / beats / winlogbeat / winlogbeat.template.json. On the Windows client, open the file C: \ winlogbeat \ winlogbeat.template.json and copy its contents into the file ~ / ELK / releases / beats / winlogbeat / winlogbeat.template.json .

 { "mappings": { "_default_": { "_all": { "enabled": true, "norms": { "enabled": false } }, "dynamic_templates": [ { "template1": { "mapping": { "doc_values": true, "ignore_above": 1024, "index": "not_analyzed", "type": "{dynamic_type}" }, "match": "*" } } ], "properties": { "@timestamp": { "type": "date" }, "message": { "index": "analyzed", "type": "string" } } } }, "settings": { "index.refresh_interval": "5s" }, "template": "winlogbeat-*" } 

Then (on the server) we load this index on elasticsearch so that it can correctly analyze the information and provide it with the usual format:
Go to the directory where we have created the file winlogbeat.template.json .

 cd ~/ELK/releases/beats/winlogbeat ll  12 drwxr-xr-x 2 root root 4096 . 8 23:10 ./ drwxr-xr-x 7 root root 4096 . 8 16:00 ../ -rw-r--r-- 1 root root 729 . 8 23:10 winlogbeat.template.json #   sudo curl -XPUT 'http://localhost:9200/_template/winlogbeat' -d@winlogbeat.template.json 

The output should be:

{"acknowledged":true}

We go to the client and run the service winlogbeat. After this, we begin to monitor the data through Kibana, defining the view from the loaded indexes:



We look at dashboards: