Facebook's Osquery Introduction

Foreword

This publication is a relatively free translation of the article “Introducing osquery” published on the Facebook blog.

Introduction to osquery

Monitoring the status of your system in real time is very important. In Facebook, we developed a framework called osquery, which allows you to look at low-level monitoring of the operating system in a new way.

Osquery represents the operating system as a high-performance relational database. This approach allows you to write SQL queries to easily and efficiently retrieve information about your system. With osquery, the current state of the OS is represented as SQL tables from which you can get information about:
')

• running processes;
• loaded kernel modules;
• open stevia compounds.

SQL tables are created through an easily extensible API. Several tables already exist and many more are being developed. To better understand the ease and expressiveness that osquery provides, the following examples will help.

Query examples

The first example illustrates how you can use osquery to interact with processes running on the current system. Specifically, this query returns the processes that are currently running. The WHERE clause in this query returns only those processes that were launched from binaries but are no longer in the file system at the moment. This is a common practice that hackers resort to, so this query will not return any results on your system, provided that your system has not been compromised.

SELECT name, path, pid FROM processes WHERE on_disk = 0; 

Interacting with the OS using SQL is a simple and entertaining exercise. One of the advantages that SQL gives us is the ability to join different tables together for system analysis. The following example shows us how it is possible to simultaneously use information from two “tables” - listening_ports and processes. This query looks for all processes that are listening to network ports. Thus, using the “table” of processes from the previous example, we can combine it with another “table” - the network ports table. Both there and there the PID of the process is used, by which the merging takes place.

 SELECT DISTINCT process.name, listening.port, listening.address, process.pid FROM processes AS process JOIN listening_ports AS listening ON process.pid = listening.pid; 

Osquery includes many tables and many more are created by us daily. The tables are easy to write, so we don’t mind and even encourage the development of new tables by third-party developers. Detailed information can be found in our wiki .

Features

Osquery is the framework we use to create new products and tools. The osquery modular source code gives us an edge over existing concepts. We release several tools as part of one open source release and we have a lot more in our plans. We also look forward to the moment when our community will present its tools based on osquery.

Interactive console

The osqueryi interactive console is a SQL interface where you can execute your queries and examine your OS. With all the expressive power of SQL and the many useful tables built into osquery, the console is an invaluable tool for diagnosing OS problems, solving performance problems, and more.

More information on using osquery is in our wiki .

Cluster Monitoring

To monitor large systems, we have a demon, osqueryd. It allows you to schedule requests to execute them across your entire infrastructure. The daemon independently takes care of the aggregation of the results of requests by time and generates logs showing the status of changes in the infrastructure. You can use it to update the security, performance, configuration, and integrity status of your infrastructure. The osqueryd daemon can also integrate your internal logs using a robust plugin architecture.

If you are interested in using osqueryd in your infrastructure, we again invite you to the wiki , as well as to the internal deployment guide .

Cross Platform

Osquery is cross-platform. And while osquery has an advantage in low-level operating systems, you can build and use it in Ubuntu, CentOS and MacOS. It will even give a definite advantage, since will allow to monitor corporate poppies at the same level as corporate Linux servers.

Native builds and documentation

To facilitate deployment, osquery comes as a regular package for all supported operating systems. There is also detailed documentation on creating your own packages. Therefore, developing and implementing your own osquery tools should be as easy as possible.
Osquery was designed to take into account the characteristics of the environments used in order to enable hot plug-in replacement in an already running system. Using the provided interfaces allows you to more deeply integrate osquery into your infrastructure if one or more of the plugins used no longer do not meet your needs.

Read more here .

Modular source code

Osquery consists of high-performance modular components with a well-documented public API. These components can be easily put together to create new, interesting applications and tools. Details on the API here .

Open source

After talking with several external companies, it came to be understood that such monitoring of the low-level behavior of operating systems is not a unique problem for Facebook. A few months later we released osquery as binary files for a limited number of companies. They successfully implemented and tested osquery on their equipment, and we received excellent feedback from them.
And now we are pleased to announce that the time has come for open source osquery. You can find all the code and documentation on github .

We look forward to feedback from the community. We will do all the work with osquery on github. This will facilitate the work of third-party developers. We hope you see the potential in osquery and do wonderful things with us.