SPLUNK VS ELK?

If you are connected with the operation of IT, then you probably have come across either Splunk, or ELK, or both products. These are the two main players in the market for log management products and operational data analytics.

In our blog, we write about Splunk and we are often asked the question, why is Splunk better than ELK? What should we pay money for a license if there is a good open source competitor? On this topic excerpts in the comments have already said a lot, but we decided to combine everything and devote a separate article to this issue.

For those unfamiliar with Splunk or ELK , talk a little about them.
')

Splunk

It is a proprietary machine data tool that offers users, administrators, and developers the ability to instantly receive and analyze all data created by applications, servers, network devices in the IT infrastructure, and any other machine data.

Splunk Enterprise obtains machine data and turns it into powerful operational analytics, providing real-time information using charts, alerts, reports, etc.

ELK

The ELK stack consists of open source components : elasticsearch (text search tool), Logstash and Beats (data sending tools), and Kibana (data visualization tool). Together they provide a fully working tool for real-time data analysis. Despite the fact that all of them are designed to work together, each of them is a separate project.

ELK provides similar functions for real-time analysis, such as creating reports, alerts, searching logs, etc.

Open source means cheaper?

Often, at a subconscious level, we identify open source with free solutions, but we must remember that the price of a project is far from just the cost of a license, but also from many other costs , for example:

• Cost of equipment;
• salary of employees;
• Costs of implementation and integration;
• Expenses for additional features;
• Support costs;
• Risk fee;
• Etc.

Often it may be that the cost of a license to proprietary solutions, partially or fully pay off by reducing other cost items compared to open source.

License

In Splunk, the license price is calculated from the number of gigabytes of data downloaded per day, regardless of the number of nodes or users. When buying a license of any size, you immediately buy all the built-in functionality, the ability to use for free a large (more than 1000) base of applications and instructions documentation. There is also a free license that allows you to download up to 500 MB per day, but has several functional limitations.

ELK uses a model close to the Freemium pricing model , which only contains basic functions for free, and you have to pay for the rest. If you want to configure security, alerts, reports, use machine learning algorithms, etc., then you need to purchase an annual subscription to the X-pack package, the cost of which will be calculated based on the number of nodes in the system .

Required equipment

Is there a difference in the required equipment? Yes, there is a significant one. One of the features of data storage in ELK is that, along with the raw data, the same data is stored into fields and all sorts of added fields and geotags. All this can increase the required storage space on disks up to 10 times compared to Splunk, which stores only compressed raw data, and all other information is attached only in the search process. And in general, if we take and put in parallel Splunk and ELK, then for normal operation ELK will require much more resources than Splunk.

Implementation rate

In addition to the obvious equipment costs, there are implicit costs or the so-called loss of profits during the implementation of the system . If we compare the deployment process and the required resources: temporary, human or technological, then Splunk significantly benefits ELK.

By installing one instance of Splunk, you immediately get a ready-made solution that includes the ability to download data and indexing, and a search interface, more than 1000 applications created and supported by the company that can automate the loading or analysis of certain data types. After installing one instance, you can immediately begin to solve the tasks. If necessary, to improve performance and fault tolerance, the architecture can be expanded to a cluster in which each instance of Splunk will be assigned its own role: Indexer, Search Head, Cluster Master, we have written here how to deploy the cluster.

In the case of ELK, the tool consists of separate modules, such as ElasticSearch, which is responsible for storing data, Kibana is a user interface for searching, viewing, and visualizing, and Logstash is tools for sending data to ElasticSearch. To get the first results, you need to configure all the components , as well as for various additional features, you need to either write add-ons yourself, or fasten add-ons already written by someone, whose support and quality are often not responsible or responsible.

From the point of view of data loading , ElasticSearch requires the definition of partitioning rules, the identification of all fields, the separation of data before they are loaded. That is, you should know everything about the data before you upload it to the system. Otherwise, you will have to change the data structure using fairly complex JSON queries. This significantly slows down the process. In Splunk you can load absolutely any data, initially it will determine only 3 parameters for them: host, source and sourcetype, and then you can select the fields and change them at any time when you understand exactly what you need from this data.

Tongue

Splunk uses its SPL query language, which combines the capabilities of the SQL language with the Unix pipelined syntax. SPL allows you to search by data and filter, transform data and add new fields, work with time series, build visualizations and apply machine learning algorithms.

ELK uses different languages: Lucene for text search, JSON for dividing data into objects, and Timelion for working with time series.

ELK proponents note that it is more convenient to use already known languages ​​than to learn a new one created specifically for one product. Although, frankly, SPL - the language is quite simple and has many similar syntax with other languages, and also has detailed documentation that will allow you to quickly deal with it.

An interesting fact: in the latest versions of ELK, the experimental language Kuery appeared , created specifically for Kibana. The developers note that in future releases the language may be completely changed or deleted, but in general, the tendency towards specialization of the language for a particular product is noticeable.

Access control

From a security point of view, access control is an important element of the system. The ability to split users into roles with different levels of access is available from the Splunk boxed solution; in ELK, a similar tool is available when subscribing to the X-Pack package.

Risks

ELK = elasticsearch + logstash + kibana. These are three different enterprises (“projects”) that have a symbiotic relationship. Reliance on three different open source enterprises for a single solution carries significant operational and legal risk to the company.

Splunk, Inc. is a large publicly registered company with high turnover and a stable position in the securities market . Therefore, Splunk can be called a viable and stable corporate partner, representing a small risk for the company .

Conclusion

In general, both Splunk and ELK are powerful operational analytics tools that have their pros and cons. When deciding which tool to choose, it is necessary to carefully evaluate all the requirements for the project, the tasks set and, on the basis of them, calculate the costs of implementation and subsequent support, as well as assess their readiness and ability to implement each tool.

We are looking forward to your comments on this topic!