Common Event Format inside

One by one, manufacturers of software and hardware and software solutions claim to receive a certificate confirming support of the HP ArcSight Common Event Format (CEF) format: Stonesoft , Tripwire , Citrix , Imperva , NetScout and a few dozen vendors ...


')

About SIEM

In accordance with the annual reports of Gartner, HP ArcSight (until September 2010 - ArcSight) confidently holds among the leaders in developing solutions of the SIEM class. In short, the essence of such decisions is to collect in one place hundreds of thousands of millions of events generated by various information security systems and to make a correlation analysis of these events. The result of the correlation are security incidents with which a person is already working - an administrator or operator.

Each manufacturer of SIEM solutions in their advertising brochures usually indicates the number of supported event sources. For example, it is claimed that ArcSight ESM supports 300+ devices and applications, while QRadar SIEM has “only” 200+. Who is there and according to sources, it does not matter now, because within this article we will be interested in the format of information security events (journals).

About magazines

There are no general requirements for the structure of IS logs and events yet. Therefore, each developer draws magazines as it is more convenient for him.

Someone simply writes them to a text file, someone provides the ability to send data to a Syslog server. One only needs SNMP, and the other wants to put all the logs in a relational database. There is Microsoft with its .evt format, CheckPoint with its OPSEC. And do not forget about SDEE.

About the problem

And the structure of the logs, and the protocols used for their transmission are different everywhere. From the point of view of integration with SIEM solutions (and not only with them), this is bad. Because for a unified log processing, they need to be normalized, i.e. lead to a single format. In a single event format easier to store. Easier to search for data and generate reports.

Normalization means parsing and mapping. Therefore, each connector between the IB system and the SIEM solution is an application that parses and maps in accordance with its configuration settings.
If the SIEM manufacturer does not have the support of the product you need, then you will have to perform the configuration settings yourself. Develop and test regular expressions. Study the database structure, master SQL * Plus. Write matching rules. Sad, in general ...

Proposed solution

“It would be nice if all the vendors of the information security systems alienated their logs in a way that everyone could understand,” thought ArcSight and in 2006 presented the CEF format they had developed for general approval . In my opinion, in this format there is nothing difficult. Just a few requirements need to be met:

Requirement # 1 - Use Syslog as a transport

Here, seemingly, it is clear.
I want - I use the UDP protocol, I want - TCP.

Requirement number 2 - fill eight required fields

Of course, the traditional syslog header should be present at the beginning of the message.
Jan 18 11:07:53 host

And behind it - the prefix CEF: and a set of required fields, separated by the symbol "|"
CEF:Version|Device Vendor|Device Product|Device Version|Signature ID|Name|Severity|Extension

Here:

• Version - CEF version
• Device Vendor, Device Product and Device Version - these lines uniquely identify the source of events. There are no products that have the same set of these three values.
• Signature ID - a unique identifier of the event type
• Name - clear to the person description of the event
• Severity - the importance of the event (from 0 to 10)
• Extension - see below

Requirement 3 - Extension field is filled in accordance with the CEF dictionary

The Extension field is a set of key-value pairs. I will give a few keys from the dictionary:

dmac is the Destination Mac Address (for example, 00: 0D: 60: AF: 1B: 65)
spt is the Source Port (port from 0 to 65535)
request is the Request URL (in the case of HTTP-request, the URL is specified)

In the description of the CEF format, the dictionary is given in full, indicating the type of data for each key and the maximum allowable size.

At the end of the document are some more general requirements. In particular, the UTF-8 encoding should be used for the entire message. It also contains the rules for the design of certain special characters and multi-line entries.

As a result, the message drawn up in exact accordance with the CEF format should look like this:
May 29 15:26:33 host CEF:0|McAfee|Antivirus|5.2|100|worm successfully stopped|10|src=10.0.0.1 dst=2.1.2.2 spt=1232

What is the benefit?

1. Connecting a new event source turns into “plug and play”
2. Reduces the load on the SIEM system itself
3. Manufacturers of SIEM solutions are no longer measured by the number of supported sources and pay more attention to other aspects (product usability, resiliency, etc.)
4. And in general, this format was invented not only for SIEM. It is already used in other solutions related to processing logs.