Writing Arcsight FlexConnector. Log file

Currently, solutions for collecting and analyzing information security events, SIEM class systems are at the peak of their popularity, modern companies set themselves the task of connecting as many of their key systems to SIEM as possible to consolidate data from these systems, analyze events and analyze information security incidents. based on the data obtained and analyzed. SIEM collects events from virtually any source: network devices, databases, various business applications, information security systems and information technologies, bringing them to a uniform form suitable for further analysis and work with the information received.

Taking into account how many different information systems can be discussed, you involuntarily think about possible options and approaches for connecting possible information systems to SIEM and my desire to share this knowledge with colleagues who are closely connected and working with systems of this class. As an example, let's take the most popular Arcsight system and consider possible options for connecting to company information systems using the FlexConnector.



In this article I want to present a step-by-step instruction for the interaction of FlexConnector for the LOG-FILE connection option in three possible ways.
')
In the beginning, let's imagine that there is an internal application that writes all the events in a separate file called “mytest_log.txt”. These events need to be sent to Arcsight ESM for further correlation, but the problem is that there is no necessary out-of-the-box SmartConnector connector. In this question, the FlexConnector type connector can always help out, which allows you to correctly send events to the Arcsight ESM / Logger.

Here is a clear example of the incoming events from the source in our file “mytest_log.txt”:

2017-08-01 01:18:00.579 INFO AUTH: LOGIN SUCCESS. USER: Ivanov. IP: 192.168.0.1
2017-08-01 01:19:13.246 INFO AUTH: LOGIN FAILED. USER: Petrov. IP: 192.168.3.13
2017-08-01 01:20:17.589 INFO AUTH: LOGIN FAILED. USER: Petrov. IP: 192.168.3.13
2017-08-01 01:21:14.646 INFO AUTH: LOGIN FAILED. USER: Petrov. IP: 192.168.3.13
2017-08-01 01:22:09.179 INFO AUTH: LOGIN SUCCESS. USER: Ivanov. IP: 192.168.0.1
2017-08-01 01:23:02.116 INFO AUTH: LOGIN SUCCESS. USER: Petrov. IP: 192.168.3.13

The next step is to install the Arcsight SmartConnector and write the FlexConnector to the current log file.


You can write a FlexConnector under the LOG File in 3 ways:

1. Writing a configuration file in a text editor

   The most popular way at the present time is to write connectors using a text editor.
   
   In the $ARCSIGHT_HOME\current\user\agent\flexagent we create the $ARCSIGHT_HOME\current\user\agent\flexagent configuration file.
   
   Sample of the finished file:
   
   #MyTest MyParser Configuration File
replace.defaults=true
trim.tokens=true
comments.start.with=#

#2017-08-01 01:18:00.579 INFO AUTH: LOGIN SUCCESS. USER: Ivanov. IP: 192.168.0.1

regex=(.*)\\sINFO\\sAUTH\\D\\s(.*)\\D\\sUSER\\D\\s(.*)\\D\\sIP\\D\\s(.*)

token.count=4

token[0].name=Date
token[0].type=TimeStamp
token[0].format=yyyy-MM-dd HH:mm:ss.SSS
token[1].name=Name1
token[1].type=String
token[2].name=Username
token[2].type=String
token[3].name=UserAddress
token[3].type=IPAddress

additionaldata.enabled=false

event.deviceVendor=__stringConstant(MyTest)
event.deviceProduct=__stringConstant(MyParser)
event.endTime=Date
event.name=Name1
event.sourceUserName=Username
event.sourceAddress=UserAddress

   The file structure is very simple.
   
   

   • Writing input parameters for the connector
     
     #MyTest MyParser Configuration File
replace.defaults=true
trim.tokens=true
comments.start.with=#
     
   • We write the regular expression for our log
     
     #2017-08-01 01:18:00.579 INFO AUTH: LOGIN SUCCESS. USER: Ivanov. IP: 192.168.0.1
regex=(.*)\\sINFO\\sAUTH\\D\\s(.*)\\D\\sUSER\\D\\s(.*)\\D\\sIP\\D\\s(.*)
     
   • Specify the number of keys (counting starts from “0”)
     
     token.count=4
     
   • We describe the keys (name, type, format)
     
     token[0].name=Date
token[0].type=TimeStamp
token[0].format=yyyy-MM-dd HH:mm:ss.SSS
token[1].name=Name1
token[1].type=String
token[2].name=Username
token[2].type=String
token[3].name=UserAddress
token[3].type=IPAddress
     
   • Specify maping fields and submessage
     
     event.deviceVendor=__stringConstant(MyTest)
event.deviceProduct=__stringConstant(MyParser)
event.endTime=Date
event.name=Name1
event.sourceUserName=Username
event.sourceAddress=UserAddress
     

   
   This method is the easiest to write and the most popular to use among Arcsight specialists.
   
   Of the minuses, I can only note the search for errors, but this problem is solved by launching the connector in the “Leave as a standalone application” state where we will see all the technical information.
   
2. Using the REGEX Embedded Utility

   The Arcsight SmartConnector has a built-in regex utility that runs the GUI to create or test the FlexConnector.
   
   

   1. We start the console on behalf of the administrator and go to the installation directory of our FlexConnector.
      Run the regex utility: C:\$ARCSIGHT_HOME\current\bin> .\arcsight.bat regex
      
      
      
      This utility allows you to create a configuration file for our connector.
   2. Create a new file: File - New FlexAgent Regex File. The configuration file should be stored in the $ARCSIGHT_HOME\current\user\agent\flexagent with the file extension FILE_NAME.sdkrfilereader.properties.
   3. Connect our event source: File - Load Log File. You can specify the path to the real logs or test.
   4. It is necessary to write a regular expression for our log. This can be done in the Regex field (for convenience, there is a Generate button).
      My regular season is: (.*)\sINFO\sAUTH\D\s(.*)\D\sUSER\D\s(.*)\D\sIP\D\s(.*)
      Click Test.
   5. It is necessary to give names to our groups.
      
      
   6. All groups by default are of type String, therefore for groups Date and UserAddress you must specify the type TimeStamp and IPAddress.
      We also describe the time format for the Date group (detailed information in the FlexConnector Guide).
      
      
   7. Next, we do the mapping of fields by dragging groups and selecting fields to display in Arcsight ESM.
      
      
      
      The regex utility does not allow to select the event.sourceAddress field for IPAddress, therefore, it needs to be added by hand:
      
      event.sourceAddress=UserAddress
      
      Also required fields are Device Product and Device Vendor. Add the following lines to the configuration file:
      
      event.deviceVendor=__stringConstant(MyTest)
event.deviceProduct=__stringConstant(MyParser)
      
   8. Click Test and check that all events have gone bad. Save our configuration: File - Save FlexAgent Regex File.

   
   We launch the connector, generate new events in the log-file and check the arrival of events in Arcsight ESM.
   
   
   
3. Using Arcsight Quick Flex

   While writing this article, I first took advantage of Arcsight Quick Flex and was pleasantly surprised. This software allows you to write a configuration file quickly and without errors, and writing submessage turns into a pleasure.
   
   

   1. Create a new project:
      Vendor: MyTest
      Product: MyParser
      Version: 1
      Log File Path: specify the directory with logs
      Project Directory: specify the project directory (not the configuration file !!!)
      
      
   2. After uploading the logs, the events have the status “Base Unparsed”. Launch the “Base Regex Editor” tab.
      
      
   3. In the field “Base Regex” we specify our regular expression.
      
      
      
      You can check the regular expression correctness by clicking the “Matching details” button.
      
      
   4. Click Tokenize and save the settings.
      
      Next, for each key, we indicate the type, format, description and field for the map.
      
      
   5. Save and return to the main page by clicking Log View. Then click Refresh and all our events become “Base Parsed”
      
      
   6. Now click Generate Parser and get the ready configuration file for the FlexConnector. Export the file to the folder with the connector:
      
      $ARCSIGHT_HOME\current\user\agent\flexagent
      
      

   
   We launch the connector, generate new events in the log-file and check the arrival of events in Arcsight ESM.
   
   

Bottom line: The article indicated the basic skills for writing the FlexConnector, but if you are interested in such things as categorization, submessage, mapping, etc., then I will be ready to describe them in future articles.