Implementing tools to create OVAL® content in Python

Greetings, colleagues!
In the process of researching the OVAL language, about which I wrote earlier in one of my articles , and the concept of a SCAP scanner, I faced a rather serious problem, namely the lack of convenient tools for creating content in the OVAL language . No, I do not claim that there is nothing at all. There is a small set of utilities presented on the official site. Most of them are paid, the rest are not very convenient solutions, most of all similar to XML-Notepad . In the end, I decided to create the small tool I needed to work on my own, using Python as the language.



Such a choice is due to the good reputation of the Python language as “rapid development language” and the presence of a rich palette of third-party libraries. Armed with language documentation, I tried to recreate MITER’s ideas in reality. The final goal for me was to set up the implementation of OVAL objects and their storage and indexing systems.
')
The primary issue was the choice of method of storing the collected information. The first (and unsuccessful) solution was SQLite 3 . I will not particularly linger on this sad experience, but experience has shown that storing an unpredictable tree in a relational database is too difficult for me. Therefore, I drew attention to the NoSQL database . Since I was planning to restrict myself to the exclusive use of the base, my choice fell on ZoDB . Of course, this decision was also hasty. But it was too late to rework on MongoDB . So for further code calculations, I would like to note that PersistentMapping () is, in fact, the usual dict () . Taking into account the uncertainty of the perception of the hashmaps database, as a hashmap, I used the keys of the same class. PersistentList () is the equivalent of list () . This replacement is necessary for storing such structures in ZoDB. This is due to the internal logic of the database. To save a class in the database, it is entirely necessary that the class be the heir of the persistent.Persistent class.

In addition, I ask you in advance to forgive me for the non-template naming of variables, methods and classes and the possible "kosawkost": I am not a wizard, I am just learning. And I will gladly accept any edits and comments from more knowledgeable colleagues.

In order to avoid quoting unnecessarily large pieces of code, I immediately cite a link to the source code on code.google.com .

In general, when analyzing the structures of the OVAL language, I noted that its main feature is extensibility. If you need to enter your own structure, it is enough to describe it in the XSD, which defines your namespace. Since the structures for the most part represent typical data sets, I decided to create a dedicated class for each language object. The main goal of such an implementation is to create a container for information that will be conveniently stored in the database. Thus, the OBJECT element was used as the base object:

# UNIVERSAL OVAL OBJECTS
class oval_object ( persistent. Persistent ) :
def __init__ ( self ) :
self . id = str ( )
self . tag = "object"
self . namespace = str ( )
self . oval = str ( )
self . version = 0
self . vHash = 0
self . deprecated = False
self . variables = PersistentList ( )
self . notes = PersistentList ( )
self . comment = str ( )
self . signature = str ( )

def assign_id ( self , UniqId ) :
self . id = "oval:" + self . oval + ": obj:" + str ( UniqId )

def hash ( self ) :
summ_hash = 0
summ_hash + = hash ( self . tag )
summ_hash + = hash ( self . namespace )
summ_hash + = hash ( self . comment )
for var in self . variables :
if isinstance ( var, variable ) :
summ_hash + = var. hashme ( )
else :
raise Exception ( "Error input parameters in object variables" )
return summ_hash

def hashme ( self ) :
self . vHash = self . hash ( )
return self . vHash

def clearme ( self ) :
hashmap = PersistentMapping ( )
for var in self . variables :
var. clearme ( )
var_hash = var. hashme ( )
if var_hash not in hashmap. keys ( ) :
hashmap [ var_hash ] = None
else :
self . variables . remove ( var )

This class is the base class for all major OVAL elements. Definition , Test , State and Oval_variable will be the heirs of this class. In order to realize the possibility of comparing elements, the methods hash () (universal for all) and hashme () (redefined by descendants) were created.
As can be seen from the names of the variables, all the basic structures are given statically. Dynamic are the possible "descendants", the structures of which we do not know. To implement such "descendants", I made the class Variable , largely repeating according to the Element ideology from the ElementTree :

class variable ( persistent. Persistent ) :
def __init__ ( self , tag = str ( ) , body = str ( ) , attributes = None , variables = None ) :
if not attributes:
self . attributes = PersistentMapping ( )
if not variables:
self . variables = PersistentList ( )
if attributes and not isinstance ( attributes, PersistentMapping ) :
attributes = PersistentMapping ( attributes )
if variables and not isinstance ( variables, PersistentList ) :
variables = PersistentList ( variables )
self . tag = tag
self . body = body
if attributes:
self . attributes = attributes
if variables:
self . variables = variables
self . vHash = 0

def clearme ( self ) :
hashmap = PersistentMapping ( )
for var in self . variables :
var. clearme ( )
var_hash = var. hashme ( )
if var_hash not in hashmap. keys ( ) :
hashmap [ var_hash ] = None
else :
self . variables . remove ( var )

def hashme ( self ) :
summ_hash = 0
summ_hash + = hash ( self . tag )
summ_hash + = hash ( self . body )
if self . attributes :
for attribute in self . attributes . keys ( ) :
summ_hash + = hash ( attribute ) + hash ( self . attributes [ attribute ] )
if self . variables :
for var in self . variables :
if isinstance ( var, variable ) :
if var ! = self :
summ_hash + = var. hashme ( )
else :
raise Exception ( "Error input parameters in variables. Self append?" )
self . vHash = summ_hash
return self . vHash

As with OBJECT , the hash calculation function was required. The class structure is maximally tailored to ElementTree.Element to simplify the future uploading of the database to the XML format.

In general, a set of objects that are convenient to fill out. But it is not enough to fill them: when creating your own objects, it is necessary to fulfill the requirements of the regulator (in our case MITRE ) for filling and storing new Definition . Accordingly, I needed an indexing and change control system for the Definition version. For this, the oval_suite () class was created. It implements a basic set of methods ( put , get , delete , search , import_xml , export_xml ), which facilitates work with content and allows you not to bother with the aforementioned problems.

One of the specific problems I encountered in the process of creating oval_suite () is the lack of memory when trying to export a large amount of data (about 100MB) to XML format in one go. Therefore, to implement this function, we had to create the final file in stages and use the ElementTree.Element function for deleting elements, borrowed from StackOverflow :

def _garbager ( self , root ) :
for element in list ( root ) :
self ._garbager ( element )
root clear ( )
del root

In general and in general, we have sketched tools for working with OVAL in Python, which can be useful in building and creating security content. This module allows you to build security content and “package” it into a database with version control and unique identifiers.

Thanks for attention! I hope someone will be interested in the topic of OVAL and these developments will be useful to him.