Function Code Integrity Monitoring
In the process of developing a multicomponent automated testing system for the security scanner, we are faced with the problem of monitoring the integrity of the code of individual test functions and conducting audits.
The number of written functional tests that the system runs has already exceeded several thousand and continues to grow. In our case, one functional test is one function. With this development method, after assigning the “Ready” status to the test, it is permanently forgotten about it.
Meanwhile, in the process of developing other test functions, it is often necessary to refactor. Moreover, this process, due to the carelessness of an automated tester, can also affect already ready-to-run debugged tests.
')
In itself, refactoring any program, even if it affects a lot of modules and functions, is a common and quite useful thing. However, with respect to test functions, this is not always the case. Each test is designed to implement a specific validation algorithm. The logic of verification that the author laid, can be violated even with minor changes in the test code.
To avoid negative consequences of such situations, we have developed a mechanism for revising the code of test functions, with the help of which one can simultaneously control the integrity of functions and duplicate their code.
For each function, a revision, a set of hash and function code can be defined:
All critical functions can be added to the revision dictionary:
For us, for example, all functions with already developed tests are critical. All revisions can be stored in a special text file (revision file), which stores a list with the date of the last revision and the dictionary of revisions:
Before the next release of the testing system, the specialist responsible for the revision can track changes in the function code and, if necessary, restore the code of old tests in a short time by simply copying them from the revision.
Of course, there are alternative solutions to the problem, for example, code inspection and the use of tools in repositories (for example, GIT, SVN). However, inspections are useless if automatic changes are made to hundreds of tests, and tracking changes in code using repository tools after several merges is a laborious and lengthy process. In addition, unit tests are usually not written for test functions, but the need for quality control and unchanged functions is preserved - this problem can also be solved by the revision mechanism.
To implement the idea described above, a small module FileRevision.py was written in Python. The Revision () class that it contains can be imported into your project and add revisions for the functions you need.
Having slightly modified the module, you can additionally implement, for example, compression of the revision file, although for small projects it is not so critical.
The code is available by reference .
class Revision ():
To see examples of using this module, you just need to run it using Python 3.2.3:
python FileRevision.py
When you first start the script will detect the absence of a revision for the fake function implemented in the example, will offer to update information about it, clear the revision file, and also display information about the previous and new revisions. Then, a revision.txt file with revisions for examples will be created next to the .py file.
Thus, using our module and having an employee responsible for generating the code revision file will increase the degree of security of the test functions.
That's all for today. We are waiting for your questions and suggestions in the comments. Thanks for attention!
Author: Timur Gilmullin, Positive Technologies automated testing group.
The number of written functional tests that the system runs has already exceeded several thousand and continues to grow. In our case, one functional test is one function. With this development method, after assigning the “Ready” status to the test, it is permanently forgotten about it.
Meanwhile, in the process of developing other test functions, it is often necessary to refactor. Moreover, this process, due to the carelessness of an automated tester, can also affect already ready-to-run debugged tests.
')
In itself, refactoring any program, even if it affects a lot of modules and functions, is a common and quite useful thing. However, with respect to test functions, this is not always the case. Each test is designed to implement a specific validation algorithm. The logic of verification that the author laid, can be violated even with minor changes in the test code.
To avoid negative consequences of such situations, we have developed a mechanism for revising the code of test functions, with the help of which one can simultaneously control the integrity of functions and duplicate their code.
Mechanism
For each function, a revision, a set of hash and function code can be defined:
(func_hash, func_source) All critical functions can be added to the revision dictionary:
{"funcName1": (funcName1_hash, funcName1_source), "funcName2": (funcName2_hash, funcName2_source), ...} For us, for example, all functions with already developed tests are critical. All revisions can be stored in a special text file (revision file), which stores a list with the date of the last revision and the dictionary of revisions:
[revision's last date-n-time, {revisions}] Before the next release of the testing system, the specialist responsible for the revision can track changes in the function code and, if necessary, restore the code of old tests in a short time by simply copying them from the revision.
Of course, there are alternative solutions to the problem, for example, code inspection and the use of tools in repositories (for example, GIT, SVN). However, inspections are useless if automatic changes are made to hundreds of tests, and tracking changes in code using repository tools after several merges is a laborious and lengthy process. In addition, unit tests are usually not written for test functions, but the need for quality control and unchanged functions is preserved - this problem can also be solved by the revision mechanism.
Code
To implement the idea described above, a small module FileRevision.py was written in Python. The Revision () class that it contains can be imported into your project and add revisions for the functions you need.
Having slightly modified the module, you can additionally implement, for example, compression of the revision file, although for small projects it is not so critical.
The code is available by reference .
Module implementation
class Revision ():
__init __ () # Initialization of parameters
def __init__(self, fileRevision='revision.txt'): self.fileRevision = fileRevision self.mainRevision = self._ReadFromFile(self.fileRevision) # get main revision first _ReadFromFile () # Getting revisions from a file
def _ReadFromFile(self, file=None): """ Helper function that parse and return revision from file. """ revision = [None, {}] if file == None: file = self.fileRevision try: if os.path.exists(file) and os.path.isfile(file): with open(file) as fH: revision = eval(fH.read()) except: traceback.print_exc() finally: return revision _WriteToFile () # Write revisions to file.
def _WriteToFile(self, revision=[None, {}], file=None): """ Helper procedure than trying to write given revision to file. """ status = False if file == None: file = self.fileRevision try: with open(file, "w") as fH: fH.write(str(revision)) status = True except: traceback.print_exc() finally: return status _GetOld () # Getting the previous revision for the function.
def _GetOld(self, func=None): """ Get old revision for given function and return tuple: (old_hash, old_source). """ funcHashOld = None # old code is None if function not exist in previous revision funcSourceOld = None # old hash is None if function not exist in previous revision try: if func.__name__ in self.mainRevision[1]: funcHashOld = self.mainRevision[1][func.__name__][0] # field with old hash of function funcSourceOld = self.mainRevision[1][func.__name__][1] # field with old code of function except: traceback.print_exc() finally: return (funcHashOld, funcSourceOld) _GetNew () # Getting a new revision for a function.
def _GetNew(self, func=None): """ Get new revision for given function and return tuple: (new_hash, new_source). """ funcSourceNew = None # if function doesn't exist, its also doesn't have code funcHashNew = None # hash is None if function not exist try: funcSourceNew = inspect.getsource(func) # get function's source funcHashNew = hash(funcSourceNew) # new hash of function except: traceback.print_exc() finally: return (funcHashNew, funcSourceNew) _Similar () # Compare two revisions.
def _Similar(self, hashOld, sourceOld, hashNew, sourceNew): """ Checks if given params for modified then return tuple with revision's diff: (old_revision, new_revision), otherwise return None. """ similar = True # old and new functions are similar, by default if hashNew != hashOld: if sourceOld != sourceNew: similar = False # modified if hashes are not similar and functions not contains similar code return similar Update () # Update revision for the specified function.
def Update(self, func=None): """ Set new revision for function. revision = [revision date-n-time, {"funcName1": (funcName1_hash, funcName1_source), {"funcName2": (funcName2_hash, funcName2_source), ...}] """ status = False if func: try: funcSourceNew = inspect.getsource(func) # get function's source funcHashNew = hash(funcSourceNew) # new hash of function revisionDateNew = datetime.now().strftime('%d.%m.%Y %H:%M:%S') # revision's date funcRevisionNew = {func.__name__: [funcHashNew, funcSourceNew]} # form for function's revision self.mainRevision[0] = revisionDateNew # set new date for main revision self.mainRevision[1].update(funcRevisionNew) # add function's revision to main revision if self._WriteToFile(self.mainRevision): # write main revision to file status = True except: traceback.print_exc() finally: return status DeleteAll () # Delete all revisions from the file.
def DeleteAll(self): """ Helper function that parse and return revision from file. """ status = False try: self.mainRevision = [None, {}] # clean revision if self._WriteToFile(self.mainRevision): # write main revision to file status = True except: traceback.print_exc() finally: return status ShowOld () # Display information about the previous revision for the function.
def ShowOld(self, func=None): """ Function return old revision for given function. """ funcHashOld, funcSourceOld = self._GetOld(func) # get old revision for given function dateStr = "Last revision: " + str(self.mainRevision[0]) hashStr = "\nOld function's hash: " + str(funcHashOld) codeStr = "\nOld function's code:\n" + "- " * 30 + "\n" + str(funcSourceOld) + "\n" + "- " * 30 oldRevision = dateStr + hashStr + codeStr return oldRevision ShowNew () Display information about the new revision for the function.
def ShowNew(self, func=None): """ Function return old revision for given function. """ funcHashNew, funcSourceNew = self._GetNew(func) # get old revision for given function hashStr = "New function's hash: " + str(funcHashNew) codeStr = "\nNew function's code:\n" + "- " * 30 + "\n" + str(funcSourceNew) + "\n" + "- " * 30 newRevision = hashStr + codeStr return newRevision Diff () # Comparing revisions and deriving diffs for a function if necessary.
def Diff(self, func=None): """ Checks if given function modified then return tuple with revision's diff: (old_revision, new_revision), otherwise return None. """ funcHashOld, funcSourceOld = self._GetOld(func) # get old revision for given function funcHashNew, funcSourceNew = self._GetNew(func) # get new revision for given function # check old and new revisions: if self._Similar(funcHashOld, funcSourceOld, funcHashNew, funcSourceNew): diff = None # not difference else: diff = ("Last revision: " + str(self.mainRevision[0]) + "\nOld function's hash: " + str(funcHashOld) + "\nOld function's code:\n" + "- " * 30 + "\n" + str(funcSourceOld) + "\n" + "- " * 30, "\nNew function's hash: " + str(funcHashNew) + "\nNew function's code:\n" + "- " * 30 + "\n" + str(funcSourceNew) + "\n" + "- " * 30) # if new function not similar old function return diff _testFunction () # Fake function to test the module
def _testFunction(a=None): """ This is fake test function for module. """ # this is comment if a: return True else: return False if __name__ == '__main__' :() # Examples of using the module when it is launched separately.
func = _testFunction # set function for review in revision revision = Revision('revision.txt') # init revision class for using with revision.txt # how to use this module for review revision of function: print(MSG_CHECK, func.__name__) funcModified = revision.Diff(func) # get function's diff as tuple (old_revision, new_revision) if funcModified: print(MSG_MODIFIED) print(funcModified[0]) # old revision print(funcModified[1]) # new revision else: print(MSG_NOT_MODIFIED) # how to use this module for update revision: action = input("Update function's revision? [y/n]: ") if action == 'y': print(MSG_UPDATE, func.__name__) if revision.Update(func): print(MSG_UPDATED) else: print(MSG_UPDATE_ERROR) # how to use this module for clean file-revision: action = input("Clean file-revision now? [y/n]: ") if action == 'y': print(MSG_DELETE) if revision.DeleteAll(): print(MSG_DELETED) else: print(MSG_DELETE_ERROR) # how to use this module for show old review: action = input('Show old revision for function? [y/n]: ') if action == 'y': print(revision.ShowOld(func)) # how to use this module for show new review: action = input('Show new revision for function? [y/n]: ') if action == 'y': print(revision.ShowNew(func)) To see examples of using this module, you just need to run it using Python 3.2.3:
python FileRevision.py
When you first start the script will detect the absence of a revision for the fake function implemented in the example, will offer to update information about it, clear the revision file, and also display information about the previous and new revisions. Then, a revision.txt file with revisions for examples will be created next to the .py file.
Thus, using our module and having an employee responsible for generating the code revision file will increase the degree of security of the test functions.
That's all for today. We are waiting for your questions and suggestions in the comments. Thanks for attention!
Author: Timur Gilmullin, Positive Technologies automated testing group.
Source: https://habr.com/ru/post/177189/
All Articles