Full code coverage

Whether it is necessary to do full code coverage with tests is quite a frequent and ambiguous topic when discussing unit testing. Although most developers are inclined to think that it is not necessary to do it, that it is inefficient and useless, I hold the opposite opinion (at least when developing in Python). In this article, I will give an example of how to make full code coverage, and describe the disadvantages and benefits of full coverage based on my development experience.

Nose testing tool

For unit testing and statistics collection we use nose . Its advantages compared with other means:

• No need to write additional code for tying unit tests.
• Embedded tools for metrics, in particular to calculate the percentage of coverage
• Compatible with Python 3 (py3k brunch on google code )

Installing nose should not cause problems - it is installed via easy_install, is available in most Linux repositories, or can be simply installed from source. For Python 3, you need to make a clone of the py3k branch and install it from source.

Initial sample code

The factorial calculation will be tested:

#!/usr/bin/env python
import operator

def factorial (n):
if n < 0 :
raise ValueError ( "Factorial can't be calculated for negative numbers." )
if type (n) is float or type (n) is complex :
raise TypeError ( "Factorial doesn't use Gamma function." )
if n == 0 :
return 1
return reduce (operator . mul, range ( 1 , n + 1 ))

if __name__ == '__main__' :
n = input ( 'Enter the positive number: ' )
print '{0}! = {1}' . format(n, factorial( int (n)))


The code works only on Python 2.6 and is not compatible with Python 3. The code is saved in the main.py file.

Unit tests

')
Let's start with simple tests:

import unittest
from main import factorial

class TestFactorial (unittest . TestCase):

def test_calculation ( self ):
self . assertEqual( 720 , factorial( 6 ))

def test_negative ( self ):
self . assertRaises( ValueError , factorial, -1 )

def test_float ( self ):
self . assertRaises( TypeError , factorial, 1.25 )

def test_zero ( self ):
self . assertEqual( 1 , factorial( 0 ))


These tests only test the functionality. Code Coverage - 83%:
$ nosetests --with-coverage --cover-erase
....
Name Stmts Exec Cover Missing
-------------------------------------
main 12 10 83% 16-17
----------------------------------------------------------------------
Ran 4 tests in 0.021s

OK

Add another class for one hundred percent coverage:

class TestMain (unittest . TestCase):

class FakeStream :

def __init__ ( self ):
self . msgs = []

def write ( self , msg):
self . msgs . append(msg)

def readline ( self ):
return '5'

def test_use_case ( self ):
fake_stream = self . FakeStream()
try :
sys . stdin = sys . stdout = fake_stream
execfile ( 'main.py' , { '__name__' : '__main__' })
self . assertEqual( '5! = 120' , fake_stream . msgs[ 1 ])
finally :
sys . stdin = sys . __stdin__
sys . stdout = sys . __stdout__


Now the code is completely covered with tests:
$ nosetests --with-coverage --cover-erase
.....
Name Stmts Exec Cover Missing
-------------------------------------
main 12 12 100%
----------------------------------------------------------------------
Ran 5 tests in 0.032s

OK


findings

Now, based on the real code, we can draw some conclusions:

• First and foremost , full code coverage does not provide a complete check of the program’s functionality and does not guarantee its operation. In this example, there were no tests to test the complex type of argument, although full coverage was provided.
• You can completely cover the code, at least in Python. Yes, it is necessary to operate with built-in functions and know how certain mechanisms work, but this is real, and it has become even easier in Python 3.
• Python is a dynamically typed programming language, and unit testing helps to do type checking. With full coverage, the likelihood that typing is correctly observed throughout the program is much higher.
• Full coverage helps with changing the API of the libraries used and changing the programming language itself (see the example for Python 3 below). Because it is guaranteed that every line of code will be called, all code and API inconsistencies will be detected.
• And as a consequence of the previous paragraph, full coverage helps test the code. For example, when working on a production system, before integrating the software, you can first test it. Often, normal debugging is not possible (say, if there are no rights to a remote system, and the administrator is doing everything), and unit tests will help to understand where the problem is.

Adaptation for Python 3

Using the example of adaptation for Python 3, I want to show how full code coverage helps in the work. So, at first we just run the program under Python 3 and a syntax error is issued:
$ python3 main.py
File "main.py", line 17
print '{0}! = {1}'.format(n, factorial(int(n)))
^
SyntaxError: invalid syntax

We fix:

#!/usr/bin/env python
import operator

def factorial (n):
if n < 0 :
raise ValueError ( "Factorial can't be calculated for negative numbers." )
if type (n) is float or type (n) is complex :
raise TypeError ( "Factorial doesn't use Gamma function." )
if n == 0 :
return 1
return reduce (operator . mul, range ( 1 , n + 1 ))

if __name__ == '__main__' :
n = input ( 'Enter the positive number: ' )
print ( '{0}! = {1}' . format(n, factorial( int (n))))


Now the program can be run:
$ python3 main.py
Enter the positive number: 0
0! = 1

Does this mean that the program is working? Not! It works only before the call to reduce, which is what the tests show us:
$ nosetests3
E...E
======================================================================
ERROR: test_calculation (tests.TestFactorial)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/nuald/workspace/factorial/tests.py", line 9, in test_calculation
self.assertEqual(720, factorial(6))
File "/home/nuald/workspace/factorial/main.py", line 12, in factorial
return reduce(operator.mul, range(1, n + 1))
NameError: global name 'reduce' is not defined

======================================================================
ERROR: test_use_case (tests.TestMain)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/nuald/workspace/factorial/tests.py", line 38, in test_use_case
execfile('main.py', {'__name__': '__main__'})
NameError: global name 'execfile' is not defined

----------------------------------------------------------------------
Ran 5 tests in 0.010s

FAILED (errors=2)

In this example, all this could be detected by manual testing. However, on large projects only unit testing will help to detect such errors. And only full code coverage can guarantee that almost all inconsistencies between the code and the API have been eliminated.

Well, actually, the working code is fully compatible between Python 2.6 and Python 3:

#!/usr/bin/env python
import operator
from functools import reduce

def factorial (n):
if n < 0 :
raise ValueError ( "Factorial can't be calculated for negative numbers." )
if type (n) is float or type (n) is complex :
raise TypeError ( "Factorial doesn't use Gamma function." )
if n == 0 :
return 1
return reduce (operator . mul, range ( 1 , n + 1 ))

if __name__ == '__main__' :
n = input ( 'Enter the positive number: ' )
print ( '{0}! = {1}' . format(n, factorial( int (n))))


import sys
import unittest
from main import factorial

class TestFactorial (unittest . TestCase):

def test_calculation ( self ):
self . assertEqual( 720 , factorial( 6 ))

def test_negative ( self ):
self . assertRaises( ValueError , factorial, -1 )

def test_float ( self ):
self . assertRaises( TypeError , factorial, 1.25 )

def test_zero ( self ):
self . assertEqual( 1 , factorial( 0 ))

class TestMain (unittest . TestCase):

class FakeStream :

def __init__ ( self ):
self . msgs = []

def write ( self , msg):
self . msgs . append(msg)

def readline ( self ):
return '5'

def test_use_case ( self ):
fake_stream = self . FakeStream()
try :
sys . stdin = sys . stdout = fake_stream
obj_code = compile ( open ( 'main.py' ) . read(), 'main.py' , 'exec' )
exec (obj_code, { '__name__' : '__main__' })
self . assertEqual( '5! = 120' , fake_stream . msgs[ 1 ])
finally :
sys . stdin = sys . __stdin__
sys . stdout = sys . __stdout__


Tests show the full coverage and performance of the program under different versions of Python:
$ nosetests --with-coverage --cover-erase
.....
Name Stmts Exec Cover Missing
-------------------------------------
main 13 13 100%
----------------------------------------------------------------------
Ran 5 tests in 0.038s

OK
$ nosetests3 --with-coverage --cover-erase
.....
Name Stmts Miss Cover Missing
-------------------------------------
main 13 0 100%
----------------------------------------------------------------------
Ran 5 tests in 0.018s

OK


Conclusion

Full code coverage is not a panacea that can protect against program errors. However, it is a tool that you need to know and use. There are many advantages in full coverage, and in fact there is only one drawback - the time and resources spent on writing tests. But the more you write tests, the easier they will be given to you in the future. In our projects, we have been providing 100% coverage of the code for more than a year now, and although there were many problems at the beginning, now it’s completely completely cover the code, that’s not a problem. all the techniques were worked out and all the necessary packages were written. There is no magic here (although you will have to work with Python magic), and you just need to start.
PS Full coverage has another advantage, which is not entirely unambiguous, but it is undoubtedly important for those who consider themselves professional - it makes you climb inside Python, and understand how it works. This kind of knowledge is useful to everyone, especially library developers.