Unit testing is not enough. Need static typing!

When I was working on my master’s thesis, I promised myself that I would publish it on the Internet under a free license. I received a degree, but, unfortunately, I came across one of the unwritten rules of universities - when you spend a lot of time on a question of interest, it bothers you.
Finally, a year later, I still publish it.
For those who are too lazy to get acquainted with my complete work (in any case, 60 pages of text is not so much for research work, but it is still not enough), I offer a brief version of the article. Please note that the short version does not take into account some important information, so I ask you to write reviews only about the full version.

In my study, I drew attention to the arguments of supporters of dynamic typing, who believe that static typing is not needed to detect errors.

The essence of their reasoning is as follows:

1. Static typing is not enough to detect bugs, so unit tests are needed;
2. Static typing becomes redundant since you have tests;
3. Due to static typing, some valid programs may issue warnings at the compilation stage.

Despite the fact that I heard these statements many times, I could not find a confirmation for them. So I decided to find out if the unit tests really help eliminate typing errors. I was also interested in another question: how often do developers use constructs of languages ​​with dynamic typing, which cannot be expressed in languages ​​with static.
')

My experiment was to find opensource projects in a dynamically typed language, covered with unit tests, and then manually translate them into a statically typed language. I wanted to find out the number of defects that were detected by type checking, and how many dynamic constructions cannot be directly translated into a static language. It should be emphasized that in this experiment there will be no rewriting of the program code, but its line-by-line translation from one programming language to another. I do not take into account defects that were not detected by type checking, or any other errors that could not occur in the original program.

Before starting the experiment, it was necessary to choose languages ​​with dynamic and static typing for work.

The criteria for a dynamic language were as follows:

1. this language must be dynamically typed;
2. language must support unit testing;
3. a sufficient number of opensource solutions in this language are needed;
4. the language must be known and have a sufficient number of supporters.

Under these criteria, Python is great.

Next was to choose a static language that takes into account several requirements:

1. the language must run on the same platforms as Python;
2. language must be strictly typed;
3. language should be respected by supporters of static typing.

Haskell met all of these requirements.

Next, I selected four random projects that had to be translated from Python to Haskell: The Python NMEA Toolkit , MIDIUtil , GrapeFruit, and PyFontInfo .

The Python NMEA Toolkit

The translation of this project led to nine errors. Three of them arose due to the wrong type of input parameters, and the remaining six due to incorrect use of the API. Only one type error was covered by the unit test. With the help of static typing, one error was corrected, which was manifested during the execution of the program. It would be possible to get rid of two modular tests, since they served only for type checking. Not a single dynamic construction was found that could not be translated into static.

MIDIUtil

The translation of MIDIUtil led to the identification of two errors. One of them could be traced using the test. Another would eliminate static typing. The MIDIUtil utility uses the struct.pack and struct.unpack methods, which cannot be translated to Haskell directly, because they use a format string containing the types of arguments and return value.
However, in all cases of use, formatting strings were hard-coded, so Haskell code could use hard-coded functions instead of them without losing expressiveness. If such strings were stored in a configuration file in MIDIUtil, then to translate it into a language with static typing would have to work seriously on the solution architecture.

GrapeFruit

The broadcast from GrapeFruit did not reveal any errors. And the only error during execution could be to eliminate static typing. It should be noted that static typing would help get rid of one test. There was not a single dynamic construction that cannot be translated into a static one.

PyFontInfo

The translation from PyFontInfo led to the identification of 6 type errors. Two execution errors could be fixed with the use of static typing. From one test could get rid of. PyFontInfo also uses struct.pack and struct.unpack, so it cannot be translated into Haskell, but this restriction can be circumvented.

results

Experience has shown that all projects could easily be translated into a language with static typing, and the changes would be minimal. It turned out that unit testing cannot be called an adequate replacement for static typing. During the experiment, 17 typing errors were discovered that were not detected by tests; but many errors were still covered with tests.

Conclusion

The results of the experiment show that unit testing cannot be considered a substitute for static typing and does not take into account all the problems. A significant problem is the fact that building a competent test for types is time consuming, and in languages ​​with static typing this is done automatically. The use of static type checking for programs written in dynamic languages ​​would allow detection of defects that are not detected by tests. And at the same time do not require major processing architecture.
Experience has shown a new look at the problem of getting rid of typing errors with the help of unit tests. I hope that this will interest someone, and they will conduct an experiment on other projects.
You can read the full article here .