Many tests do not happen

Some time ago I decided to slowly introduce automated testing and TDD into my practice. I admit honestly, it turned out all this with varying success. But the fact that life has become much more interesting is an indisputable fact. Different adventures began to happen to me. And, as in all adventures, sometimes it became a bit scary. I want to tell about one such case.

In the project in which I participated, it took a lot of work with time intervals from a minute to a year. A programmer is bad (or, on the contrary, too good) who has not written a single library of working with dates in his life. I am no worse and no better than others, so I decided to stretch my brains and create some code.

How it all began

The project was on Delphi. Deciding to stick to the PLO, I wrote the abstract class TTimeIterator , from which the rest of the classes were generated for movement in increments of minute, hour, day, month, year. The idea of ​​these iterators is that by getting the start and end point, they align them with the correct interval boundaries and allow them to move clearly when they hit the right time points. I cite an abstract class declaration, without cuts, as it was implemented in the project.
')

 TTimeIterator=class(TObject) private StartPoint, FinishPoint:TDateTime; CurrentPoint:TDateTime; CurrentPointNumber:integer; function DTRound(p:TDateTime):TDateTime; virtual; abstract; function DTNext(p:TDateTime):TDateTime; virtual; abstract; function DTPrev(p:TDateTime):TDateTime; virtual; abstract; public constructor Create;overload; function Dump:string; function GetTotalPoints:integer; virtual; abstract; function GetCurrentPointNumber:integer; function GetCurrentPoint:TDateTime; procedure MoveNextPoint; function IsCurrentPoint:boolean; procedure SetStartPoint(DateTime:TDateTime; IncludeMode:TTimeIncludeModeType=INCLUDE_MODE); procedure SetFinishPoint(DateTime:TDateTime; IncludeMode:TTimeIncludeModeType=INCLUDE_MODE); end; 

The following discussion focuses on the implementation of the function

  function GetTotalPoints:integer; virtual; abstract; 

This function, after establishing the starting and ending point, allows to get the total number of points through which the iterator will pass. For the classical iterator paradigm, it is not required, but for some operations in the project such a function would be very useful.

As anyone who has worked with dates knows, the intervals minute, hour, day, and even a week do not cause much trouble. Problems begin with months. The task was to implement the GetTotalPoints function on a monthly iterator.

Strange function MonthsBetween

Without hesitation, I found in the standard library a function with a name and arguments that leave no doubt.

 function MonthsBetween(const ANow, AThen: TDateTime): Integer; 

Not very difficult tests passed with a bang. Some kind of vague feeling made me write tests more complicated. One of these tests broke, and out of the blue.

After a small investigation, I saw that in the opinion of the MonthsBetween function between 01.01.2012 0:00 and 01.05.2012 0:00 not four months (January, February, March, April), but only 3. It interested me. As the famous proverb says: if all goes wrong, read the documentation. I opened Help and read with horror:

In months, between two TDateTime values. MonthlyBetween returns of 30.4375 days per month.

Like this! The function works approximately, taking the average length of a month in 30.4375 days.

A few words to justify the company Borland

After the first shock had passed, I made up my mind with brains. Indeed, the implementation of the function is strange. But the logic is there. And in Help on this logic there are direct instructions. The problem is that when setting some time points, it is difficult to determine exactly how many months between them. And the result depends on the look at the problem. What to take for the whole month? How many months did it take between February 28 and March 31? And between February 28 and March 28? The soil is very slippery.

Why do I think that Borland is not quite right

First, even if we cannot correctly determine the value of a function for all points, then for points that fall exactly within the month limits the required result is obvious. And at these points, the function must return a value that does not contradict the everyday logic of life.

The second objection is of a purely practical nature - I could not think of a single case in which the function, in the form in which Borland made it, could be useful. It is not even suitable for statistics, which, as you know, likes to operate with average values.

Why did I get scared

I was not afraid because of Borland, which wrote such a strangely working function. Especially since the correct implementation takes several lines of code.

I was scared not because of my own levity, when I included the function call in the code without really reading the documentation.

I was frightened because in a large number of cases this function returns the correct values, and if it returns an incorrect one, it differs only by one. I roughly estimated the effect of this error on the system as a whole. This would mean that sometimes we would get data in the report for the previous month instead of the current month (and reports for different months are as similar as twins).

I imagined how this error deftly passes through the collection of unit tests. Then overcomes integration testing. Easily overcomes manual testing and acceptance tests. And the system has been functioning for decades in the real world, occasionally giving out false data.

From the point of view of reliability theory, we would get a floating, difficult to diagnose defect of unknown origin. How many such defects are present in systems that ensure our safety and livelihoods? I dont know. I just LUCKED that I discovered this defect.

I was a little comforted by the following thoughts:

First of all: without testing, I definitely would not have discovered this kind of error.
Secondly: I remembered the words of one of our politicians. He was asked: "Is it true that Russia was lucky with oil prices." He replied: "Lucky fools, and we work from morning till evening."

How it all ended

I wrote the correct, in terms of my requirements, function,

 function DeviceTimeExactMonthsBetween(StartDate, EndDate: TDateTime):integer; const BASE_YEAR=1990; var y1,y2,m1,m2,d1,d2 : word; StartMonths, EndMonths:integer; begin DecodeDate(StartDate,y1,m1,d1); DecodeDate(EndDate,y2,m2,d2); StartMonths:=(y1-BASE_YEAR)*12+m1; EndMonths:=(y2-BASE_YEAR)*12+m2; Result:=EndMonths-StartMonths; if d2<d1 then dec(Result); if Result<0 then Result:=0; end; 

and in the Unit tests, along with others, added the following code:

 CheckNotEquals( // Headfire function DeviceTimeExactMonthsBetween( StrToDateTime('01.01.2012'), StrToDateTime('01.05.2012')), // BorlandFunction MonthsBetween( StrToDateTime('01.01.2012'), StrToDateTime('01.05.2012')), //Test name 'Month BorladFailureTest' ); 

Although the Borland company no longer exists, who knows, all of a sudden, the MonthsBetween function will ever work more correctly.

How lucky am I?

Already, when I decided to make this case in the form of an article, I wondered how lucky I really was to discover the error. For a dozen minutes, he sketched out a small visualization. Below is the text (by the way, it shows how iterators of dates are used in practice).

 procedure TMainForm.DrawFaults(BeginDate,EndDate:TDateTime); var IteratorForBegin,IteratorForEnd:TTimeIterator; x,y,diff:Integer; color:TColor; begin IteratorForBegin:=CreateDeviceTimeIterator(MONTH_INTERVAL); IteratorForEnd:=CreateDeviceTimeIterator(MONTH_INTERVAL); IteratorForBegin.SetStartPoint(BeginDate , INCLUDE_MODE); IteratorForBegin.SetFinishPoint(EndDate, INCLUDE_MODE); while (IteratorForBegin.IsCurrentPoint) do begin IteratorForEnd.SetStartPoint(IteratorForBegin.GetCurrentPoint, INCLUDE_MODE); IteratorForEnd.SetFinishPoint(EndDate, INCLUDE_MODE); while (IteratorForEnd.IsCurrentPoint) do begin diff:=DeviceTimeExactMonthsBetween(IteratorForBegin.GetCurrentPoint,IteratorForEnd.GetCurrentPoint) - MonthsBetween(IteratorForBegin.GetCurrentPoint,IteratorForEnd.GetCurrentPoint); color:=clBlack; //  ,    -  -   if diff=0 then color:=clGreen; if diff=1 then color:=clRed; y:=IteratorForBegin.GetCurrentPointNumber; x:=y+IteratorForEnd.GetCurrentPointNumber; Canvas.Brush.Color:=color; Canvas.Ellipse(10+x*12,10+y*12,20+x*12,20+y*12); //   // Canvas.Pixels[10+x,10+y]:=color; IteratorForEnd.MoveNextPoint; end; IteratorForBegin.MoveNextPoint; end; end; 

Here is a picture of the MonthsBetween function within 2012 (when I wrote the project). Greens correspond to cases when MonthsBetween correct result, red when it is one less.


The situation is not very good, especially since I wrote the project in the second half of the year. (When testing, I try to use dates as close as possible to the current one). Most likely that the tests will pass.

Here is a visualization of the function in the twenty-year range (2010-2020)


We see that the number of errors increases and becomes approximately 50 to 50. Thus, when long intervals are included in tests, the probability of detecting an error increases. What actually happened. When I was not lazy and took a bit more interval, the error was revealed.

Conclusion

Yes of course. By reason, I understand that for any project there is an optimal number of tests and the amount of test coverage. It depends on the purpose of the program, the technology used, the industry and the reliability requirements.
But when I look at the launch vehicle at the start, ready to go into space. Like a submarine carrying a nuclear weapon, it hath hatched the hatches and is preparing to dive, going on a semi-annual hike. When I look at how a new shopping center is being built on a nearby street, the strength calculations of which were carried out on modern clusters. When I see it, I wish with all my heart that there should be as many good and different tests as possible.