iTrace. How are the letters written?

What is this application?

iTrace is a mobile application to teach children how to write letters. Electronic writing on the iPad. Now it is used in several countries of the world (mainly in the USA) for teaching children to write. Misha Bogorad invented and organized all the work on the project, and I had the opportunity to participate in the project as the developer of all kinds of viscera, mainly drawing letters and analyzing the quality of their drawing.

Idea complexity

The idea of ​​iTrace is no different from ordinary writing. We take a letter, we ask the child to draw it, prompting if it is difficult for him. First, the letter is large and you can make a big mistake, then it decreases, and the tolerance is also less and less. In the end, the child, through habit, remembers how to spell a letter.


In the article I will tell about the difficulties that had to be faced and how they managed to be solved. If the topic is interesting, ask in the comments, I can tell you more about the technical part.

In addition to organizational difficulties (draw a few thousand pictures for prizes and animations, find training specialists who helped work out the methodology, understand where to find music and voice acting for the application, and so on), we also encountered technical difficulties. The three main ones are optimization for work on old devices, drawing letters of different thickness and quality control of entering letters by a child.
')

Optimization

The optimization task came from the audience of the application. Children often use old ipads, and in particular, the first. This is not the fastest device, with not the largest amount of memory. I had to figure out how to work with the animation, what size it should be in order to work on the old hardware, how many frames per second can be made, and so on.


Also had to optimize work with resources in the application. It would seem that just put them in a bundle and everything that is optimized there, but, alas. First, the resources themselves are large. It took each picture to run through the optimizer, analyze which format is minimal, choose the best one. This is partially done by the project's build system, but it’s better to control yourself. Secondly, the installation of the test build (and, as a result, the installation of the application itself on the iPad) took an enormous amount of time. Both of these processes are associated with copying and unpacking files, and when the number of files is several thousand, the process can take considerable time. I knowingly at first remembered the first Aypad. Installing a test version on this device took about 10 minutes.

At this moment I remembered the pack files. This is a technique that has long been used in games where there are many, many textures. All files are packed into one file, the work with which can be organized very effectively (for example,). It would be possible to come up with its own format, but I successfully thought about trying zip without compression and tests, the speed was almost the same as accessing files directly.

The only problem that hindered a bit was that the zip files lack random access to the files, I had to build my own table of correspondence with the names of the file locations, which in the simplest case looks like this:

{"Levels":{"":[3149,48427877],"iphone_cursive_word_levels.csv":[3153,48428251],"iphone_cursive_levels.csv":[3152,48428149],"iphone_regular_levels.csv":[3154,48428358],"regular_word_levels.csv":[3157,48428662],"save_before_rollback.zip":[3158,48428762] ... 

After that, a little memory optimization (for a large number of files, it occupies a significant place), for download speed (it is cached when the application starts), but after that the application was installed in 20-40 seconds.

The DPLPacker code can be found here: https://github.com/bealex/DPLPacker And the simplest work with a pack looks like this:

Create a file

 _zipFile = [[DPLZipFile alloc] initWithZipFile:_zipFilePath]; 

Check that the file is in the archive, we get it (by path and name):

 NSData *data = nil; if ([_zipFile fileExistsForPath:filePath]) { data = [_zipFile dataForPath:filePath]; } return data; 

You need to not forget to carefully understand the @ 2x-pictures, if we pack them. Unlike regular files, the system will not download the necessary version for us:

 - (UIImage *)imageAtPath:(NSString *)filePath { CGFloat scale = 1; if (DPL_isRetina()) { if (![filePath contains:@"@2x"]) { NSString *filePath2x = [filePath stringByReplacingOccurrencesOfString:@".png" withString:@"@2x.png"]; if ([self fileExistsAtPath:filePath2x]) { filePath = filePath2x; if (DPL_isIPad()) { scale = 2; } else { scale = 1; } } } else { scale = 2; } } else { if (![filePath contains:@"@2x"] && ![self fileExistsAtPath:filePath]) { NSString *filePath2x = [filePath stringByReplacingOccurrencesOfString:@".png" withString:@"@2x.png"]; if ([self fileExistsAtPath:filePath2x]) { filePath = filePath2x; scale = 2; } } else { scale = 1; } } NSData *data = [self dataWithContentsOfFile:filePath]; if (fabs(scale - 1) > 0.01) { if (DPL_OSVersionMajor() >= 6) { return [UIImage imageWithData:data scale:scale]; } else { return [UIImage imageWithCGImage:[UIImage imageWithData:data].CGImage scale:scale orientation:UIImageOrientationUp]; } } return [UIImage imageWithData:data]; } 

How curve is this curve?

In all other applications that teach children to write letters, learning itself does not occur. The child is simply shown how the letter should be written, and then he can draw it as he wants. There is no feedback, no possibility to verify the correctness, correct if necessary. Why is it important? Because there is a school that teaches in a certain way (in the USA there are three basic spellings of letters, plus italics, options for the right / left hand, and minor differences between different schools).


If a child learns how to write "as it came to him" in the application, he will have to relearn. It will be difficult, painful and unpleasant.

Therefore, the main feature that I wanted to implement was to be a spell check. If the child led the line in the wrong place, you need to immediately tell him that it is not there. If you started not from the beginning - tell where it is, the beginning. This problem also had to be solved.


The complexity was composed of two parts. The first is how to make the processing fast. It should not significantly delay the drawing of the letter even on the old, first, Aipad. The second is how to determine exactly what the child did “not that.” Here are examples of errors that iTrace catches:

 IWTaskErrorCodeErrorTooBig = 1, //     IWTaskErrorCodeLineExitedCorridor = 2, //    IWTaskErrorCodeCornerDrawingDistanceWrong = 3, //    ""  IWTaskErrorCodeStraightLineDrawingDistanceWrong = 4, //     IWTaskErrorCodeCornerDoubleEnter = 5, //      IWTaskErrorCodeLineNotCovered = 6, //        IWTaskErrorCodeWrongStart = 7, //     (  -  —  ,      ,      ) IWTaskErrorCodeTooOverextended = 8 //    . , ,  ,   ,     .    —      () ,       ,              —      . 

The first part was decided by relatively ordinary methods. At first I implemented all the algorithms using high-level structures (Objective-C classes, collections from there), but when I saw in the profiler that I spent too much time working with them (even on the boxing / anboxing of numbers from NSNumber), I switched to regular C -structure After that, he entered several caches in order to recalculate only the end of the drawn line, and not all of it. This allowed to remove the brakes when drawing long lines, and to achieve the desired performance.

The main task of determining “what the child did wrong” was to determine what “wrong is”. What are the mistakes? We have identified a few:

• the beginning of drawing is not from the correct point,
• line completion,
• moved too far sideways from the perfect line,
• went "the wrong way." This error is different from the previous one, since we can go back along the perfect same line,
• cut off the corner

Then they tried to figure out how to formalize all these errors. It is clear that you need to compare the perfect line and drawn. It is clear that you need to break them into small segments, compare them and then accumulate the accumulated error. The difficulty was in the corners. You can’t throw them out completely, the corners are an important part of the letter. But the error in them accumulates very simply and very quickly.

After two months of trial and error, something like this came out:

• we divide the curve into “linear segments”,
• between segments we have areas that we call “corners”. The angle is just a small segment where the direction of the line changes dramatically. This can be either a real angle, or some loop, or the beginning / end of a line.
• on linear segments, we consider as previously thought. We look at the similarity of the directions of the segments and the distance between the ideal / drawn curve. We accumulate an error.
• in the corners we look at the difference between the length of the ideal and the drawn curve. And that's all. Surprisingly, it turned out that if you correctly select the allowable difference, then this simple rule checks the drawing of corners well.
  

Angles (large circles) are visible on the image. They are searched for automatically, according to the curve specified in the SVG format.

Russian language

It was logical to assume that the application will need to localize in different languages. But what languages ​​it will be and when we will do it exactly after the start was not clear. And the development was carried out without looking at any other language.

When the desire and opportunity to support the Russian language appeared, it turned out that it was not quite easy.

First, you need to translate the interface. Moreover, if the interface should be tied to the language of the system, the learning language may be different. You need to be able to switch them. And you need to make so long Russian got everywhere in the interface.

Secondly, the beautiful font we used did not contain Russian letters. I had to order a revision of the font.

Third, it took more pictures. In Russian, there are more letters, more exercises. In addition to the pictures, a new voice was also needed.

Fourth, it was necessary to refine the algorithm for working with letters. Diacritics (“y”, “yo”), small strokes (“y” or “yi”) - all this complicated the algorithm for controlling the quality of drawing.

By the way, packing resources into a file occurred. Having created several such packages and switching these files, it turned out to be very convenient to switch between languages.

Little things

Of course, there were a lot of little things in development. For example, iTrace can print “real” paper recipes. To make it more interesting for children, a labyrinth is drawn from below each recipe. It is generated every time anew, it was interesting to choose the parameters so that the children were both interesting and not very simple / difficult.


There were problems with aprouvom. For example, when we tried for the first time to turn on the Touch ID for a parente gate (a special task that parents, but not children, solve to enter the application settings), we were refused, saying “no”. I had to talk with representatives of Apple, to come up with a more accurate algorithm for working with Touch ID, after which they took the feature.

We also did not immediately figure out how to make a purchase inside the free version. At first, they wanted to make a purchase of each small feature, but after discussion they decided to make only one purchase “for everything”.

It was interesting to draw the letters themselves. They are drawn in a lot of places, and in dotted lines, and in different thicknesses on the drawing screen, and in history, where you can see the child’s attempts to draw letters with errors ... everywhere their requirements, their difficulties.

The result is a great app. Take a look. There is a free version, with the purchase inside, and paid. More than 350 thousand people have already looked, and many like it. :-)