Japanese poetry in the service of learning English: an application for memorizing the pronunciation of words

One of the most unusual projects of our autumn hackathon was Skyeng Haiku: a meditative multimedia application about word gathering. Today, its authors will tell you what it is, why it is needed, why it is useful, how they did it, and they will give a link to the working version.

We continue to publish stories about projects brought to the MVP state during our two-day marathon .

Idea

It was based on the idea of ​​helping students memorize the correct pronunciation of words. Among the characteristic errors of pronunciation are pronouncing letters that are actually either soundless or form diphthongs. I wanted to come up with something that would get rid of this bad habit. So the haiku theme appeared: this is a short poem of three lines, in which rhyme is not needed, but the rule is observed: in the first line there are 5 syllables, in the second 7, and in the third again 5. For example:
')
Patient snail
Slides across the morning dew
In the zen garden


Another source of inspiration is Magnetic Poetry : sets of fridge magnets from which poems can be collected; fun, invented by an American songwriter during a creative crisis. Additional bun - at Magnetic Poetry you can download ready-made sets of words (correctly selected by the number of syllables, the ratio of parts of speech, etc.); just what was needed for the MVP version.



By combining haiku and “magnetic poetry,” we get a product prototype, where the user has to drag the words from the set in accordance with rule 5-7-5 of syllables. Meaning and grammar are not important, it is important to understand the number of syllables in each word, Japanese entourage, background music and voice of the resulting poem. The more insane its content, the better - mnemonic links will work, the word will be remembered.

Engine

A simple application for a couple of dozen lines of code written on the vue.js framework. This is one of the increasingly popular frameworks for writing rich applications, it is often compared to Angular, React, and the like. The author of the engine in real working life does not have the opportunity to write anything with his hands, so the hackathon helped us recall the skills and in passing threw the idea for a hobby - now the engine periodically dopils a little on weekends.

The application consists of several components:

the word is the minimum component with which the application works, knows only about itself and the number of syllables in itself

line - can count the number of syllables in the words included in it and signal when the necessary meaning of syllables has been reached (in our case there are three such lines, but there can be any number of them in general, this is set through the application settings)

music player

SpeakFromMyHeart component stitching words in lines into text and losing synthesized speech

and, of course, the main application that collects it all, tracks the state of the haiku as a whole, controls the loading of words, plays the interface sounds. Further division of logic by components suggests itself, and we will do it in some output.

For dragging words tried several libraries of drag-n-drop, stopped at Dragula . As it turned out, it works well on the desktop, but strangely on mobile devices - the entire screen is dragged along with the word. Everyone who passed by the development team tried to help somehow, for which they were grateful, but it didn’t work out the problem during the hackathon. Later Dragula and vue-dragula were replaced with VDDL for drag-and-drop , because this library is better documented and has no problems with nested components.

Definition of syllables

There are a number of libraries in the world that are designed to help poets looking for rhymes: they give estimates of words, suggest how many words there should be in a rhyming word, etc. It was originally intended to use the Words API - the most famous such library, which can, among other things, determine the number of syllables in a word. However, during its testing, there were some doubts about the correctness of the results issued, and there was no opportunity to resolve these doubts with the help of methodologists on the hackathon.

Therefore, we decided that if we already have a fixed set of words, then for MVP it will be easier to count the syllables manually, and for now, disconnect the Words API and figure it out after the hackathon. Now we have come to the conclusion that the glitch visited us rather, and not this library — we will use it when working with arbitrary sets of words.

Voice acting

The voiceovers of the resulting lines were hilarious. The atmosphere of the application is important for us - we needed a sad pensive voice ... well, haiku. First we tried to use the built-in voice acting with the Web Speech API . It turned out that the set of votes depends on the platform and the browser: for example, on Mac it was a pleasant feminine timbre, and on Ubuntu it was pretty disgusting. Having played around with the sets, we eventually came to Amazon Polly, where there is a voice that is somewhat reminiscent of the Twin Peaks atmosphere. It is a pleasure to work with Polly - a generated string is sent to her, she sends back a stream of bytes, which should be sent to the browser with the appropriate headers to play. Polly supports SSML markup, allowing you to select voices, mark up intonation, pauses, speed, etc. For example, like this:

<prosody rate=\"x-slow\">   <break time='1s'/>   <break time='1s'/>   </prosody> 

- we set slow speech with thoughtful pauses between lines.

First, by the way, we used the ChattyKathy library, a small AWS SDK wrapper, which makes it very easy to use it to send a request to Polly and play synthesized speech. But in the end, they began to synthesize speech on the back end in order not to bother with authorization through AmazonCognito.

Polly's problem is that she is paid. For a hackathon, this is a trifle, but we cannot lay it out, so we cannot hear how our haiku sounded on the hackathon, in the current version it is impossible - only in the video:



Well, we are still back to the Web Speech API (yes, this is a link to the current version). Unlike Polly, it does not yet support SSML, experimental (not supported by all browsers), but free.

By the way, they found a funny bug in Chrome: according to the Web Speech API specification, if there is an unsupported tag in the text, it should be ignored; "Chrome" also reads all the markup. We did not immediately understand what was happening when for the first time we heard from the speakers 'less then prosody rate equals slash ...'

Development opportunities

• Connect custom word sets from Words. Combining words for study with official and auxiliary words, you can expand the dictionary and make new phrases. Here lies the difficulty: firstly, in the set there should be enough words of different lengths to compile a haiku, and secondly, the user set should be diluted with words of general vocabulary and follow the diversity of parts of speech. And this is a topic for a separate rather complex product.
• Integration into Vimbox. We have a section with games, while in its infancy, and it needs to be developed. It is possible to give access to it not only to students, but also to those who are only thinking about studying at Skyeng - as a lure.
• Standalone application for mobile platforms. We connect new media resources: music and themes. First of all - the night mode with a dark background and low contrast to stick in the bed before going to bed.
• Adding gamification - time counter, playing together, etc.
• The ability to share the results in social networks.

Just in case, once again link to the current version of Skyeng Haiku .

Well, as always, we remind you that we are actively looking for talents ! Come to work with us, we have fun!