Reversible Cyrillic transliteration

Perhaps someone else remembers writing SMS, and sometimes letters, in “transliteration.” But why transliteration today, when everywhere is already unicode? Unfortunately, inherited applications go out of service much slower than we would like. For example, today scanners are used that do not allow the Cyrillic alphabet in the names of patients. Given that the information system used by the same department perfectly understands the Cyrillic alphabet. And the tomograph operator needs not only to call the patient for research, but also to correctly write down his last name in some documents. Similar situations may occur in different places.



That is, the problem arises to somehow transfer the text data to the legacy system in order to:

• the person - the operator of the inherited system was able to read the received text "by sound"
• if necessary, it would be possible to unambiguously restore the original Cyrillic text

To avoid being bored, let's add more detailed requirements related to compatibility and simplicity for a person:

1. use only letters in the narrow sense, without punctuation and diacritical elements (this will also save the register)
2. convert each original letter independently of the rest (without complications like “at the beginning / end of a word”, etc.)
3. replacements as short as possible, ideally single-letter
4. inverse transformation rules are as simple as possible, for example, replacements must meet the Fano condition
5. close to sound substitutions, in the view of the "ordinary person" - in practice, it is a kind of mixture of Latin, English, French, German and, sometimes, Spanish phonetics

Of course, the listed requirements are not quite (except for the first two), but rather heuristics.



You can find many ready-made options for translating the Cyrillic alphabet into Latin. But among them there was nothing that would satisfy all the requirements to an acceptable degree. Either uses accented characters as standards, then throws out letters (usually “b”), then offer irreversible (u -> shch) or phonetically wild (w -> w) replacement options, or have other fatal flaws .

')

So we make our bike. Actually, you need to create a correspondence table, and describe the algorithm for converting there and back.

Table

Let's start with all the obvious one-letter replacements:

BUT

B

AT

R

D

E

Yo

F

H

AND

Th

TO

L

M

H

ABOUT

P

R

WITH

T

Have

F

X

C

H

Sh

U

B

S

B

Uh

YU

I

A

B

V

G

D

E

Z

I

K

L

M

N

O

P

R

S

T

U

F

Remembering the requirement of possible short replacements, and since for “C” we use “S”, with a clear conscience we use the symbol “C” for “C”.



For the remaining letters, the tradition (and simply the lack of Latin characters) tells us to use two-letter combinations. For simplicity, the direct and, in particular, the inverse transformation, it would be good if the combinations form characters that are not used outside the combinations. By Fano's condition, such a special character should be at the beginning of the combination, but the tradition is too strong and we will write the letter “H” at the end of the combination. But, if the “H” symbol is not used separately and the conversion algorithm is allowed to “return” (in fact, remember) to the previous input symbol, for the postfix combinations, the “inverted” analogue of Fano condition can be considered fulfilled. That is, the algorithm will be able to uniquely determine them.



The special symbol for consonants is the same everywhere - “H”. And for vowels, there are two options: "Y" and "J". Although “Y” is more familiar, it is also often used separately, for “Y” or for “Y”. And "J" is rather perceived as a purely auxiliary character.



Resolved, use for vowels "J". And by the way, the freed "Y" is used for "Y".



Since “J” is now a special character, it cannot be used for “”, and only “ZH” remains. Similarly, for “X” you cannot use “H”, and only “KH” remains.



Now we can write down common and selected combinations and single characters:

BUT

B

AT

R

D

E

Yo

F

H

AND

Th

TO

L

M

H

ABOUT

P

R

WITH

T

Have

F

X

C

H

Sh

U

B

S

B

Uh

YU

I

A

B

V

G

D

E

Zh

Z

I

Y

K

L

M

N

O

P

R

S

T

U

F

KH

C

CH

SH

EH

JU

Ja

The common and good (in the sense of our requirements) replacements ended here, and we are embarking on the unsteady ground of “gag”, analogies and compromises.



Let's start with "Y". "Y" is already busy (remember about reversibility), and phonetically it is a bad substitute. Let's look at the solution for "E" (taken, by the way, from ISO / R 9, 1968). By analogy, "Y" should be replaced by "IH". It is strange that this option has not met anywhere.



With “Yo” the situation is also strange. There is a clear, but not suitable for us option "E". And there is a phonetic option "JO". But in the Russian alphabet "E" is not accidentally made on the basis of "E", and not "O". "E" often alternates with "E", for example, "Kl yo n - kle new", and never alternates with "O". This results in another heuristic - the “alphabetic” (non-phonetic and non-graphic) proximity of letters. As a result, for "E" we construct a replacement for "JE".

Let's pause:

BUT

B

AT

R

D

E

Yo

F

H

AND

Th

TO

L

M

H

ABOUT

P

R

WITH

T

Have

F

X

C

H

Sh

U

B

S

B

Uh

YU

I

A

B

V

G

D

E

Je

Zh

Z

I

Y

K

L

M

N

O

P

R

S

T

U

F

KH

C

CH

SH

Ih

EH

JU

Ja

How nice it would be to stop at this and say that the problem was solved in the first approximation. But there are still three letters, without which it can not do. For the remaining letters there are no adequate options. Signs are usually replaced with apostrophes, and literal substitutions are either just arbitrary or “witty,” like “b” -> “q”. For "U", a replacement without accents is usually 3 - 4 characters long, and there will still be problems with it.



After long searches and sufferings, I had to stop at this reasoning: for letters that do not correspond to sounds, you cannot use letters for which sounds exist. And we are left with only “special” symbols for the formation of combinations. But according to Fano's condition they cannot be used separately, the combinations will become ambiguous.



Exit - use a combination of special characters with each other. This, apparently, will complicate the conversion algorithm a little more, but it seems possible to preserve uniqueness.



For a hard sign (it’s only a separator), the replacement of “HH” seems intuitively appropriate (it cannot be read like a pause, a separation).



And for a soft sign, the chain of associations (“J” -> iroded vowels -> softening of the previous consonant) + (“H” -> division) leads to the replacement of “JH”.



You can not call a beautiful decision, but among rotten apples, the choice is not great.



Unfortunately, such a choice makes it impossible to use the replacement “Sh” -> “SHH”. The sequence "SHH" will mean "", and this combination is found in Russian (for example, "congress"). Here again there are no nice decisions, and it is necessary to look for at least somehow motivated. The sound “” is close to the softened “”, and by analogy with the soft sign, this can be represented by the prefix “J”. I understand that now I refer to myself that the code is still 3 and not standard. But, as they say, "other writers we have for you are not."



As a result:

BUT

B

AT

R

D

E

Yo

F

H

AND

Th

TO

L

M

H

ABOUT

P

R

WITH

T

Have

F

X

C

H

Sh

U

B

S

B

Uh

YU

I

A

B

V

G

D

E

Je

Zh

Z

I

Y

K

L

M

N

O

P

R

S

T

U

F

KH

C

CH

SH

Jsh

HH

Ih

Jh

EH

JU

Ja

Algorithm

Converting from Cyrillic to Latin is trivial. We do not pay attention to the register for brevity.



For example, the result of a couple of well-known pangrams:

Shirokaja ehlektrifikacija juzhnihkh guberniy dast mojshnihy tolchok podhhjemu seljhskogo khozjaystva.

Shheshjh zhe ejshje ehtikh mjagkikh francuzskikh bulok da vihpey chaju.

It does not look very good, but the main purpose of this variant of transliteration is still the full name:

Aleksandr Ivanovich Lebedjh

Georgiy Konstantinovich ZHukov

Inverse transformation is much more interesting. Especially considering that it would be good to explain it to a person (not from IT) for execution “in mind”.

Apparently, it is necessary to start with special cases.

• As we read from left to right, the first thing we pay attention to the symbol "J". Behind him must go one of the five characters: "E", "H", "U", "A" or "S" (for "S" should in this case be necessarily more "H"), and it turns out what's in the table for two or three letter combinations.
• If “J” is not present, we look, whether the letter “H” follows the symbol. Here is the most difficult moment for attention: in this case there should not be an option when the third character again goes "H" (this is the code "HH"). That is, you need to see and analyze three characters in a row. This is where the violation of the Fano condition auknulo (well, that once).
• If neither “J” nor a single “H” in the vicinity of the symbol was found, safely replace it in the table as a separate letter.

After a short workout, as practice shows, people are able to perform the inverse transformation manually. But to force them to do it without special need, of course, is not necessary. You can also automate (again, for simplicity, only for the string and in uppercase):

Total

It would seem that a simple and long-solved problem, and what a space for creativity and discussion.



Seriously, it turned out a working algorithm for reversible transliteration of all the letters of the Russian Cyrillic alphabet into Latin letters. In this case, the result, with a discount on the rigidity of the requirements, is acceptable to read. Can be used to integrate with legacy systems and libraries, to generate identifiers.



I hope someone will find the solution useful, and the path to it will be entertaining.

Addition

Based on the discussion in the comments. Need to be shorter and more formal.

There are no cancellable requirements:

1. As a result of transliteration, only letters of the main Latin should be obtained .
   (there are 26 of them)

   abcdefghijklmnopqrstuvwxyz
2. Transliteration must be completely reversible

There are standards . None meet the requirements.

It is necessary to choose one standard and modify it minimally, only to meet the requirements.

Modifications are uniquely determined by the chain. If you do not like the result, write, please, from what point of the chain do you disagree.



I. Choose a starting standard.

The most phonetically correct, of course, is BGN. But BGN is fundamentally (even deliberately) not reversible.

Closest to the base Latin and to reversibility, in my opinion, “GOST 16876-71 / table 2”, and we choose it.

BUT

B

AT

R

D

E

Yo

F

H

AND

Th

TO

L

M

H

ABOUT

P

R

WITH

T

Have

F

X

C

H

Sh

U

B

S

B

Uh

YU

I

A

B

V

G

D

E

JO

Zh

Z

I

Jj

K

L

M

N

O

P

R

S

T

U

F

KH

C

CH

SH

SHH

″

Y

′

EH

JU

Ja

The scheme is not perfect in terms of our requirements. Have to change.

Ii. The first striking "JJ". Why this is bad, you can see in the "theoretical" spoiler. We follow the rule “look at BGN in any incomprehensible situation”. That is, "Y" -> "Y".

Iii. Now left without the code "Y". BGN does not help. There is a phonetic analogy in the “I-S” and “E-E” pairs. To obtain the code of the letter "E", the standard adds a postfix to the code "E". We also do: “Y” -> “IH”.

Iv. Non-letter replacements for “b” and “b” remain. In order not to destroy phonetics, we can use only prefixes and postfixes.

V. Note that no one has forbidden (in the "theoretical" spoiler) to use postfix as a base character in combination with the prefix (at least), and vice versa. That is, we have the codes "JH", "JHH" and "JJH".

Vi. It remains to distribute this wealth. The most frequent “b” is the shortest code: “b” -> “jh”.

VII. “Kommersant” has no sound, “H” is easier not to voice when reading. So, choose from the remaining code, where more “H”: “” -> “JHH”.



Happened:

BUT

B

AT

R

D

E

Yo

F

H

AND

Th

TO

L

M

H

ABOUT

P

R

WITH

T

Have

F

X

C

H

Sh

U

B

S

B

Uh

YU

I

A

B

V

G

D

E

JO

Zh

Z

I

Y

K

L

M

N

O

P

R

S

T

U

F

KH

C

CH

SH

SHH

Jhh

Ih

Jh

EH

JU

Ja

The “SHH” code is the only one using postfix length 2. But the total code length is only 3, the letter is rare. Reversibility is not violated (and even “easy reversibility”).

There is nothing more in the standard to touch.





The code is given only for experiments and a descriptive description of the inverse transform algorithm.







Thanks to constructive criticism in the comments, the decision has changed. My understanding of the problem has become deeper. I thought about the "adult" implementation.

Thanks to all! Habr makes things better.