Do not forget about language and cultural features

Sooner or later, all are faced with problems associated with linguistic and cultural diversity in writing programs. I was very surprised to learn that some of my friends who write in C ++ solve these problems with their bikes. For those who still do not know what std :: locale is, I would like to briefly show how to work with it using an example and what happens if I forget about it ...

std :: locale (localization) is an object that allows to take into account the cultural and linguistic characteristics of users. In essence, this is a container of special classes — facets , to which the program refers if it needs to perform actions that depend on natural language. The program entrusts such actions to localization facets. Any custom facets can be added to the localization. But the most interesting are the standard ones, since they are implemented in any localization and can be permanently or temporarily replaced:

• collate (string comparison)
• numeric (input / output numbers)
• monetary (money input / output)
• time (input / output time)
• ctype (character classification)
• messages (selection of messages)

In reality, we constantly use facets, even without knowing it. The standard template library uses lacalization for I / O. boost :: regex for case conversion, etc. Localization is set by the platform. Users of * nix systems are familiar with such strings as "ru_RU: UTF-8", "en_US.UTF-8" - these are the names of localizations in the platform. The program uses custom localization. If localization is not specified by the user, “classic” is used.

Example of using localization and cell overrides

Consider an example in which we will try the technique of replacing the standard localization facet. Usually, streaming I / O is considered, but I would like to focus on what can happen if you write code that is dependent on localization, without knowing what it is. Let's try using locales with the common library boost :: xpressive (you can use boost :: regex, but for those who first hear about xpressive, it will be useful to read about it):

2. #include <boost/xpressive/xpressive.hpp> #include <iostream> using namespace std ; using namespace boost :: xpressive ; int main ( int argc, char * argv [ ] ) { sregex xpr = sregex :: compile ( "" , regex_constants :: icase ) ; smatch match ; string str ( " !" ) ; if ( regex_search ( str, match, xpr ) ) cout << "icase ok" << endl ; else cout << "icase fail" << endl ; return 0 ; }
3. #include <boost/xpressive/xpressive.hpp> #include <iostream> using namespace std ; using namespace boost :: xpressive ; int main ( int argc, char * argv [ ] ) { sregex xpr = sregex :: compile ( "" , regex_constants :: icase ) ; smatch match ; string str ( " !" ) ; if ( regex_search ( str, match, xpr ) ) cout << "icase ok" << endl ; else cout << "icase fail" << endl ; return 0 ; }
5. #include <boost/xpressive/xpressive.hpp> #include <iostream> using namespace std ; using namespace boost :: xpressive ; int main ( int argc, char * argv [ ] ) { sregex xpr = sregex :: compile ( "" , regex_constants :: icase ) ; smatch match ; string str ( " !" ) ; if ( regex_search ( str, match, xpr ) ) cout << "icase ok" << endl ; else cout << "icase fail" << endl ; return 0 ; }
6. #include <boost/xpressive/xpressive.hpp> #include <iostream> using namespace std ; using namespace boost :: xpressive ; int main ( int argc, char * argv [ ] ) { sregex xpr = sregex :: compile ( "" , regex_constants :: icase ) ; smatch match ; string str ( " !" ) ; if ( regex_search ( str, match, xpr ) ) cout << "icase ok" << endl ; else cout << "icase fail" << endl ; return 0 ; }
8. #include <boost/xpressive/xpressive.hpp> #include <iostream> using namespace std ; using namespace boost :: xpressive ; int main ( int argc, char * argv [ ] ) { sregex xpr = sregex :: compile ( "" , regex_constants :: icase ) ; smatch match ; string str ( " !" ) ; if ( regex_search ( str, match, xpr ) ) cout << "icase ok" << endl ; else cout << "icase fail" << endl ; return 0 ; }
9. #include <boost/xpressive/xpressive.hpp> #include <iostream> using namespace std ; using namespace boost :: xpressive ; int main ( int argc, char * argv [ ] ) { sregex xpr = sregex :: compile ( "" , regex_constants :: icase ) ; smatch match ; string str ( " !" ) ; if ( regex_search ( str, match, xpr ) ) cout << "icase ok" << endl ; else cout << "icase fail" << endl ; return 0 ; }
10. #include <boost/xpressive/xpressive.hpp> #include <iostream> using namespace std ; using namespace boost :: xpressive ; int main ( int argc, char * argv [ ] ) { sregex xpr = sregex :: compile ( "" , regex_constants :: icase ) ; smatch match ; string str ( " !" ) ; if ( regex_search ( str, match, xpr ) ) cout << "icase ok" << endl ; else cout << "icase fail" << endl ; return 0 ; }
11. #include <boost/xpressive/xpressive.hpp> #include <iostream> using namespace std ; using namespace boost :: xpressive ; int main ( int argc, char * argv [ ] ) { sregex xpr = sregex :: compile ( "" , regex_constants :: icase ) ; smatch match ; string str ( " !" ) ; if ( regex_search ( str, match, xpr ) ) cout << "icase ok" << endl ; else cout << "icase fail" << endl ; return 0 ; }
12. #include <boost/xpressive/xpressive.hpp> #include <iostream> using namespace std ; using namespace boost :: xpressive ; int main ( int argc, char * argv [ ] ) { sregex xpr = sregex :: compile ( "" , regex_constants :: icase ) ; smatch match ; string str ( " !" ) ; if ( regex_search ( str, match, xpr ) ) cout << "icase ok" << endl ; else cout << "icase fail" << endl ; return 0 ; }
13. #include <boost/xpressive/xpressive.hpp> #include <iostream> using namespace std ; using namespace boost :: xpressive ; int main ( int argc, char * argv [ ] ) { sregex xpr = sregex :: compile ( "" , regex_constants :: icase ) ; smatch match ; string str ( " !" ) ; if ( regex_search ( str, match, xpr ) ) cout << "icase ok" << endl ; else cout << "icase fail" << endl ; return 0 ; }
14. #include <boost/xpressive/xpressive.hpp> #include <iostream> using namespace std ; using namespace boost :: xpressive ; int main ( int argc, char * argv [ ] ) { sregex xpr = sregex :: compile ( "" , regex_constants :: icase ) ; smatch match ; string str ( " !" ) ; if ( regex_search ( str, match, xpr ) ) cout << "icase ok" << endl ; else cout << "icase fail" << endl ; return 0 ; }
15. #include <boost/xpressive/xpressive.hpp> #include <iostream> using namespace std ; using namespace boost :: xpressive ; int main ( int argc, char * argv [ ] ) { sregex xpr = sregex :: compile ( "" , regex_constants :: icase ) ; smatch match ; string str ( " !" ) ; if ( regex_search ( str, match, xpr ) ) cout << "icase ok" << endl ; else cout << "icase fail" << endl ; return 0 ; }
16. #include <boost/xpressive/xpressive.hpp> #include <iostream> using namespace std ; using namespace boost :: xpressive ; int main ( int argc, char * argv [ ] ) { sregex xpr = sregex :: compile ( "" , regex_constants :: icase ) ; smatch match ; string str ( " !" ) ; if ( regex_search ( str, match, xpr ) ) cout << "icase ok" << endl ; else cout << "icase fail" << endl ; return 0 ; }
17. #include <boost/xpressive/xpressive.hpp> #include <iostream> using namespace std ; using namespace boost :: xpressive ; int main ( int argc, char * argv [ ] ) { sregex xpr = sregex :: compile ( "" , regex_constants :: icase ) ; smatch match ; string str ( " !" ) ; if ( regex_search ( str, match, xpr ) ) cout << "icase ok" << endl ; else cout << "icase fail" << endl ; return 0 ; }
18. #include <boost/xpressive/xpressive.hpp> #include <iostream> using namespace std ; using namespace boost :: xpressive ; int main ( int argc, char * argv [ ] ) { sregex xpr = sregex :: compile ( "" , regex_constants :: icase ) ; smatch match ; string str ( " !" ) ; if ( regex_search ( str, match, xpr ) ) cout << "icase ok" << endl ; else cout << "icase fail" << endl ; return 0 ; }

')
Some will be surprised that the issuance of the program strongly depends on the platform . Moreover, on one platform, the program may produce different results. It's all about the locale. If we assume that the encoding of the example file is windows-1251, then the “icase fail” result can be achieved on a platform in which the user locale has an encoding different from cp1251. The most common example of such a platform is mingw (downloaded as a sourceforge binary) + Windows. In this case, the boost :: xpressive algorithms simply do not know which characters in the extended part of the cp-1251 code table are letters. And the ctype facet of classical localization is to blame for this. Having reported the correct ctype of localization with which xpressive works, we will achieve the desired result. In the simplest case, if the necessary localization is installed in the system, we just need to make it global

2. // set global localization
3. std :: locale cp1251_locale ( "en_US.CP1251" ) ;
4. std :: locale :: global ( cp1251_locale ) ;

or report it to the regex compiler

2. std :: locale cp1251_locale ( "en_US.CP1251" ) ;
3. sregex_compiler compiler ;
4. // tell the regex compiler what localization to use
5. compiler. imbue ( cp1251_locale ) ;
6. sregex xpr = compiler. compile ( "world" , regex_constants :: icase ) ;

Everything would be fine, but on a platform where localization is not supported ru_RU: CP1251 our code will throw an exception. At best, the name is incorrectly specified, at worst - the necessary localization is not in the system. We will solve this problem by implementing our own ctype facet (it is he who will explain to xpressive which characters are letters and how the case changes).

The simplest example of implementing the ctype facet and the example for encoding CP1251:

2. #include <boost / xpressive / xpressive.hpp>
3. #include <iostream>
5. using namespace std ;
6. using namespace boost :: xpressive ;
8. / ** @ brief A very simplified example of the ctype facet for working correctly with
9. * Encoding Cp1251 * /
10. class ctype_cp1251 : public ctype < char >
11. {
12. public :
14. / ** @ breif mask in ctype_base is an enumeration of all possible types.
15. * characters - alpha, digit, ... * /
16. typedef typename ctype < char > :: ctype_base :: mask mask ;
18. // for brevity, let's rename the constants
19. enum {
20. alpha = ctype < char > :: alpha ,
21. lower = ctype < char > :: lower ,
22. punct = ctype < char > :: punct
23. // other masks
24. } ;
26. / ** @ brief Main constructor. r - characterizes the area of ​​life
27. * cell. For details, see the book of Straustrup. * /
28. ctype_cp1251 ( size_t r = 0 )
29. {
30. // initialize the table of masks. The index is the negative part of the char.
31. // That is, ext_tab [1] - the mask for the character char (-1) - 'I'
32. ext_tab [ 0 ] = 0 ;
33. for ( size_t i = 1 ; i <= 32 ; ++ i )
34. ext_tab [ i ] = alpha | lower ;
35. for ( size_t i = 33 ; i <= 64 ; ++ i )
36. ext_tab [ i ] = alpha | upper ;
37. // ... the rest of the characters in this example are uninteresting
38. for ( size_t i = 65 ; i <= 128 ; ++ i )
39. ext_tab [ i ] = punct ;
40. }
42. ~ ctype_cp1251 ( )
43. { }
45. protected :
47. / ** @ brief Answers the question whether the character c matches the mask m * /
48. virtual bool is ( mask m, char c ) const
49. {
50. if ( 0 <= c && c <= 127 )
51. return ctype < char > :: is ( m, c ) ;
52. else if ( - 128 <= c && c < 0 )
53. return ext_tab [ static_cast < size_t > ( c * - 1 ) ] & m ;
54. }
56. / ** @ brief Converts the character c to uppercase * /
57. virtual char do_toupper ( char c ) const
58. {
59. if ( 0 <= c && c <= 127 )
60. return ctype < char > :: do_toupper ( c ) ;
61. else if ( is ( lower, c ) )
62. return c - 32 ;
63. return c ;
64. }
66. / ** @ brief Converts the character c to lower case * /
67. virtual char do_tolower ( char c ) const
68. {
69. if ( 0 <= c && c <= 127 )
70. return ctype < char > :: do_tolower ( c ) ;
71. else if ( is ( upper, c ) )
72. return c + 32 ;
73. return c ;
74. }
76. // not to complicate the example, we will not override the rest
77. // virtual functions
79. private :
80. // ban on copying
81. ctype_cp1251 ( const ctype_cp1251 & ) ;
82. const ctype_cp1251 & operator = ( const ctype_cp1251 & ) ;
83. mask ext_tab [ 129 ] ; // @ <masks of the extended part of the code table CP1251
84. } ;
86. int main ( int argc, char * argv [ ] )
87. {
88. // create an instance of the cell
89. ctype < char > * ctype_cp1251_facet = new ctype_cp1251 ( ) ;
91. // Create a new localization based on the current, using
92. // defined above facet. You can define a global
93. // localization with the described facet, then all the classes and
94. // functions will use it.
95. locale cp1251_locale ( locale ( "" ) , ctype_cp1251_facet ) ;
97. // create regex compilers with specific localization
98. sregex_compiler compiler ;
99. compiler. imbue ( cp1251_locale ) ;
101. sregex xpr = compiler. compile ( "world" , regex_constants :: icase ) ;
102. smatch match ;
103. string str ( "HELLO WORLD!" ) ;
104. if ( regex_search ( str, match, xpr ) )
105. cout << "icase ok" << endl ;
106. else
107. cout << "icase fail" << endl ;
108. return 0 ;
109. }

Now the result of the program will not depend on the specific platform. By overriding the standard facets or adding new ones, you can control the behavior of the algorithm / program depending on the cultural and linguistic characteristics of users.

A complete description of the std :: local class and techniques for using facets can be found in the 3rd special edition of Bjørn Straustrup’s C Programming Language ++, in the appendix. To clarify the structure of the facets, you can use any STL manual. For example here .

The task of converting encodings is solved by implementing the codecvt facet. If it is interesting, I will tell about it in the next article.

______________________

The text was prepared in the Blog Editor from © SoftCoder.ru