We write the filter of "bad" words
Many of you have probably been to public multiplayer chat rooms more than once. Whether it is a forum, a webchat or a chat server in the provider's LAN, an ideal order can be achieved only with a small number of users (no more than a hundred, in my opinion). When the community grows, there are about the same problems: mate, spam and flood - one bored individual posts “Everybody to Kantra!” Every minute, another curses with and without and stuff like that.
The ideal mat filter has not yet been invented. But we are not going to do this, and we will consider the implementation of the necessary minimum.
Of course, we are talking to him, who is arguing :). From my point of view, a mat can be a place for wit or expressing more expression. But not everyone is so easily perceived by him and with this it is necessary to reconcile (or create your own community a la “how much matter will fit”). Especially unsightly is the flow of weakly motivated mat from not too smart / educated citizens. Well, it will leave on their conscience.
')
The Russian language is very rich in word and phrase formation, so you can think up a lot of abusive constructions - sometimes you wonder. So we (for now) will not try to deal with these phenomena. The solution to the forehead is a dictionary filter. The dictionary itself, I think, will not be difficult to collect. I got about 250 words :).
But the first experiments show that this method does not give a positive result. Users are trained to bypass it extremely quickly, seeing the failures of a pair of first fire victims. How do they do it? Basically several options:
Regular expressions come to the rescue. Create a replacement table of characters with the above.
Now we need to make a regular expression library from our dictionary. In principle, they can be constructed on the fly. The principle of analysis is simple: any sequence from the table can be in place of a letter. So that people do not cunning by inserting spaces, dots, etc. between the letters to bypass the filter - we also consider their appearance between the letters. Well, so that there are no false positives, we will only work out the entire word, without taking into account the derivation of complex structures.
Yes, the latter of course somewhat reduces the effectiveness of the filter, but still, the modified words are no longer very mat and we can cut off most of the basic words and phrases with a dictionary. In principle, if the filter is triggered only to hide ugly words (and not for an immediate ban-Hammer :)), then this condition can be discarded.
As a result, for the well-known word of three letters the regular form will look something like this:
In general, the essence is simple: we make a dictionary, from it we generate a library of regulars and apply them in our filter.
Disadvantages of the approach:
Those. in general, it is not a panacea, but it has a certain deterrent effect.
Flood or meaningless stream of characters, smiles, sentences can be annoying. In our context we will also consider spam as a kind of flood, repeating phrases like “Fse f kantra! Created on xxxx "or" People, go to my super mega-link ". However, these phenomena are more characteristic of chat rooms.
You can track this whole thing purely mechanically, without using any semantic text analysis - are there simple methods to do this?
With flood, repeated symbols and smiles, in general, everything is clear: we watch the most frequent flooding schemes in live chat and model them using quantifiers. Just some examples:
The situation is more complicated with repetitions of phrases. The approach to the forehead is rather primitive: we compare the n-last messages of the user and in the case of coincidence we “work”.
The first method of resistance is to add spaces to the line, change the number of punctuation marks (dots, exclamation marks). However, we can normalize the line, taking into account such changes: for example, we simply remove all unnecessary.
However, here we are waited by failure with the second wave of resistance: the phrase begins to change slightly, add interjections, introductory words, etc. This is where fuzzy comparison algorithms come in handy.
For example, you can calculate the Levenshtein distance . The implementation of this algorithm is not very complicated and is available in most popular languages, and in php it is generally native. In fact, it returns the line difference in the number of edits required to get the second line from the first.
Next, the case of technology: we determine the minimum size of the message to which it is worth responding, count the distance, and if it does not exceed, for example, 20% of the message length, then count it again! It's simple. The length of the message and the threshold of the distance is chosen empirically, so as not to interfere with ordinary users.
This method, of course, will help only in counteracting bored individuals who send out all by hand (hence the minor edits and the repeatability of phrases). If the user sends disparate messages with frightening speed, then most likely this is a spam bot and other approaches or actions of a live moderator are needed here.
So, we have considered in general terms the methods of countering mate, flood and spam. The proposed methods are most relevant for chat systems of short messages. In many ways, the simplicity of the approaches is due to the use of such a generally rather difficult thing as regular expressions.
There are alternative methods, including attempts to morphological and semantic analysis of phrases. However, if you follow the ideology of the same AIML, human-like behavior can be obtained quite empirically, without embedding between input and output "too much computer."
I hope the information will be useful to anyone. I would welcome any comments. As time comes, I will try to present my own implementation of the above actions to the public.
The ideal mat filter has not yet been invented. But we are not going to do this, and we will consider the implementation of the necessary minimum.
Mat
Of course, we are talking to him, who is arguing :). From my point of view, a mat can be a place for wit or expressing more expression. But not everyone is so easily perceived by him and with this it is necessary to reconcile (or create your own community a la “how much matter will fit”). Especially unsightly is the flow of weakly motivated mat from not too smart / educated citizens. Well, it will leave on their conscience.
')
The Russian language is very rich in word and phrase formation, so you can think up a lot of abusive constructions - sometimes you wonder. So we (for now) will not try to deal with these phenomena. The solution to the forehead is a dictionary filter. The dictionary itself, I think, will not be difficult to collect. I got about 250 words :).
But the first experiments show that this method does not give a positive result. Users are trained to bypass it extremely quickly, seeing the failures of a pair of first fire victims. How do they do it? Basically several options:
- Replacing consonants with paired ones consonant with them (for example, “x” by “k”);
- Replacing letters with similar looking Latin (“n” - “n”, “and” - “u”);
- A mixture of Cyrillic and translit;
- The use of pseudographics.
Regular expressions come to the rescue. Create a replacement table of characters with the above.
(
'' => [ '' , 'a' , '@' ],
'' => [ '' , '6' , 'b' ],
'' => [ '' , 'b' , 'v' ],
'' => [ '' , 'r' , 'g' ],
'' => [ '' , 'd' , 'g' ],
'' => [ '' , 'e' ],
'' => [ '' , '' , 'e' ],
'' => [ '' , 'zh' , '*' ],
'' => [ '' , '3' , 'z' ],
'' => [ '' , 'u' , 'i' ],
'' => [ '' , 'u' , 'y' , 'i' ],
'' => [ '' , 'k' , 'i{' , '|{' ],
'' => [ '' , 'l' , 'ji' ],
'' => [ '' , 'm' ],
'' => [ '' , 'h' , 'n' ],
'' => [ '' , 'o' , '0' ],
'' => [ '' , 'n' , 'p' ],
'' => [ '' , 'r' , 'p' ],
'' => [ '' , 'c' , 's' ],
'' => [ '' , 'm' , 't' ],
'' => [ '' , 'y' , 'u' ],
'' => [ '' , 'f' ],
'' => [ '' , 'x' , 'h' , '' , 'k' , '}{' ],
'' => [ '' , 'c' , 'u,' ],
'' => [ '' , 'ch' ],
'' => [ '' , 'sh' ],
'' => [ '' , 'sch' ],
'' => [ '' , 'b' ],
'' => [ '' , 'bi' ],
'' => [ '' ],
'' => [ '' , '' , 'e' ],
'' => [ '' , 'io' ],
'' => [ '' , 'ya' ],
)
* This source code was highlighted with Source Code Highlighter .Now we need to make a regular expression library from our dictionary. In principle, they can be constructed on the fly. The principle of analysis is simple: any sequence from the table can be in place of a letter. So that people do not cunning by inserting spaces, dots, etc. between the letters to bypass the filter - we also consider their appearance between the letters. Well, so that there are no false positives, we will only work out the entire word, without taking into account the derivation of complex structures.
Yes, the latter of course somewhat reduces the effectiveness of the filter, but still, the modified words are no longer very mat and we can cut off most of the basic words and phrases with a dictionary. In principle, if the filter is triggered only to hide ugly words (and not for an immediate ban-Hammer :)), then this condition can be discarded.
As a result, for the well-known word of three letters the regular form will look something like this:
m/(?:^|\W)((?:|x|h||k|}{)\W*[yu]\W*[uyi])(?:$|\W)/ig;
In general, the essence is simple: we make a dictionary, from it we generate a library of regulars and apply them in our filter.
Disadvantages of the approach:
- A large dictionary is needed;
- The morphology of the Russian language is not taken into account;
- False alarms are possible (“Forge while the iron is hot!”)
Those. in general, it is not a panacea, but it has a certain deterrent effect.
Spam and flood
Flood or meaningless stream of characters, smiles, sentences can be annoying. In our context we will also consider spam as a kind of flood, repeating phrases like “Fse f kantra! Created on xxxx "or" People, go to my super mega-link ". However, these phenomena are more characteristic of chat rooms.
You can track this whole thing purely mechanically, without using any semantic text analysis - are there simple methods to do this?
With flood, repeated symbols and smiles, in general, everything is clear: we watch the most frequent flooding schemes in live chat and model them using quantifiers. Just some examples:
/(.)\\1{15,}/is;
/(.{3,})\\1{5,}/is;
/(\n|\r|\r\n)\\1{3,}/is;
The situation is more complicated with repetitions of phrases. The approach to the forehead is rather primitive: we compare the n-last messages of the user and in the case of coincidence we “work”.
The first method of resistance is to add spaces to the line, change the number of punctuation marks (dots, exclamation marks). However, we can normalize the line, taking into account such changes: for example, we simply remove all unnecessary.
However, here we are waited by failure with the second wave of resistance: the phrase begins to change slightly, add interjections, introductory words, etc. This is where fuzzy comparison algorithms come in handy.
For example, you can calculate the Levenshtein distance . The implementation of this algorithm is not very complicated and is available in most popular languages, and in php it is generally native. In fact, it returns the line difference in the number of edits required to get the second line from the first.
Next, the case of technology: we determine the minimum size of the message to which it is worth responding, count the distance, and if it does not exceed, for example, 20% of the message length, then count it again! It's simple. The length of the message and the threshold of the distance is chosen empirically, so as not to interfere with ordinary users.
This method, of course, will help only in counteracting bored individuals who send out all by hand (hence the minor edits and the repeatability of phrases). If the user sends disparate messages with frightening speed, then most likely this is a spam bot and other approaches or actions of a live moderator are needed here.
Total
So, we have considered in general terms the methods of countering mate, flood and spam. The proposed methods are most relevant for chat systems of short messages. In many ways, the simplicity of the approaches is due to the use of such a generally rather difficult thing as regular expressions.
There are alternative methods, including attempts to morphological and semantic analysis of phrases. However, if you follow the ideology of the same AIML, human-like behavior can be obtained quite empirically, without embedding between input and output "too much computer."
I hope the information will be useful to anyone. I would welcome any comments. As time comes, I will try to present my own implementation of the above actions to the public.
Source: https://habr.com/ru/post/101232/
All Articles