Quiz for bots on Twitch.tv
As you know, there are bots on Twitch.tv. I'm not talking about “bad” bots for cheating the audience or chat, but about bots that allow the streamer to add this or that functionality to the stream or chat in the stream. Among the most famous of them: Moobot, Nightbot and MirRobot. About the latter just will be discussed. Mirrobot is written by our compatriot and deservedly enters the number of the most popular bots. One of its possibilities, which is highly demanded, is to conduct a quiz in the chat.
The essence of the MirRobot quiz: with a certain frequency, the bot asks the audience a question in the chat. If during a certain time, no one can answer correctly, the bot gives hints, indicating the number of letters in the answer and opening some letters (according to the principle of “Hangman” or “Fields of Miracles”).
An example of gameplay in the quiz:
As soon as someone answers correctly, firstly, he is a young man, and, secondly, the bot remembers that this particular spectator answered correctly. In the chat on a special team is also available a list of those viewers who gave the most correct answers to quiz questions. For each channel this list is different.
One day, experimenting with how one could find answers to arbitrary questions on the Internet, I remembered the quiz on Twitch and decided to try to make a bot that would participate in the quiz myself. I also wondered what would happen if the same player was on the quiz top on completely different channels. Does this cause the suspicion of the viewers or the developer? What will be the reaction? Or maybe someone has already made a bot before me, and there are no real people in the top anymore?
The first working version of the bot was written during the night of the May holidays. The bot knew how to connect to the desired channel on Twitch, analyze the chat, recognize when a quiz question appeared in the chat, and search for an answer on the Internet.
')
Unfortunately, they failed to immediately find a normal service that could answer arbitrary questions in Russian. Google and Yandex are not suitable for such purposes, since they do not provide a ready answer, but links to pages on the Internet, which should be, at least, minimally analyzed. For a person, such an analysis is not a problem, but how to make a car do it is a separate task. Ideally, I was looking for something similar to Wolfram Alpha, but with the support of the Russian language. By the way, if anyone has an eye on a similar open service, write in the comments.
For the quiz on Tviche, it turned out to be enough to search for answers on one of the numerous sites devoted to crossword puzzles, crossword puzzles and other similar games. In principle, having tried several similar sites, I chose the one on which the most simple format of the search results page appeared. Looking ahead, I’ll say that it turned out that such “crossword puzzles questions” make up the main MirRobot base. The efficiency of the algorithm for general educational issues turned out to be quite high (70%), and, having written this functionality at the very beginning, it was not necessary to return to it anymore.
After a series of improvements, the bot was able to:
What did not work out or was not implemented for various reasons:
Used technologies:
As a result, the bot was fully independent. The global challenge has been achieved. The last 2 months the bot worked autonomously.
The ability not to stand out was the most important for the success of the bot. It was easy enough to teach the bot to answer questions. Based on its technical capabilities, the bot is in most cases able to answer the quiz question in the very first second, but this immediately arouses suspicion: the person is not able to print the answer, he doesn’t even read the question completely during this time. Another peculiarity is that a person does not equally quickly answer different questions: simple ones - quickly, difficult ones. Therefore, if you simply put any significant delay on the answer, then it turns out that the live participants will “click” simple questions, and the bot will masterfully answer all the complex ones. Such an “erudite” who gives the right answers to completely different specialized questions will immediately attract attention. I did not realize the assessment of the “complexity” of the question.
In the screenshot in the group of one of the streamers, they are indignant about those who are looking for answers to the quiz on Google:
As a result, in order not to stand out among ordinary viewers, it was necessary to significantly reduce the likelihood that the bot will correctly respond:
The result of such “optimizations” (strangely called “optimization” decrease in efficiency) is that on one of the very popular channels with several thousand live viewers and quiz questions with a frequency of 1-2 minutes, the bot won the quiz on average 5-7 times at one o'clock. For channels on which the activity of live viewers was low, the likelihood of a bot to win simply aimed at a basic probability of answering (20-30%).
At the same time, such an understatement of the bot's efficiency allowed for a “clear conscience”. It turned out that the bot answered little and mostly in cases when there was no one among the spectators who would know the correct answer. We can say that the bot competed with “googles”, that is, with those who did not know the answer right away, tried to find it on the Internet, or could only guess after a few clues.
Of course, with such efficiency, the bot could not stand for a short distance with live quiz participants who could easily give the correct answer before the bot. Looking further into the statistics of the bot against the background of the top change, it was clearly understood that an active live player for a limited time period gives more correct answers than the bot. Sometimes on the same active channel, the bot could not afford to give the correct answer within a half to two hours. This state of affairs turned out to be acceptable: the bot did not give many correct answers, did not irritate anyone in the chat with its participation and did not arouse suspicion, but steadily moved to the top quiz, due to the fact that, unlike a person, he did not rest, did not sleep, did not tired, and consistently participated in each stream.
At some point, I “screwed” the chat bot's engine to the bot so that it could respond to private messages and try to completely mimic a person. The engine, judging by the articles on the Internet, showed itself very well in Turing tests. Of course, Twitch has its own specifics of chatting: for example, you need to filter out messages from some emoticons. But quite often it was possible to maintain the illusion, and sometimes the dialogues turned out quite funny. It was a little embarrassing that sometimes the bot trolled to the interlocutor and seemed to be a 10-year-old girl, but in general the problem of communication was solved.
The fact that the bot was in the top at the same time almost 50 channels, no one noticed. Here, in my opinion, there is a large fragmentation of groups of spectators, as well as the fact that of the approximately 2 thousand channels where MirRobot is used, the quiz was included on only a few hundred. If we talk about the channels where there was a quiz, and where there were at least a few dozen people in the chat, there were about 160 such channels, most of which were either irregular rare streams or channels with less than 50 viewers (such were excluded ), or channels on which only short-term experimented with a quiz. There were literally a couple of active channels with a “big” top that could not be entered, and the quiz for them had been turned off for a long time, but the old top remained.
The graph below shows that the bot steadily went to the top of various channels:
On the other hand, the bot was noticed on the channels where there is very little activity in the chat. This included both the channels, where there were just a few spectators, and the channels, where the audience was turned on with a small number of “live” people. On several channels, the bot was still sent to the ban. The fact that it issued, as I understood it, was:
Many who are just looking for answers in Google. These people in the chat do not like, and even against them are actively fighting.
So, at some point, the bot author simply began to replace some of the letters of the question with similar-looking English, which really made the search significantly more difficult. Google and sites with databases of questions are not able to handle such abracadabra. Yandex also copes with such requests, which left the Googlers at least some chances. As for the bot, then it was easy to update the algorithm for analyzing questions.
It is like the eternal confrontation of shield and sword. You can take advantage only in a short period of time.
Surprisingly, it turned out that there are other bots or automatic scripts for playing the quiz! At least once I saw in a chat that someone with an absolutely fresh account answered the quiz literally within 1 second after the question appeared.
There were those who could not be definitely identified as a bot or a script. Nevertheless, with regard to a number of nicks, judging by several factors, I was 90% sure that some kind of automation was used. Here is an example of a log of all messages from a suspicious participant in one channel in 6 hours:
What can be noticed by this log:
The above log is very atypical for a living person. Everything is aggravated by the fact that for a given participant such behavior is observed not just one evening, but already 7 days in a row. Plus, this study was registered quite recently - in early September, probably only for the quiz.
I even tried to talk them in PM about this topic, but the intelligible dialogue did not take place. Whether these people use a bot, some kind of partially automated help, or just sit on the stream all night long to play a quiz - it’s impossible to determine with 100% probability. This question is still open.
It was possible to get something, but - no, there was no benefit. First, the essence of the experiment was not to get any prizes. Secondly, the prizes were not worth it.
For example, on one of the channels a weekly contest is held, in which the top responders are awarded gold in World of Tanks. As it turned out, the bot steadily broke into the top and could get prizes. It turned out later by simply searching for the nickname used in the search engine, which indicated the posts in the relevant community.
Was it interesting? - Yes. The revelation was that someone also automated the quiz game.
Was this helpful? - Yes, but only as the development of technology and the creation of something new.
Ps. So far, the bot continues to work in automatic mode. We assume that the experiment has not yet reached its final stage. So, if you, while straining your convolutions, are fighting in an intellectual quiz, make sure that you haven’t put a smart script against you.
The essence of the MirRobot quiz: with a certain frequency, the bot asks the audience a question in the chat. If during a certain time, no one can answer correctly, the bot gives hints, indicating the number of letters in the answer and opening some letters (according to the principle of “Hangman” or “Fields of Miracles”).
An example of gameplay in the quiz:
As soon as someone answers correctly, firstly, he is a young man, and, secondly, the bot remembers that this particular spectator answered correctly. In the chat on a special team is also available a list of those viewers who gave the most correct answers to quiz questions. For each channel this list is different.
Idea
One day, experimenting with how one could find answers to arbitrary questions on the Internet, I remembered the quiz on Twitch and decided to try to make a bot that would participate in the quiz myself. I also wondered what would happen if the same player was on the quiz top on completely different channels. Does this cause the suspicion of the viewers or the developer? What will be the reaction? Or maybe someone has already made a bot before me, and there are no real people in the top anymore?
The first working version of the bot was written during the night of the May holidays. The bot knew how to connect to the desired channel on Twitch, analyze the chat, recognize when a quiz question appeared in the chat, and search for an answer on the Internet.
')
Unfortunately, they failed to immediately find a normal service that could answer arbitrary questions in Russian. Google and Yandex are not suitable for such purposes, since they do not provide a ready answer, but links to pages on the Internet, which should be, at least, minimally analyzed. For a person, such an analysis is not a problem, but how to make a car do it is a separate task. Ideally, I was looking for something similar to Wolfram Alpha, but with the support of the Russian language. By the way, if anyone has an eye on a similar open service, write in the comments.
Implementation
For the quiz on Tviche, it turned out to be enough to search for answers on one of the numerous sites devoted to crossword puzzles, crossword puzzles and other similar games. In principle, having tried several similar sites, I chose the one on which the most simple format of the search results page appeared. Looking ahead, I’ll say that it turned out that such “crossword puzzles questions” make up the main MirRobot base. The efficiency of the algorithm for general educational issues turned out to be quite high (70%), and, having written this functionality at the very beginning, it was not necessary to return to it anymore.
After a series of improvements, the bot was able to:
- Successfully play the quiz simultaneously in several chat channels.
- Independently find new channels on which MirRobot was turned on and there were some minimum number of viewers.
- Maintain your base of correct answers to questions in order not to rely on the effectiveness of online search and be able to answer specific questions about computer games.
- Keep track of the top quiz participants through channels and take into account their position in the top to make a decision about the answer;
- Connect again in case of disconnection from the Internet.
- Check the consistency of their intended answer and tips in the quiz.
- Work completely autonomously: hosted in PaaS-cloud, provide the necessary analytics to the web for monitoring and maintain the database in DBaaS.
- Simulate chatting with a personal mention (this functionality was implemented much later).
- Simulate the pace and manner of the response of the person.
What did not work out or was not implemented for various reasons:
- The detection of whether a bot is banned in a chat on a channel is not implemented.
- There were two annoying bugs that I could not fix. In case of their detection, the bot simply rebooted.
Used technologies:
- Python 3 (about 1 thousand lines of code).
- IRC (Twitch chat is actually IRC chat).
- Waitress / Flask for web-interface.
- MongoDB (mlab.com).
- PaaS IBM Bluemix.
As a result, the bot was fully independent. The global challenge has been achieved. The last 2 months the bot worked autonomously.
Bot against man
The ability not to stand out was the most important for the success of the bot. It was easy enough to teach the bot to answer questions. Based on its technical capabilities, the bot is in most cases able to answer the quiz question in the very first second, but this immediately arouses suspicion: the person is not able to print the answer, he doesn’t even read the question completely during this time. Another peculiarity is that a person does not equally quickly answer different questions: simple ones - quickly, difficult ones. Therefore, if you simply put any significant delay on the answer, then it turns out that the live participants will “click” simple questions, and the bot will masterfully answer all the complex ones. Such an “erudite” who gives the right answers to completely different specialized questions will immediately attract attention. I did not realize the assessment of the “complexity” of the question.
In the screenshot in the group of one of the streamers, they are indignant about those who are looking for answers to the quiz on Google:
As a result, in order not to stand out among ordinary viewers, it was necessary to significantly reduce the likelihood that the bot will correctly respond:
- Introduce the rule to give answers in no more than 30–40% of cases.
- Increase delay before attempting to respond (random value).
- In some cases, to give the correct answers, despite the fact that someone has just given the correct answer, creating the illusion that the person writes to the chat.
The result of such “optimizations” (strangely called “optimization” decrease in efficiency) is that on one of the very popular channels with several thousand live viewers and quiz questions with a frequency of 1-2 minutes, the bot won the quiz on average 5-7 times at one o'clock. For channels on which the activity of live viewers was low, the likelihood of a bot to win simply aimed at a basic probability of answering (20-30%).
At the same time, such an understatement of the bot's efficiency allowed for a “clear conscience”. It turned out that the bot answered little and mostly in cases when there was no one among the spectators who would know the correct answer. We can say that the bot competed with “googles”, that is, with those who did not know the answer right away, tried to find it on the Internet, or could only guess after a few clues.
Of course, with such efficiency, the bot could not stand for a short distance with live quiz participants who could easily give the correct answer before the bot. Looking further into the statistics of the bot against the background of the top change, it was clearly understood that an active live player for a limited time period gives more correct answers than the bot. Sometimes on the same active channel, the bot could not afford to give the correct answer within a half to two hours. This state of affairs turned out to be acceptable: the bot did not give many correct answers, did not irritate anyone in the chat with its participation and did not arouse suspicion, but steadily moved to the top quiz, due to the fact that, unlike a person, he did not rest, did not sleep, did not tired, and consistently participated in each stream.
At some point, I “screwed” the chat bot's engine to the bot so that it could respond to private messages and try to completely mimic a person. The engine, judging by the articles on the Internet, showed itself very well in Turing tests. Of course, Twitch has its own specifics of chatting: for example, you need to filter out messages from some emoticons. But quite often it was possible to maintain the illusion, and sometimes the dialogues turned out quite funny. It was a little embarrassing that sometimes the bot trolled to the interlocutor and seemed to be a 10-year-old girl, but in general the problem of communication was solved.
Have you noticed the bot?
The fact that the bot was in the top at the same time almost 50 channels, no one noticed. Here, in my opinion, there is a large fragmentation of groups of spectators, as well as the fact that of the approximately 2 thousand channels where MirRobot is used, the quiz was included on only a few hundred. If we talk about the channels where there was a quiz, and where there were at least a few dozen people in the chat, there were about 160 such channels, most of which were either irregular rare streams or channels with less than 50 viewers (such were excluded ), or channels on which only short-term experimented with a quiz. There were literally a couple of active channels with a “big” top that could not be entered, and the quiz for them had been turned off for a long time, but the old top remained.
The graph below shows that the bot steadily went to the top of various channels:
On the other hand, the bot was noticed on the channels where there is very little activity in the chat. This included both the channels, where there were just a few spectators, and the channels, where the audience was turned on with a small number of “live” people. On several channels, the bot was still sent to the ban. The fact that it issued, as I understood it, was:
- He did not respond to arbitrary requests to him in private messages and in the general chat. I implemented the chatbot engine much later than the start of the project.
- On the round-the-clock channels - present around the clock and periodically answered in the chat.
What about others?
Many who are just looking for answers in Google. These people in the chat do not like, and even against them are actively fighting.
So, at some point, the bot author simply began to replace some of the letters of the question with similar-looking English, which really made the search significantly more difficult. Google and sites with databases of questions are not able to handle such abracadabra. Yandex also copes with such requests, which left the Googlers at least some chances. As for the bot, then it was easy to update the algorithm for analyzing questions.
It is like the eternal confrontation of shield and sword. You can take advantage only in a short period of time.
Surprisingly, it turned out that there are other bots or automatic scripts for playing the quiz! At least once I saw in a chat that someone with an absolutely fresh account answered the quiz literally within 1 second after the question appeared.
There were those who could not be definitely identified as a bot or a script. Nevertheless, with regard to a number of nicks, judging by several factors, I was 90% sure that some kind of automation was used. Here is an example of a log of all messages from a suspicious participant in one channel in 6 hours:
What can be noticed by this log:
- All messages are very short and unrelated. This is an attempt to give answers to the quiz. In addition to attempts to answer the chat for a few hours from this member was nothing. That is, a person does not write anything else to the chat.
- All messages are in small letters and no typos.
- There are almost no messages with a time difference of less than a minute, that is, only one answer attempt is made to the selected quiz question. In the case where the answer must be in a certain form (case, verb tense, and so on), no attempt is made to guess the form of the word.
- The average difference between messages on average is 3-5 minutes, while 1–1.5 minutes pass between quiz questions on this particular channel. It is strange that a person answers only some selective questions, despite the fact that the quiz is almost constant.
The above log is very atypical for a living person. Everything is aggravated by the fact that for a given participant such behavior is observed not just one evening, but already 7 days in a row. Plus, this study was registered quite recently - in early September, probably only for the quiz.
I even tried to talk them in PM about this topic, but the intelligible dialogue did not take place. Whether these people use a bot, some kind of partially automated help, or just sit on the stream all night long to play a quiz - it’s impossible to determine with 100% probability. This question is still open.
Was there a catch?
It was possible to get something, but - no, there was no benefit. First, the essence of the experiment was not to get any prizes. Secondly, the prizes were not worth it.
For example, on one of the channels a weekly contest is held, in which the top responders are awarded gold in World of Tanks. As it turned out, the bot steadily broke into the top and could get prizes. It turned out later by simply searching for the nickname used in the search engine, which indicated the posts in the relevant community.
Some statistics
- Work time: 4 months
- Given correct answers: ~ 8 thousand (on average 60–70 per day)
- Collected in its own database of questions and answers: ~ 83 thousand (110 thousand in total in the quiz)
- Located in the top ~ 50 channels (out of 160 with a quiz)
- Banned in at least 3 channels
Conclusion and conclusions
Was it interesting? - Yes. The revelation was that someone also automated the quiz game.
Was this helpful? - Yes, but only as the development of technology and the creation of something new.
Ps. So far, the bot continues to work in automatic mode. We assume that the experiment has not yet reached its final stage. So, if you, while straining your convolutions, are fighting in an intellectual quiz, make sure that you haven’t put a smart script against you.
Source: https://habr.com/ru/post/397521/
All Articles