Graphical interface or chat bot in project management: what is more effective? .. Practical experiment

About 30 years ago in many books on artificial intelligence it was stated that in the future, human communication with the computer will take place in natural language, and all other interfaces will become a thing of the past. The same picture can often be seen in various science fiction films. But is the voice interface really more efficient? In our experience, we will replace the project management system in the organization with a chatbot with a voice interface and see what happens.



Small retreat:

I am always baffled by the fact that on Habré for some time now divided "Management" and "Development". And if the article on the development of management systems? Honestly, I have never written anything in the “Management” section, but logic suggests that the current material is more appropriate here - it’s not about programming any more, but about experience in project management.
')
But, sorry for deviating from the topic of the story, we will return.

Stage 1. Start

It all started small. Our company has traditionally had problems with the distribution of tasks and control of the result ... More precisely, as long as 1.5 people worked there were none. And when the state rose sharply to 8, they started - a notepad and instant messenger have become a sharply ineffective means of organizing work. Then we tried to use Trello and Asana, and, to a certain extent, this solved the problem. But one day we were visited by a “great” idea to attach a voice interface and see what happens. It seemed a small matter - “only this” was needed because with a voice or via chat you could create a task and find out what tasks need to be done.

At the time of the beginning of the realization of this idea, there were no analogues (Cortana in Microsoft Dynamics understood only the English teams).

It took a couple of weeks to build the prototype. Initially, machine learning decided not to bother and make the recognition of the meaning of commands on patterns. Based on the fact that templates can be done very much, and believing that the basic set of commands will be small, and everyone will remember them very quickly. For voice recognition, simply plug in the API from Google (experience has shown that it works best).

The first result looked like the image above. Window, input line, and chat.

Since only practice is a criterion of truth, it was decided to introduce this result for half of the employees in order to be able to compare what is more convenient.

Stage 2. It is impossible to use it!

Absolutely everyone involved in testing said this phrase. Measuring the time needed to perform typical operations, we found that it slowed down. For example, creating a task now took more than a minute (in the “control group” 27 - 40 seconds). At first, a stopwatch was used to measure the time.

The reasons turned out to be many. First, speech recognition works “better than human” only in theory (and, probably, only in English). Even the simple “does Dmitry Ivanovich” easily turns into “Dmitry Ivan stinks of HIV”. And to say any long sentence without errors is difficult at all. Therefore, all the polls wrote the command text. Cursing at the same time obscenely, which further complicated the work of the program.

Remembering the command was also not an easy task. Someone remembered, and someone continued to write as it comes to mind. It was found that the subtle difference between “what are the tasks” and “what are my tasks” is not always captured by man. And if during the creation of the task all the necessary information did not fit into one, then you need to hesitate for a long time: “set the deadline for the task to place the button on the website by September 15”

Finally, the fact that the GUI is solved by dragging for five seconds, the chat bot turns into “the task of correcting an error in a program that does not work transferring error correction from section to section”. In short, the command line with voice control does not replace the GUI

Stage 3. Work on the bugs

On good, on stage 2, perhaps, would cost would stop. But the donkey man is a stubborn creature. We analyzed the situation, asking ourselves the question why to explain it to another person in a faster voice, but not to the chat bot. The main conclusions were:

1. People rely on the context and understanding of the task of the interlocutor. Those. “Take this crap there and attach such garbage to it” - much shorter;
2. People use paper and pen to explain something;
3. People understand the unsaid instructions. The meaning of the same phrase changes according to the context;
4. People use short names and sometimes nicknames.

Based on this, new features have been implemented:

1. The ability to use the context of the teams - one or more recently created or mentioned tasks.
2. The ability to mention fragments of the name of the task, similar names, etc.
3. List of employees and search for the most similar names in the list.
4. Minimal GUI to display tasks
5. Expanded the list of options in the templates and added a few dozen new commands.
6. The function of registering complaints about the system

In addition, improved the method of measuring time. Time is now measured by a set of 4 typical use cases (set a new task, find out what to do, note that the task is done, see the results of the work). In each case, 5 examples were added and the average time was calculated. New tests showed an approximate equality of the effectiveness of the chat bot and the usual system. The number of complaints was subjectively somewhat reduced (did people get used to it?), But still there was a lot of discontent.

We also managed to find small (subjective) pluses of the chat bot:

1. Some people claimed that the chat window is less “stressing the brain” because it has fewer buttons and labels
2. When working from a phone, typing commands with text (and we remember that most still do not use voice input) became much easier after the keyboard learned typical commands. It was argued that this is faster than pressing the interface buttons.

As you can see, the pluses are still very shaky. Analysis of the complaints showed that:

1. On desktops, there are still very few buttons for performing frequent tasks.
2. There are a lot of teams, remembering them is hard, writing for a long time
3. Not enough usual functionality

Stage 4. Functional, intellect, interface

In reality, a human programmer will never ask the boss “what tasks I have”, will not wait for the enumeration of all tasks, then say “now I will do the task this and that” (yes, it's funny. And many on chat bots are so and work in reality).

Thinking over the problem, we reduced the set of executing teams to “give the task”, “the task is ready”, “there is a problem”, “give another”.

Although it is easy to say. In practice, “give the task” has become a complex intellectual team that analyzes the priorities of tasks, deadlines, complexity, workload of performers, and selects the next task automatically.

For the performers at that moment came happiness. No need to think! Push nothing! You take the task, you do it, then you take the next one. All together they said - the new system is the coolest! We do not want another.

And we understood the important point of the meaning of the interface in a natural language - not to understand long, hard-wrapped sentences. In fact, a chat bot is only effective if it has an intellect that understands the subject area. When a person turns to a program that understands his speech, he subconsciously expects that the system will be smart and be able to solve some of his problems. Explain what to do, at least. Or maybe she will also appoint the executor, distribute the tasks of the sick employee. But in reality, a chat bot possesses, at best, the intelligence of a dog - “stand, lie, sit”. Therefore, having tried the option when the program “has more rights to decide what to do,” we got an interesting result.

In fact, not so simple. There were many concerns that such a system with an auto selection would not cope with real situations. Plus, a large load is created on the manager, who must supply the system with the necessary information and correctly prioritize (in our version, the system still does not decide what and how, but rather tries to fulfill the set plan). The practice did make some adjustments, but in general, surprisingly, the new method turned out to be simpler and more efficient.

True, from the point of view of the performer, the chat bot was mostly not needed. Enough four buttons with the above names.

The second major innovation was the self-learning module for understanding commands. We have left the recognition of commands on the pattern so far, but added the ability to automatically create new templates based on the analysis of the hits. As a result, a lot of templates were quickly created, all the variations of pronunciation (like “stinks”) were studied, and it became possible to use voice commands (and adapt them to a specific person).

Finally, there was a performance gain - work with the system became faster (on a test set of typical tasks) on average by 45% compared with the “control”.

In addition, I noticed some subjective advantages:

1. You can receive information, set tasks and at the same time walk back and forth near the computer (sometimes it is easier to think so)
2. In the evening and on the road, it’s much faster and easier to create a task from the phone. Therefore, fewer tasks become lost and forgotten.
3. You can talk with the program as a person on the phone

I don’t have a desire to return to “traditional” systems. On the contrary, the idea that you have to enter tasks with a keyboard and mouse causes horror. Although, in an office where there are a lot of people, dictating something to a program is psychologically uncomfortable.

We have also developed the program interface. That's how it began to look like:

The functional is close to what is in any other project management system. Deadlines, dependencies between tasks, sections and subsections, notifications, reports.

Conclusion

As a result, we have achieved increased efficiency. Although it turned out in a sense, the porridge from the ax - to the “ax” (chat bot) had to add a lot of functionality, without which it was useless. Having already had quite significant experience in creating language processing systems, we stepped on a rake many times and spent more than 6 months until the moment when the system began to bring us real benefits.

And, of course, the fact that we managed to tell in the article is only the tip of the iceberg - the most basic points we encountered.

PS