📜 ⬆️ ⬇️

Telegram-bot for Redmine. How to simplify the life of yourself and people

In any company that uses a project and task management system, sooner or later there is a desire to combine it with some popular messenger to simplify communications. Especially if through this system there is interaction with customers.

The article will discuss how to make friends with Redmine Telegram and not break the existing business processes.



A small background, how the idea of ​​a subject was born.
')
Our company is engaged in the administration of servers and sites 24/7, the development of infrastructure for large and developing Internet projects, the design of fault-tolerant systems and the automation of DevOps.

To interact with customers, we use Redmine, which we zatyunili under our processes and hung on it a number of pleasant things that make our lives easier.

When we started, it was in 2011, we decided that we would communicate with customers only through our task tracker. This is due to the fact that in addition to supporting the servers and responding to incidents, most of our work consists of infrastructure development tasks. It is important for us at any moment to understand well the current state of the infrastructure, the vector of project development, properly assess needs and bottlenecks, propose solutions, coordinate them and implement. The matter is complicated by the fact that there are a lot of projects, not one person works at each, but a team of administrators. At the same time, administrators work in shifts, some of them may not always intersect with each other in the office, and tasks need to be done. Therefore, the issue of synchronization of knowledge and actions within the team and between shifts is extremely important for us.

Considering all these factors, we decided that the only possible way to keep the situation under control is to ensure that all communications and all work take place in one place - in the system of tasks. Naturally, we run project wiki and do much more to solve the problem of synchronization of actions and context, but the key point has always been a single point of interaction with customers. If you add another channel to the communications, for example, instant messengers, say, for a quick discussion of some issues, 100% customers will begin to abuse this, we will quickly drown in the abyss of endless comments, and important information will start to get lost. And the prospect of chaotic communication with clients in 5 different windows against the Redmine tasks, which were decomposed into clear business processes, did not appeal to administrators.

In the end, we lived for about 5 years. During this time, many customers have come to love our Redmine, its simplicity and functionality. Someone, having worked with him with us, decided to implement it in his company. Only occasionally isolated clients were interested in whether it was possible to set tasks for us not only through the system of tasks, but also through some instant messenger, because it would be more convenient.

But over the past 2 years the number of such requests has increased significantly. So much so that it was impossible not to take them into account. And we once again thought about what to do with it. Sales managers, as usual, tried to push the idea in some miraculous way to embed communication with clients via Skype, WhatsApp or Telegram into our work processes. Administrators absolutely did not want this.

After violent disputes and discussions, we had the idea to develop a Telegram bot that would allow all customer messages to be laid out according to existing tasks in Redmine or create new ones where administrators will work with them in their familiar environment.

Squared a small plan - first the minimum functionality, then the pleasant little things. After some time, we received a prototype of the future bot, who successfully delivered a message written to him from Telegram to Redmine, and vice versa! This gave confidence that the idea is viable. And we have made every effort to bring it to the end and run.

In order to better explain how it all works - the article is divided into two blocks. The first block contains a general description of the bot and examples of its work. In the second - the device and technical aspects of the bot.

At the end of the article are links where to download and how to set up our bot.

So, what have we got?

User Authorization


First, in order for the user to work with the bot, it must be authorized.
To do this, the user must have an active Redmine account.

He needs to invent a username and register it in the settings of his Telegram account:



The same username must be entered in the settings of the Redmine-account in a special additional field “Telegram”:



Next, the user finds a bot in the Telegram and presses Start. The bot authorizes it and welcomes:



Setting goals


In order to set a new task or add a comment to an existing task, it is enough just to write text to bot, and he will suggest options. In this case, you can attach files to the message:



A written comment can be added to the last active task, select a task from the list, create a new task based on it or do nothing and terminate the dialog.

If you select the “Add to the last task” item, the user's comment will be added to the task that was last answered. Bot shows it above.

If you select the “Create a new task” item, the bot will switch to the new task creation mode:



And will offer the user:








The user can skip these steps, if he is satisfied with the default values, and immediately click "Create a task", you only need to specify the title for it:



After this, the bot will notify the client about the creation of the task (the link to the task is clickable and, if necessary, you can immediately go to the ticket if necessary):



Which in Redmine will look like this:



If the user clicked on "Select task" - the bot will offer a list of currently open tasks to which you can send a written comment:



At the same time, the user sees only those projects and tasks to which he is connected to Redmine. This applies to all actions in the bot.

Correspondence on tasks


Bot supports the delivery of notifications of changes in tasks. For example, our administrator completed the task and asks the client to check the result:



Here is what the customer will get:



You can respond to tasks through the Reply to the corresponding message in the Telegram. This is especially convenient when work is being done on several tasks at once (the answer will fall right into the right ticket without additional questions):



Total. The client created the task, communicated on it, and did all this through the Telegram. He never had to go to the Redmine for this. The employee received and processed the task in his usual form in Redmine, as part of standard business processes, he did not have to switch to the instant messenger, and then take care of transferring the correspondence to the task system.

Attachment support


In the bot, we implemented support attachments. And it works in both directions. You can attach anything: photos, documents, video and audio. You can even attach stickers, though they look so-so in the browser, and it’s a pity, sometimes a well-chosen sticker extends the life of our entire office.

Suppose a monitoring system reported a problem to us on one of the servers. The administrator through Redmine created the task and attached a monitoring system schedule to it. The client will receive the following message:



To which he can reply with a voice message:



The voice message will be delivered to the task and listened by the administrator.

Restrictions


Of course they are. We encountered the following:

  1. Long messages. Telegram has a limit on the length of one message - 4096 characters. Initially, long comments from Redmine were simply cut off. I had to write a handler who broke long comments into parts. And so that it was pleasing to the eye, that is, so that the commentary did not break strictly on the 4096 character in the middle of the word, and the splitting took place neatly, at the junction of the phrases. For customers, the inconvenience is that in this case several messages will arrive.
  2. Formatting Formatting in Redmine and Telegram are two different universes. But it is understandable, and their purpose is somewhat different. Redmine formatting is displayed in Telegram as service characters, and sometimes it is not very convenient to read. So far we have not figured out how to display them in a normal form, but do not give up hope to eliminate this trouble. From Telegram to Redmine, too, not everything is transferred. For example, if the text contains certain emoticons, then having received such a message Redmine will simply cut it off. These are features of the Redmine database structure. We have some ideas for solving this problem and they work correctly on a test bench, but for now we are afraid to use them on a combat base.
  3. Names of images. If in Telegram you make an attachment of an image (namely images), then the bot will receive a file with the name in the format file_xyz.jpg. Accordingly, in Redmine it will fall with the same name. Here we can not do anything yet, because the bot itself receives files with such names.
  4. The interface and all the bot messages are only in Russian, but in one of the upcoming versions we plan to add support for other languages.

Usage statistics


A little more than two months have passed since launch. First observations:

  1. Currently, about 3.5% of tasks are created and updated by clients through a bot.
  2. As already mentioned above, the bot found application not only among clients, but also actively used by ourselves. It is very convenient to receive notifications on tasks (arrive faster than in the mail) and immediately respond to them. Feels like, with the launch of the bot, we have increased the speed of communication on internal tasks.

Briefly about how our bot works



So let's get started ...

The basic scheme of the bot is as follows:



It seems to be nothing complicated - all changes to Redmine using hooks are delivered to the bot, he looks at the type of event, in which project it happened, determines the Telegram accounts of the recipients of this message and delivers them. From the Telegram side, everything happens in a similar way - the type of the message is determined, to which task it belongs and is delivered to Redmine.

Now let's add some action and get acquainted with some features of message delivery in Telegram!

Telegram breaks large messages into smaller ones. These messages come with some delay. All attachments are delivered as separate messages and sometimes with a long delay (as they are downloaded by the user). Those. it’s just not possible to take and forward all received messages, otherwise we will spam users and spoil our karma.

And what to do? We decided to create a message queue for this and use Redis to store them. Each incoming message our bot puts in a queue and updates the delay time for its processing. When this delay has expired, we consider that the message has been completely formed and now you can choose a task and send it to Redmine.

Next, the question arises: how to choose the desired task in Redmine? After all, in order for the bot to understand what to do with the data received, he needs to ask the user about it, and this, anyway, is another message (even if it is a button click) and it must somehow be linked to the user's previous actions. This session will help us.

The first incoming message from the user's Telegram account generates a session that contains:


Each subsequent message from Telegram may either supplement the session with data, change its state (for example, start the process of further processing and delivery of data), or destroy the session (for example, if the user changed his mind to write his comment), or cause an error (for example the type of incoming data does not correspond to the current state of the session).

Another important component of the bot is caching. Our Redmine consists of a fairly large number of projects, users and some other data. The bot often needs to access this data and if he goes to Redmine every time, this will lead to noticeable delays in processing requests.
The bot has a separate process, which only deals with the fact that it periodically receives data from Redmine, which is required for other processes, and stores it in Redis.

What about MySQL?

It was noted above that, in addition to Redis, the bot also uses MySQL. Of course, all data could be stored in Redis, but despite the presence of periodic dumps - the data is still stored in the memory and an unexpected system failure can lead to their loss, and among them there is very important information that needs to be treated especially reverently. To better understand what is being said, let us recall the first part of the article. It was said that the user can not only choose from the dialogue with the bot how to proceed with his message (create a new task or select an existing one), but also make an answer to the message. And the bot must understand what task to add it to.

In Telegram messages have identifiers that are not related to the task identifiers in Redmine. In order to make friends with them, you need to establish some correspondence and decide how to store this data. But before that, let's try to understand how this data is formed, what will happen if this data is lost, and based on this - understand their importance.

Suppose a user writes his first message in a Telegram bot. The bot opens the session and asks the user what task to add a comment to. After the answer is received - the bot sends the data to Redmine and at the same time maintains this correspondence in itself.

Further, when the user makes a reply to this message in the Telegram, the bot will receive the number of the “parent” message and determine the number of the Redmine task to which the comment should be sent. The described correspondence will also be established for this new message, i.e. reply then you can do it. The same is true for events coming from Redmine. Our clients believe that this is a very convenient mechanism and, judging by the statistics, they use it very actively.

Now let's imagine that all these matches we have lost. Those. The client habitually clicks the reply on the message that came to him from the administrator, writes his comment and sends it to the bot. But since the bot according to the described algorithm is no longer able to understand what task to send a comment, it will give an error message and ask additional questions in order to find out what to do next.

Other important data types that cannot be lost (moreover, they cannot be lost even more than the first ones) are the Redmine and Telegram user IDs. If this happens, users will simply stop receiving information about events occurring in Redmine until something is written to the bot. And since according to the polls, the bot is almost more often used as a notification agent (to replace the mail), then we can get even more trouble.

In order not to upset users - put this data in a less fast, but more reliable MySQL.

In fairness it should be noted that our team is still debating whether to abandon MySQL and transfer this data to Redis. Therefore, it is possible that in the future we can fully switch to working with Redis, and maybe in general to some other data storage system. Fortunately, the architecture of our application allows making such replacements relatively simple.

Scaling and failover


Now I would like to talk about how to ensure the scaling and resiliency of the bot ...

With the advent of the bot, our Redmine in some cases turned into a chat and the intensity of messaging is constantly growing. Some of our clients also want to integrate it in their infrastructure. And considering that our development is completely open source - it’s not known what kind of company or administrative / developer team will want to use it and it’s not known what kind of bot will face. Therefore, we are still at the stage of elaboration of the project laid in him the possibility of scaling.

In our project, scaling can be divided into two classes:


To understand how to scale the bot - let's take a closer look at the process of receiving and processing messages.

The concept is as follows:



The only task of the “Rest API” process is to put a message received from a Redmine or Telegram into a queue (ra-queue) and expect the following. This queue is constantly monitored by the processes of “Queue-workers” and when data appears in it, they are read.

Messages from Redmine are immediately processed and sent to the desired users in the Telegram. With events from Telegram all is more difficult. Since they cannot be immediately sent to the Redmine (the reasons are described above), then they are shifted to another queue already divided by users. The type of message determines the time during which we are waiting for other messages. Sooner or later, all these messages are read, undergo preliminary training, in one way or another they change the state of the session and can be sent to Redmine with all attached files that are downloaded from the Telegram at the same step.

It is important to note here that, thanks to this approach and the existing locking mechanism, processing always takes place on one particular node. This is what allows you to create as many bot instances and with as many “Queue-workers” processes as necessary within the existing load.

When Redmine receives a message, the hook is triggered and the same message is sent back to the bot, after which it is delivered to the rest of the Telegram accounts subscribed to the task.

Thus, the bot infrastructure in a distributed version can be represented as follows:



But everything described in no way limits the ability to use the bot in the usual standalone version with one MySQL server and the usual non-cluster Redis option.

How to try the bot?


To demonstrate the work of the bot, we made a demo-version available at demo.nxs-chat.nixys.ru .

To try it in action - you must perform a few preliminary steps:



Important note! The bot will send messages to Telegram in the following cases:

How to get a bot?


The bot is completely open and can be received both as source codes and as a package (packages are currently available only for Debian 8, but will soon appear for Debian 9 and CentOS 7).

Link to the Github repository with bot source codes . There is also an instruction for installing a bot from packages and setting it up.

Conclusion


In the near future we plan to add the following functionality:

Source: https://habr.com/ru/post/347526/


All Articles