Effective technical support tools

If you look at the set of tools used by our outsourcing technical support in your daily work, you can see that they all serve the same goal - to answer customer questions and requests as quickly and efficiently as possible. And everyone had their own good reason to appear.

In the entire spectrum of existing IT services, technical support has always been the salt of the earth, the alpha and the omega of computer-related services. Support exists for both software products and infrastructure. And the goal is everywhere the same for her - to help the user and solve his problems, to be a link between him and the serviced infrastructure. And if the personal and professional qualities of technical support staff remain outside the scope of my influence ( I am the administrator, not a psychologist ), then creating a comfortable ecosystem for work is one of my direct professional duties. Technical support supports users, and I support technical support.
')

Systems about which the story goes

• Service Desk
• Time tracking system
• Monitoring
• Collecting logs
• Internal Wiki
• Password storage
• Telephony
• CMDB
• Remote control
• Terminal
• Profitability Analysis System

And the first was service desk

Servicedesk, also known as tiketnitsa, solved one of the global problems of technical support - rolling up applications in the context of limited resources. All applications and related data previously recorded on paper and in journals. All this was periodically lost, spoiled, it grew legs and joyfully went on foot to Zimbabwe. What undoubtedly pleased the local natives. Suffice it to say that according to one application, the client expected a reaction from us within 4 months and did not wait.

The implementation of the service desk turned into a whole epic, because it imposed on employees additional responsibilities for its management. And they desperately resisted. I had to make the introduction smooth and gradual. The choice of service desk was also a non-trivial task, but in the end, I settled on Request Tracker. In principle, he met my basic needs. And, of course, where without dopilok. At first, the service desk was installed in a pilot, test mode. Each employee gently took up his sleeve and sat down at the computer. Then he gave a hot, energetic lecture on how cool life is with the service desk, how it improves our lives, and in general, just give me time, and the star of captivating happiness rises. Little by little, it began to fill.

Time tracking system

The next step was the system of writing off time. Servicedesk has been refined, additional fields have appeared in the applications, and the manager has the opportunity to evaluate what and how technical support spends time. For the possibility of technical support "in one window" added the ability to receive applications for e-mail, as well as from Zabbix. That is, upon the occurrence of an event of a certain level of importance, a request is created in the service desk, which the duty officer assigns to one of the employees. Now based on data from the service desk, reports are being prepared for managers. They can pinpoint exactly what technical support was spending time during the week.

Monitoring

The monitoring system was introduced at about the same time as the service desk. As such a system, I stopped at Zabbix, because it was she who had enough power and reliability along with a certain ease of control. Its development cannot be called purposeful, and now, within the framework of systematization, the system is globally reworked and optimized. Nevertheless, she began to solve a certain circle of problems even then. I began to quickly find out about typical failures in the client’s system and respond faster to accidents. After integrating with the service desk, it became much more difficult to ignore the accident, because an unclosed application immediately raised questions. Over time, Zabbix was overgrown with my own checks, scripts and optimizations. SMS sending was also introduced at the occurrence of critical events.

Most recently, monitoring has moved to PostgreSQL, because MySQL performance is no longer satisfied. Not without minor squirmish, but they all managed to win. Now it is difficult to imagine my life without this tool. I expect to seriously improve and develop it. First of all, I want to completely rework the system of patterns and triggers, move away from the standard ones and look at monitoring precisely from the perspective of uninterrupted provision of services, when success is considered to be prevented by an incident, and not quickly eliminating its consequences.

Collecting logs

The second part of the monitoring system can be considered the collection of logs. Admins do not talk about it so much, because there is a perception that picking logs is a thankless task. What I fundamentally disagree with, because some emerging hardware failures, for example, a violation of the memory can be much faster calculated by the corresponding logs than using any other metrics.

At one time, we had a system of collecting logs as a pilot project, but after the move, they never reach out to restore it. ElasticSearch was used as the core of the system as one of the highest-quality tools for quickly searching in huge data arrays. Kibana was first used for visualization, but after upgrading to version 4, it became inconvenient. I had to go to Grafana. The primary parser was Logstash, which I want to replace with the more lightweight Heka. In general, the classic stack for collecting logs. To reduce the load on the disks, it is planned to collect only messages of the warning level and above, and store them not for a year, as was done initially, but for about a month. Then I will link the logging system with specific metrics and link it with Zabbix.

Internal Wiki

To provide assistance in a timely and efficient manner, technical support must store a lot of information, and not only about customers. These are internal regulations, guidelines, data providers, and all sorts of notes. And the storage should be universal, not requiring the installation of special software, preferably accessible from anywhere, where there is Internet. And this is a wiki.

It all started with MediaWiki, because I already had experience using this engine. However, a number of complaints have accumulated during the application, as a result of which I decided to change the wiki. After a bit of searching, I stopped using Dokuwiki. And it seems to me that this is the ideal documentation engine. Starting with the fact that all data is simply stored in files, and neymspeysy - it's just a folder in which the files are located. And ending with all sorts of amenities, like dice with comments, built-in support for syntax highlighting, categories, tags, users and groups, as well as integration with the domain.

A separate awesome chip - the creation of isolated feats, with its main menu, start page and other, to which you can give access to individual users. It is very convenient to provide it to client employees for self-registration of information on a wiki, while not giving access to the entire documentation field. Not everything went smoothly at once. First, the articles had to be moved “by hand”, the syntax was slightly different and autotransfer failed. Secondly, at first there was a lot of confusion between namespaces, categories and tags. I regularly had to manually move articles created anywhere. Third, not everyone was familiar with the concept of the wiki: “look, if you haven’t found it, create it”. But all this is over, now the backbone of documentation has been formed, regulations for processes are being actively written, a validation system has been established, and rules for the design of articles have been developed.

Password storage

The next pain point was passwords. They turned out to be many, not systematized, they were stored in several places, conflicted with each other, were incorrect and created a lot of headaches with remote tuning. I began to look for what can be done about it. I have seen almost all of the web applications for storing passwords (2013). Why the web? Universality, nothing is required except a browser, there is access from everywhere, a simplified installation ( if you already have a web server, add another site there - 10 minutes ).

The first tool used was Teampass. He was powerful, supported almost everything that was required, but ... turned out to be utterly stupid inside. His engine was made of several others, nailed to each other with rusty nails and tied with blue electrical tape. Half of the stated things simply did not work, the other half worked crookedly and randomly broke, it was necessary to constantly fix and patch something. Special thanks for the installation procedure. She just broke the base when updating the version, and finally and irrevocably ( you still do not back it up - then we go to you ). I have an article with a long list of patches that I had to impose on the installer so that everything went fine. Well, the speed of his work left a wish and a wish. By the way, I recently watched its latest version in test mode - everything has become much better. The code was cleaned, almost all the bugs were fixed, although there are still two billion pages in the database log, regardless of how many records there are.

Inner Chernyshevsky asked his eternal question: "What to do?", And I began to look further. In the end, stopped at Rattic. Yes, he looks a little frivolous, but inside is not so bad. Written in Django, with a Bootstrap interface ( hello, mobility ), supports everything you need, integrates with LDAP. There is a separate password page, that is, you can link to it on the Wiki, for example. Does not support category tree, only tags, but already used to it. There were no minor edits, however, nothing fundamental, rather, improved usability. The main disadvantage of the lack of categories - passwords on the tag can fall out a lot - 12-15 pages of 20 passwords each. Therefore, the next wave of optimization is now scheduled - regulation of the list of tags, a strict limit on their number, tag navigation bar at the top. Otherwise, Rattic has everything in order - password change history, viewing and editing logs, sharing in a couple of clicks ( there was no built-in, did this: I got into the AD group for each user who needs to share passwords, and, if necessary, I give this group has the right to view the password ), view the list of passwords that any of the users saw. Moreover, if you delete a user, the system will kindly show the list of passwords that he saw and prompt you to immediately add them to the change queue.

As a result, the search time for passwords was reduced from 20 to 1-2 minutes, which was a big step forward.

Telephony

When the core of the toolkit was formed, I wanted more. And the first step was its own telephony system. At one time, telephony services were purchased from a cloud provider, but then decided to move to their own solution, which was more economical and more universal. And did not lose. I decided not to install the standard Asterisk, but try its competitor - Freeswitch. And it was the second successful decision. Not everything worked out right away, some things, let's say, redirection to mobiles had to be finished for almost another week, and all because Freeswitch has its own special views on multichannel. But when it was all over, they simply forgot about him. Because he did not fall, just worked for months and worked. After a recent update, for the first time in 200 days I rebooted it myself.

Gradually, he overcame all sorts of additional services, communication with CMDB, blacklist, receiving faxes with mail forwarding, banning objectionable with calls to Somalia. Its only drawback was the configuration through configs, the system just can not figure it out. Therefore, the decision was made to install FusionPBX, which now seems to me to be the most ready for production. As a result, technical support received real queues with notification of what position we are on (although, during the whole period of using the telephone station, there was never a situation when there was no one to answer the call), managers - normal listening to conversations, experts - additional monitoring and control of the station . For example, now you can see the log of each call in the CDR, to understand what went wrong. In the future, there is a desire to turn Freeswitch into a secure communication server, keep text conferences, voice and video there, reject all additional communication tools, and cross it with the OpenXCAP server. Waiting for the release of the 18th version.

CMDB

The youngest, but very promising tool in our holder is CMDB. It was chosen according to the following criteria: web, light, extensible, functional, convenient. In the end, opted for Itop. Now he is in the process of implementation, there is only a phone book that interacts with telephony, but he has good prospects. It will integrate the documentation repository, the service desk, the self-service portal, and, in fact, customer infrastructure data, contracts, and so on. Plus, it has support for ITIL and ITSM, which gives us the groundwork for development. In recent versions, the self-service portal has drastically improved, and I hope that in the next one the same fate will befall the main interface. Now it is still used as a phone book with the support of fuzzy search and automated data collection from the FusionInventory Agent is being actively implemented.

Remote control

Technical support of IT-infrastructure and users is unthinkable without remote control. From the word at all. It is not always possible to explain to the user what you want; not always and they can tell what is happening. Because you need to connect yourself and watch. Remote support tools are now available for every taste and budget. For a long time, I used Teamviewer, which fully met my requirements and was also easy to manage. But for the convenience you have to pay. Teamviewer traffic goes through third-party servers, password storage and access sharing algorithms are closed, articles on its hacking have appeared on the Internet - no one knows what is going on under the hood. Therefore, I decided to change the remote support tool and now I am gradually migrating to UltraVNC. Yes, I am well aware of its shortcomings and tried to compensate them as much as possible without disturbing the users. An additional encryption plugin is bolted to UltraVNC, and quite an adequate AES256 / RSA2048, an UltraVNC Repeater installed and configured so that customers do not have problems connecting to the remote control system. A system for generating numbers based on the MAC addresses of network cards has been developed; a driver is installed in the system for smoother operation. Future plans include the own UltraVNC one Click build, there you have to think about branding and setting it up for a long time. In general, the tool is quite working, one of the clients has been successfully working for several months without any problems. Of course, it didn’t go without a fly in the ointment: if you disable the encryption plugin, UltraVNC does not warn about this, but simply raises an unprotected session. Also, his service sometimes suffers from unmotivated crashes, which is treated by setting up the service for an unconditional restart. Also, the use of UltraVNC Repeater does not protect against duplicate numbers, that is, UltraVNC with a backup number will start as if nothing has happened, but cannot register at the repeater. Now this problem is solved by the rules of creating numbers and using an expert if necessary. I already wrote about this system in more detail here .

Terminal

All the tools I use work in my remote terminal. This has several goals at once. The most important thing is the security of the session, all protected data is stored in one place, on the local computers of employees there is nothing compromising me or the client. I get a fairly simple scheme for managing and controlling the perimeter of security, access to remote desktops and the password system is only from a remote terminal. Next to the terminal is the Active Directory, which is the main access control panel, all of our systems are connected to it. Since all business-critical data is concentrated in one place, backing up our systems is also not a difficult task. An additional advantage of the terminal is that your work environment is available to you immediately and does not depend on the place where you are, go to the remote desktop and all of your programs and documents are open.

Profitability Analysis System

When customers and applications from them becomes quite a lot the natural question arises: am I not working in vain, inefficiently or at a loss? Which client takes the maximum time and brings a minimum income? What level of tasks do each client have and what is the time they take away from tech support? And, most importantly, how much does it all cost? The answer to all these pressing questions was the customer profitability system, which was created on the basis of our service desk. But this is running a little ahead. The first version of such a system was generally a separate base, which was recorded retired time, type of work and the client. It was abandoned because of its bulkiness and inefficiency - its filling took time and the effect was minimal. After the implementation of the service desk, I wrote the processing that once a week made selections from the service desk base and based on them created views for management. The inconvenience of this option was that it was necessary to tear off the employee to prepare these reports. As a result, a small module was written to the service desk, which was based on the queries created and the manager can now independently prepare a report for any period of interest. This report presents the hours spent on each of the customers, correlated with the corresponding hour value of each of the employees. This allows us to see the cost-effectiveness of work on each of the clients in almost any real time, at any interesting time interval, which made it possible to adjust our efforts and costs.

Total

Each tool used was in its place and closely interacts with everyone else, and together they help to solve the issue of timely and quality assistance to our customers.

PS Friends, ask questions. If any systems and topics are interesting, I will be glad to your comments. And tell me what I should write about next time.