XIMA Chronicall emergency troubleshooting telephone monitoring and reporting system

Greetings to you, readers and writers Habrahabr!

I am currently learning, in parallel working as a system administrator, in an interesting startup - MyMoneyPark. Naturally, at first, this work includes everything that is possible, including “enikeystvo”, but this is an experience that hardly hurts a student.

One area that I “rule” is telephony. For us, the stability of the telephone system is very critical, especially since the project was launched a little more than three months ago. In addition, in order to have an idea of ​​the number of calls and the necessary “capacities” to process them, in order not to lose a single caller or to make schedules for each employee, the monitoring and reporting system from XIMA was very useful to us. And about my troubleshooting, which I had to spend working with this system, I will tell you under the cut.

About the system

Chronicall is a XIMA development that allows you to store call history, build reports, view detailed information about each perfect call. Contains a number of additions that we, as a consumer, are not currently interested in. Like any software, Chronicall has a number of errors and flaws that are imperceptible at first glance, but, nevertheless, sometimes have a detrimental effect on the construction of such important reports for us.
')
It seemed that the simplest situation, the head of the customer service line (LOC), received "basic" access to the utility, working through Java in the form of an applet. The first thing he will do is familiarize himself with the product, and, quite successfully, after the first restart, the system will refuse to start.

How it all began

In the morning after the necessary updates that needed a reboot, I received a letter from the head of LOC that the Chronicall system refuses to start, namely, its part with a list of available reports, complaining about the lack of connection with the server. The message is straightforward, it looked simple - "It appears that you have lost connection to the Chronicall Server. Please make sure you are connected to the network. ”

The first steps

Having not found the answers in Google and not having received answers from the XIMA-FAQ to the questions posed, I began my own deep troubleshooting. It was quite easy to assume that all Java applet requests are served by an HTTP server. By the way, in Chronicall the HTTP server is tomcat. Therefore, a direct request to the URL with a list of available reports showed a banal server error - 503. Perhaps someone would have acted differently, immediately starting to examine the server logs, but seeing the connection error, you can not help thinking about problems in the network. Therefore, it was the thought of checking the availability of the server to the client that occurred to me first.

Continuing the search, the second stage I turned to the HTTP server logs, where there was an error. For me, the fact that at first glance at the error that the Java applet gives out is difficult to guess what really causes it is a bit strange.

The error itself did not bother me, having experience in debugging Java applications, I could try to find by hand what interfered with the launch of the miracle utility. But, unfortunately, without listening to reason, I started with restarting the service, reinstalling the JVM, clearing the cache, and searching for “pitfalls” as incompatible with different versions of the Java machine, in general everything that the XIMA FAQ offered, the manufacturer’s website and thematic forums. So nothing helped. The last thing I naturally sent the error report to the developers with a detailed description of the problem. I believed the bug-report slogan about the speed of their work, I thought they would answer very quickly, but I missed it, the answer came in a few weeks, and I’ll tell you later.

Reasonable thought

My "light" study showed that only the report editor, which contains a selection tree consisting of available reports, disbanded by type in folders, remains inactive. With all this, the data in the database was written, the reports on the schedule worked great. Everything worked except for the editor (those reports). Fortunately, at this stage of the study, I remembered what had “done” our head of customer service line changed:

Then it was possible to get closer to solving this problem. The figure above shows the report tree, with the new folder “New Folder”, at the very bottom. This folder was created by the head of LOC, exploring the platform. And, as you can see, you can create it, but to remove it, the functionality that we have does not allow. Also this functionality does not allow using this folder for its intended purpose - to group reports.

Now it becomes clear that the function “ensureUniqueFolderIDs” from logs, apparently, and is responsible in part for loading the list from the figure above. It remains to find where the data for the selection tree is loaded from and to try to eliminate the error.

Elimination

The database schema used has no hint of a list of possible reports. Therefore, walking through the filesystem folders, I came across one called “reports”. There are a lot of files without extensions, several templates (as the name implies) and a number of JASPER files, apparently for open source JasperReports. But the file with the name “model” attracted me the most. In it, among the binary "junk", I found the names of my folders and the names included in the report folders.

Deleting this file helped confirm my guess - it is from it that the program loads the tree. It fixed the problem - the new file was automatically created after the restart, but unfortunately, it turned out to be empty, like the report tree. The fact is that the “model” file contains, apparently, the serialized tree data, which did not allow me to “scrub” the “New Folder” out of it with a flick of the wrist. It would take a long time to sort through the byte, the working day was nearing its end, and it would not be reasonable to leave everything to the next. The only thing that could speed up the process was the installer file. True, I was somewhat frightened by a complete reinstallation. Therefore, having parsed it into files, first of all I replaced the “model” file. This procedure not only returned everything to its place, but also added new reports that were useless in our case.

findings

On the one hand, the desire of developers to limit the client, who has not bought an “advanced” license, is understandable. And on the other hand, the simplest trifle, which without editing the files and long searches for at least some clues, cannot be eliminated, but which leads to the complete incapacity of the system, and moreover is misleading by an “implausible” error message. A trifle that can stop working for a few hours, if you understand it yourself and you will be lucky or for a few weeks if you use a bug report.

A huge amount of humor of this story is added by the developers who deigned to respond to the bug-report, as already noted, in a few weeks. Admiring the work done, they offered our company to take the place of their beta tester for simple little buns. But for lack of time to use this opportunity, we did not work out and they did not write to us anymore.

Why the appearance of this folder so discouraged the program for me remains a mystery so far, the developers have not bothered to give a clear answer to me, but I hope the problem has been solved. Therefore, I very much hope that no one else will have to deal with such a phenomenon.

PS

If I initially went the wrong way, I will accept any criticism. If there are more simple ways to solve this problem, I would really like to know about it.
Do not judge strongly strictly for the first article you have.