Project to optimize the distribution of incoming Voip calls

I work as a Voip system administrator for one small French company, how I got here is a separate story.
I will show the results of the team’s work on the project whose goal was a global equivalent call distribution strategy for call centers, depending on the number of agents capable of receiving calls. The phrase certainly succeeded,


')
In the example, it looks like this:

Center1 = 20 agents
Cent2 = 10 agents

With a load of 50%, there should be 10 free agents in Center1, and 5 agents in Center2. Here is such parity.
When the number of calls exceeds the number of free agents, customers must stand in the same global queue, one after the other. If a certain number of agents become free, then customers move in order of priority to the agents.

Everything should be wrapped in a web interface, with fashionable gadgets.

Old system:

Our client, 6 call centers, all of them satisfy the same service with a single incoming number. The number of calls depends on the day of the week, but basically it is 15000-20000 per day. The centers themselves worked according to the following scheme:



Before me, the servers on tricboxes (asterisk) worked without servers, the system was archaic, there were problems with sound quality. I put two servers with open accounts, Opensips is like a proxy, only for calls, they can be sent according to the rules set anywhere. But this system also began to show its limit. The servers at opencipse distributed calls with the Dispatcher module, the principle is simple, the server had a text file with the following data:

1 192.168.1.1:5060 #Cent1
1 192.168.1.2:5060 #Cent2
1 192.168.1.3:5060 #Cent3
1 192.168.1.4:5060 # Center4
1 192.168.1.5:5060 #Cent5
1 192.168.1.6:5060 #Cent6

The call comes, the open-end reads the first line> the call to the center 1> We redirect
The call comes, the open-ended reads the second line> call to the center 2> Redirect

When we come to the end of the file, we start reading in a circle. Now imagine that we have 100 lines, what's this? 100% correct
Every morning I received a percentage for each center, rounded and generated a file with a small ruby ​​script. This allowed to raise the efficiency by 5% A little? 5% is 9 people, with an average salary of 1,700, multiply = € 15,300 savings per month. The number of agents is different, from 180 to 1, that is, varies throughout the day. After some time, I began to notice some anomalies , then one center declares fewer resources than it actually is, then another ... Here it is a human factor. Yes, and general opacity, locally on Trixboxes there was complete reporting, but globally, how many people waiting in line at 10:30 am I am not able to say. It turns out that without a global visibility it is just hard to predict the load, and the number of agents may be wrong. My 5% went to zero because of the “Factor X” and global statistics.

Idea! And how would it be, so what would it be like that ...

In between, an open version 1.6 came out with an interesting Load Balancer module. The principle is interesting, open-storage keeps in mind the number of specified resources, and sends calls where they are free. It remains to get the number of agents from Trixbox every minute and modify the database. Eureka ?!

Not quite

1: He does this not equivalent to the number of agents, that is, he clogs the first resource to 0, then the second. (in our case the first center, the second)
2: Opensips cannot play audio, and in general there are no queues, this is a proxy.
3: Alpha version.

Decision,

1: I wrote to the main developer of the open-source (Bogdan) he made the necessary algorithm for us. You can see here
2: Use asterisk for audio and queue.
3: We baptize, test, test.

New system:

The asterisk takes a call, and tries to call through an open-ended subscription, if there are resources, it passes, there are no resources, we are waiting. All customers are in one place which reduces the number of unanswered calls in comparison with the old system. Updating the number of resources is every X seconds. What are these resources? This is the number of logged in agents in Triksboh.



We set up 4 servers, everything needs to be duplicated, network power, network adapters in different switches, two servers on separate hangars with different power sources, between them double optics "ring". Physical characteristics are:

Dell r410 double quad core xenon, download statistics:

Opensips

top - 19:05:59 up 35 days, 4:16, 2 users, load average: 1.10, 1.24, 1.21
Tasks: 192 total, 1 running, 191 sleeping, 0 stopped, 0 zombie
Cpu (s): 9.2% us, 1.3% sy, 0.0% ni, 89.4% id, 0.2% wa, 0.0% hi, 0.0% si, 0.0% st
Mem: 4043228k total, 3905396k used, 137832k free, 256656k buffers
Swap: 6094840k total, 108k used, 6094732k free, 3101996k cached

Asterisk:

top - 19:10:31 up 37 days, 23:40, 3 users, load average: 0.04, 0.05, 0.12
Tasks: 175 total, 1 running, 174 sleeping, 0 stopped, 0 zombie
Cpu (s): 0.0% us, 16.7% sy, 0.0% ni, 83.3% id, 0.0% wa, 0.0% hi, 0.0% si, 0.0% st
Mem: 4037580k total, 2375616k used, 1661964k free, 156400k buffers
Swap: 6094840k total, 168k used, 6094672k free, 1876072k cached

On an asterisk removed late, usually they on average 0.5 Load. We are thinking of adding memory and processors at the open-source system, the system is eating, optimization has not helped.

Launch

The nightmare, the system kicked, stalled in the middle of the day, the client spat, we worked at night. Bugs came out, the problem is that they were difficult to detect without real server load, synthetic tests did not help. It took us 5 days to fix bugs. The schedule was interesting, you work at night, in the daytime you look so that it would not fall. I SAW these 180 people who did not work for me, their curses went straight to my brain. During these five days I understood what it means to be in a burning pan. But on the fifth day it worked, after there were only minor modifications. Customer satisfied.

Interface

The web interface is simple and functional, there are no extra ryushechek, and here it is. The truth had to rub the names, because of legal issues.

Login:



Global statistics in digital form. All in French, I will try to explain:

Nombre d'appels totaux pour la journée : The number of calls received.
Temps de réponse estimé : Average waiting time.
Appels en attente : The number of people in the queue (they have not yet been answered).
Appels reçus : The number of calls received.
Agents [Total: Libre: En com] : Agents [All: Free: Speak]
Appels abandonnées : Number of clients who hung up before contacting an agent.
Efficacité Globale : Overall effectiveness,% of the answers from the total accepted.
Charge Globale : Global load, well, if 10 out of 10 are talking, and 0 people are waiting, then the load is 100% and so on ...



The same, but statistics on one center:



Charts , pay attention to the second schedule, now the number of agents is faster and more efficiently adapted to workload, when there is nothing to do they do other things instead of sticking out at the phone that rings once every half hour.
But it is clear that they can still put pressure on lunch.



The rest of the screenshots I will insert a smaller one, and even so the topic is long. Who cares he will look.







Race results

7-10% increase in productivity. This is only for dialing centers, a global time saving for other tasks. The disappearance of routine tasks, such as the compilation of resources by the management centers of the dialer.

Our website: www.ipbx-france.fr

Special thanks to the Master of Code team for their work on the web part of the muzzle and internal scripts.
Thank you all for your attention.