Geekly Articles each Day
📜 ⬆️ ⬇️

Interesting cases of work organization on the second line of technical support



The story is this: being one of the largest IT companies on the Russian IT outsourcing market and services, we very often find ourselves in a situation of decision support at the junction of two or three manufacturers. And when a problem arises, for example, on which it depends whether people will be paid retirement on time or not, the following story generally begins:
“Well, you, our hardware works fine, the problem is in software,” says the first vendor and sends it to the second one.
- How can you! Our software is tested, the problem is in the hardware, - says the second and sends it to the first.

If at this stage you begin to acquaint them, gather meetings and explain that you need to work, and not to translate arrows, then the end users can crowd out into the streets.
')
Therefore, we work instead of them. Once - even included in the oxygen-insulating devices after fire alarm triggering.

SLA


We must immediately say that large manufacturers of hardware and software that have been working on the Russian market for a long time are in a rather ambivalent relationship with us. If you look at marketing, we are trying to "squeeze" each other out of the object: they sell their service contracts, and we own. Sometimes, when ours ends, they begin - and sometimes, and vice versa, when vendor restrictions bother the customer. Or it turns out that one well-known manufacturer's support is natural scam.

In general, our contract is understandably somewhat more expensive, but it also does not contain the limitations inherent in a pure iron vendor.

On the other hand, if we are talking about technical teams, then we often work side by side while marketing is having fun there. The fact is that any exploitable bug is both a problem for the enterprise and a reputation loss for the vendor. Well, just a joint in life, which is not very desirable. Therefore, between our second lines, as a rule, there is a deep mutual understanding, there were even cases when we helped each other with high loads - today you will change the company's engineer at the facility, and tomorrow they will cover your back with a difficult data center move, for example.

So, the customer usually buys the equipment directly or through someone (sometimes through us) and can immediately purchase not only a warranty, but also extended support from the vendor with departures, telephone consultations and other amenities. The problem is that, naturally, the vendor doesn’t want to take on the customer’s risks, and therefore not exactly what I’d like to see, for example, an insurance company, a large retail company, a responsible NGO, or, in particular, a financial organization, is prescribed in the SLA.

Further the question is solved too simply. If a business has a very strong IT team (and, as a rule, the same banks have one), then it does support within the framework of the required SLA. If there is no such team, or our work is cheaper (and this often happens taking into account the scale of geography or the fact that “our own” team sits in the Netherlands), we write the “right” SLA together with the customer and take some of the risks.

For the implementation of complex SLA (at 4 o'clock) we often set the duty directly at the customer's site. This is especially true for those cases when a large load falls on the kernel systems during the seasonal peak: for example, the “X-team” is often on duty at the banking data centers during the New Year credits period.

On objects, as a rule, on the night from the 31st to the first, for example, the duty officer still meets at home with his family, but does not drink. He has a special emergency laptop with internet. We do not use VPN links with our workplaces on information security policy, therefore, for such situations, a cytriksov terminal is used, where the necessary applications are published.

We don’t support end users specifically (second line) at all; we work with the customer’s IT department or with our customer support service. We are able to be friends of support teams through our ticket system, and logically these two instances can be in the same service, but do not overlap within the organization. The IT department does not rake the shaft of unqualified calls. If there is a qualified question - through an escalation procedure, a qualified answer arises. This greatly influences the pricing, because, for example, I remember in 2005 one bank ordered more than 35 specialists with certificates not lower than MCSE for the first line - I was just one of them - they paid almost 10 thousand dollars for each engineer contract. Now this, well, uh, is a bit expensive.

Now, by the way, due to the economy in crisis, the tradition of two- and three-year contracts is returning. Previously, they were almost always annual, long contracts were not considered separately. And now everyone understands that it is cheaper for two or three years at once.

Project examples:

Where do people come from for distant objects


Naturally, if the customer has a vast geography, we often need knowledgeable people far, far to the north or the Far East. Again, naturally, our SLAs at 4 o'clock concern cities, and not mining facilities at the fields (about the peculiarity of trips to the bears, my colleagues from the group of “cosmonauts” told and told). So, fly, for example, to Vladivostok for a lot more than 4 hours prescribed in the contract. Therefore, you need to have a person in place.

An example of such a project - we had a service contract for a network of cinemas. There it was necessary to keep people “within walking distance”, but at the same time it would be inexpedient to take even 1/10 of the rate - everything is set up so that there may not be a single incident in a year.

Therefore, we are looking for local administrators and engineers through special employment services and freelancing. I will not say exactly how it ultimately looks from the point of view of HR, but in the end I am offered about 20 candidates for the city, each of whom has the necessary skills. We call each other up, check our skills - there are only 10 people left. Five of them, by convention, are suitable, and with three we finally agree that if it is necessary, in 2 hours you have to drive up and decide. Pay by incident. Draws up and solves all the documentary issues of the same contractor who sought them.

This is done for each project - pre-selection among potential outreach is very useful.

Then, if something happens, we already have the necessary contacts and agreements - and a person leaves the place with a head and hands. In case he “falls off” (for example, he falls ill), we simply call the second and third. At this time, the duty "X-Comady" works with them remotely. If something complex is diagnosed, in parallel another engineer flies into place.

Remote support peaks


It is also very important to work out cases when a sharp expansion of remote first-line support is needed. This is also our internal service, which is able to scale quickly. For example, for the first line of remotes for retail usually need 30 people. So all year except December, when the peak begins. In December, you need 70 people, for example.

The reality of the market is such that tariffing for the customer goes backwardly, that is, it simply will not keep these 40 extra people in reserve. Therefore, it is our risk and our question what to do with them.

On the one hand, of course, not all of our customers have seasonal peaks at the same time, so the effect is a bit distributed, but still, roughly speaking, you need to grow no longer from 30 to 70, but from 40 to 70 people per month.

Therefore, we have seasonal front line workers. 80 percent of these people work with us not the first year, and work only in a high season. They arrive a month before the start of the hot time and hardly receive additional training — they already know what and how. We, most often, rent a place for such teams next to the customer’s office and land at the time of the peak.

Where do they go after? It depends. I know several people who leave for Goa and surf there for half a year, and then come back to our support, I know a cool IT specialist who half a year has been sweeping the floor in Israel (and then there are 3 more months of rest), I know people who they take a vacation at another job and come to us to support, I know many freelancers who also come to our temporary offices for the season. Naturally, there is a small rotation.

Cases of support


In addition to the absolutely ordinary ordinary tickets for replacing fans and other pleasures of life that require departure, a lot of interesting things come to the second line. But interesting at first - then year after year the situation recurs. For example, we are often asked to get some tricky adapters for different crap - most often it turns out that retail speakers steal large speakers, they buy new ones and cannot assemble them correctly. It happens, they show SCS and they say, they say, can you do something wrong? We, of course, appreciate and for the additional budget we do, if the customer is satisfied. Often, on top of the contract, they are asked to add another camera to the video surveillance system, to mount the fire, to repair and reconfigure the alarm system - all this is done not by our team, but by engineering support. We are looking for the right people and do it quickly and efficiently. Sometimes we drive ourselves, but more often - in a row. Once from a non-standard we were even asked to hang a chip for each builder at a construction site. We sewed RFID marks into them in overalls, and then assigned rights and permissible territories to everyone. It is clear that half of the construction materials are not stolen at this level, but it was precisely the theft from the workers that fell three times - the tracks were written and it was clear who was not where the plan should be.

We are not limited by the range of work - even to mount the outlet, even unplanned cleaning from viruses and reinstalls - we can do any non-standard request. In CPPK (suburban trains), we probably have the most complete set possible - telecom, servers, hardware at stations, video surveillance, software, and a dispatch center, and even partial support for application software (although generally rarely give to external support). The story there is such that even the ticket to the second line of their butt is routed through our system according to our method - the customer is extremely comfortable that everything is done through one window. All escalation is through us.

Recently, quite often there have been requests for us to take the “native” support team to our office. At first, we didn’t really understand what was the point until one of the representatives of the western company explained that it was cheaper to sign a contract with us. Three times. The fact is that an IT specialist in the European model costs about 5 salaries per month. This is a place in the office, external obligations (for example, any kind of charity is counted on their heads). We call our prices - not the lowest ones on the market - and people start smiling and saying: “Yes, ok, wrap”. Of course, there are those who still compare the price of the contract with the salary of the administrator on the spot. And there are those who consider all the money and all the costs. They have an approved rate for their IT department's normal hours - and they compare with it.

There is also a reverse process - our teams sometimes buy


One large bank somehow decided in principle that there would be no more outsourcing at all - and we gave them our entire support team as football players. So the bank won and people are good, no need to look for work again. For 8 years, it was already three times. A similar story was in one large bank - but there they asked our two tough Ikskomites for a year. Sometimes, redeem specific employees. Again, the principle is like with football players: if an employee doesn’t mind, then we say - that’s how much the courses cost, we eventually invested 10 thousand dollars in it –– if you are satisfied, take it. Take away. Directly to give ugly - we often have reciprocal agreements, which we do not accept without the agreement of the former employees of each other (the outsourcer and the customer on the support contract) for 3-5 years.

Warehouses


Sometimes, when the delivery of spare parts is clearly not within the SLA, we keep a small spare parts kit on the customer’s side. As a rule, in 90% of cases, this is a trifle such as spare printers, spare pre-configured system units, or cartridges. Once a couple of months we arrive, we change the burned-out power supply units, motherboards, etc. on the replaced equipment, we renew the ZIP. In large points there are also repair kits for engineers and our sets of accessories for data centers. Good practice - each such warehouse, if it turns out, is supplied with an employee on the spot, responsible for it. This is so that a properly trained aunt could issue a new printer in return for something that does not print. At the Far Eastern sites we make emergency envelopes - “do it once, do two, if the light does not light up, call for reinforcements”.

Sometimes it even comes to the point that these special warehouses of iron for the customer are made even for new iron. For example, we supported a small service company with a whole complex of offices distributed in Moscow and the region. Each of them has 2 computers, and they most often sit on the already ready infrastructure of shopping centers. The beauty is that they are superfluid: the point can work for two weeks, and then move to another shopping center. Or they can open a dozen for 2 days, and then the whole dozen will go to a nearby city in late spring. With them, we approved specs for a specific iron for a point, outlined fixed prices for six months, and then simply purchased ahead for several months. They put a ticket like "we open in 3 days, here is the address", we send the iron with a kit for the transport company, and on the spot it is already waiting for an employee from our "contract soldiers". And the point unfolds in a matter of hours.

A few years ago there was a very interesting service contract with one “small proud oil company”. Our top manager was called by their CEO, asked for a “bored computer technician”, but did not specify the details. The task was transferred to the X-team, our engineer-expert arrived. The bald uncle of the Rubik's Cube complexion sits on the spot. On each finger around the ring. Such a flashback in the nineties. He greeted him politely and did two things:
  1. He called a beautiful, well-trained girl and explained - this is an engineer guy, he will now come to us. Help him in everything. Generally all.
  2. He called me and said: “I now know your man. Let no one except him come at all. ” The price of leaving a server expert for a broken printer did not bother him; trust was much more important for him.

The engineer returned from the first exit with square eyes, but in a posh car. He says, I, when I saw the situation and my uncle saw this, I immediately understood that if I didn’t do it, I wouldn’t go out alive.

Vendor-Supported Joints


On-site service units on service contracts work very closely with us inside. Because most often difficult situations are always at the junction of areas: for example, a condo died, IT services need to be removed so that the server does not overheat before being replaced. Or something has ceased to be pinged, and software vendors say, this is your piece of hardware, and they say - and then what’s right for us, you better look at the small and soft. Together they do not work, there are simply no procedures and practice. Therefore - be sure to check related areas. If there is a suspicion of a firewall hardware and system software - we are doing our own and then calling an engineer for a piece of iron. Next time ours will ask us. Together we decide. It is generally worth a lot. It gives people confidence that their colleagues will not start football.

There are contractors who outsource, argue with vendors, we delve into and decide. A huge number of competencies - always a little more opportunities to check adjacent areas.

More from the interesting - the vendor rarely makes recommendations based on the usual support. And we have the whole clinical picture - we see constant sores and can advise that we change the customer so that the symptoms do not occur. That is, we treat immediately at the level of architecture, if the customer wants. Standard report - look, if there are similar incidents, try to find the root causes of these problems.

With vendors, in general, the relationship is - we are doing the current service and transmit unique iron problems or something quite serious, like helping to set up a new homogeneous data center. On the other hand, vendors sometimes do not ordinary things, for example, HP perfectly support SAP.

As for the technical teams' mutual assistance, most often, we know each other from projects a decade ago, and there is no sensation of rivalry (even if we intersect in marketing). For example, when a customer “burned” a project for commissioning in a new building, there was such a situation - the server equipment had already been raised, work all over the country without interruption and pauses, and there were still training alerts in the building, you need to release it. A couple of times we agreed that the duty officer would stay with the KIP and the “suit”, while the rest would stay with the laptops on duty from the dining room of the neighboring building. But once they conducted a direct serious adult doctrine, and everyone had to go out at all. By agreement with the customer, we transferred 20 services to the vendor’s iron support, they helped us at that time. Then we will somehow help them somewhere in the Hot Key, they know it.

We also have a scheme for working out emergency moments within the company. For example, if suddenly the entire X-team is loaded (this happens with simultaneous 2-3 accidents in Moscow, for example, when the substation falls), then the first-line team leaders can cover engineers from the support pool for a couple of hours. For each situation there are contacts of at least three employees with competencies.

Links

Source: https://habr.com/ru/post/279481/

More articles:


All Articles