Geekly Articles each Day
📜 ⬆️ ⬇️

Let's start the year with a positive: how admins and CIOs sit down



In fact, of course, just sitting down is quite a serious task, even with complete break-up. But losing a job in a day and never returning to the IT sphere is more than hundreds of such cases.

I will tell a few tales. Some of them changed the details so that you could not find out the company. If you recognize your own - do not worry, for sure these cases are almost typical for the whole country. You know the joke about the first flown woodpecker.

Three CIOs for the year


In the same retail network, three CIOs changed at once. The latter changed with me. A call from the region in the morning, they say: it is urgent to open a new point. They have an internal SLA - a week. The man explains: so and so, everything is ready, if you wait a week - this is a week of rent. It is extremely expensive, asks for help, than it is possible to get faster. CIO refuses him. Then the same man calls the founder. The next day, the CIO is no longer working. The reason - did not understand the needs of the business, went to the process formally.
')
We were here on an outsourcing contract. The new IT director turned to us and asked the guys for work, we did not refuse - such force majeure laid out in our service contracts as an accident. I must say that we have repeatedly encountered this with different customers, and therefore we immediately lay down a small reserve for urgent work. Everyone is sure that they will not, but they always happen.

On the other hand, there are also reverse examples. In one bank, we have maintained an integration bus for several years. The CIO did something wrong there, and changed him, his deputy - and terminated the contract with the support counterparty, that is, with us. Our team was kicked out of the facility the very next day, 9 people. They gave the infrastructure to two administrators who knew how to roll patches. They began to learn quickly, but still it didn’t really help, as we maintained 24x7, and these two still sometimes slept. As a result, they were persistently forced to live without sleep and work for five, until they realized that everything was in vain, and did not recruit a new team.

About charity and its consequences


There is another bank for the New Year 2007–2008 reduced the position of CIO. It so happened, as a result of IT, it became involved in the purchase of junk, and no one was engaged in support. And, predictably, at some point EVERYTHING FALLED. It was on holidays. We as a result, on friendly terms for the new CIO, raised the ABS, in fact - in the framework of charity, without a contract.

In one large regional educational organization they purchased a huge fleet of communication equipment and a large fleet for a server node. About ten million rubles. After that, they disbanded the entire IT staff. On Saturday evening, a man called my colleague and said: “Hello, I am Nikolai, an engineer here, and I don’t know what to do, because on Monday there will be a large-scale check. Can you at least explain what the equipment is, how it works and where did we spend the money? ”As a result, until Sunday lunch, the guys sat on the phone, interrupting only on timeouts of cellular operators and going to the toilet. Nikolay gave access, and with a colleague they remotely deployed a videoconferencing network. On Monday, they successfully held the first conference. As far as I know, the engineer then pounded for another month with questions, until we politely made it clear that this service was paid.

Once there was competition in solving the problem. Call from the customer: "Everything is bad, the network does not work." We got the command to "save." Moved our guy from the X-team and sales manager. When we arrived, it turned out that there were 5 teams from other integrators. About ten years ago we would be the only ones, and here it smelled of money, and therefore the hive was buzzing. The problem was not solved for a long time. By evening, there are already three teams left. Ours kept, at 5 in the morning, only we and the vendor were in place - they solved the problem together. A week later, billed, without discussing the amount the customer paid. A rare case, because when the problem is solved, the service rendered is worthless.

There was also a customer from a bank who counted the cost of support in our IT services calculator. At the output, the result is given in the first approximation, without knowing the nuances of the customer's infrastructure, the average case in the country is laid out - when there are architectural difficulties in the network. This is followed by a detailed conversation, survey and other curtsies - of course, the final cost can vary greatly, and in both directions. Nevertheless, their CIO marinated us for a long time: they say, why it is so expensive, why you think that we have a collapsed network. As a result, we didn’t go further than the preliminary audit, we refused the contract. There was a problem in infrastructure services - a jamb at the level of architecture, a time bomb. But they believed that crutches are an excellent building material. How am I going to pay three million a year for support? To make it a little clearer: a few years later, to describe the situation, the phrase was said: “Banking standards for backup are only on paper, and helpdesk is two bottles of brandy to the auditor”.

According to my data (we later met with participants in the story a few years later), they paid 14 million to solve the problem. At that moment, the bank was standing, a hard crisis management was involved, and the bill for the decision was generated by a colleague from another integrator. They picked it up somehow on the same sea of ​​crutches - and after that they nevertheless ordered support and alteration of the network.

Often, large companies are willing to pay more to reduce risks. For example, there was a tender of an American company in the Russian Federation, which had a bunch of small suppliers for two dozen services. They managed to administer all separately. Announced a contest for everything. And it turned out one big contract for support from infrastructure to support application SAP. Now there is a general trend - many contests are aimed at getting rid of small suppliers, consolidation of service contracts. Niche solutions on one server for a dozen people die off - they are trying to include all such things in a big contract. The tendency to consolidation came from Western business, they now practice it almost everywhere.

Often we are hired, in which case there was someone to shift the responsibility. In general, we take the money for this, because we take responsibility.

"Rule of Ten"


In one distant snowy city, we set balancers to optimize traffic. The procedure is simple: new glands are placed in the gap between the channel and the data center, the old ones are removed. The situation was slightly complicated by the fact that the guys, taking this opportunity, decided at the same time to change the interfaces from 1G to 10G, since in the new year the replacement of core network switches was planned. All this took place at the end of December, the deadlines fell, it was critical to close before the end of the year. Quickly agreed, talked with the engineers, a week, two weekends - Saturday and Sunday - night work should go, then observation. A colleague flew in and heard the epoch-making phrase: “On this vodka. We thought here, let's do it differently. ” There they had to replace another piece of metal, and they invented a new scheme, drew it on a flipchart. For approval - a day. But, what we really liked, was a very detailed rollback plan, which, according to its elaboration, was registered at the hardcore level, that is, to all the little things and cases like the landing of aliens.

A colleague clarifies why. They say: "We must be very, very careful." It turns out, they say, “there is a moment: the limit on the number of critical incidents per year is not more than 10. The ninth was in November. Any break in communication with the center is a criticism. We work very carefully and gently. At stake is a prize. If anything, everyone will be left without gifts. ” From Friday to Saturday at the backup data center. We looked, transferred the traffic to the backup site, from Saturday to Sunday did mostly - transferred back. Sunday monitored on Monday watched in full force. It all ended well. Since then, we have loved banks, where there are similar rules. I must say that they called us, in particular, to hedge themselves (in general, they could have mounted them themselves). But smooth operation — I mean premiums — is more expensive.

What are they sitting for?


As it happens in reality, I think, it is already clear. Now a couple of examples of precisely “hard landings”.

In December 2013, a crash occurred in Citibank in North America that lasted several hours. The failure was caused by the system administrator. Then the employee wanted to take revenge on the leadership, resetting the configuration settings of 9 out of 10 routers belonging to the bank. Sysadmin Brown pleaded guilty in court. As a result, he was sentenced to 21 months in prison and a fine of 77,200 dollars.

The NASA Inspector General’s Service conducted an audit at the IT department of the US Space Agency, during which it turned out that the former CIO was not able to competently organize the management of the organization’s IT infrastructure. Inspectors found that as a result of ineffective control over IT spending, the federal budget annually lost more than a billion dollars.

We are more often fined. The directors of the IT department of a large department fined the court 20,000 for failing to issue regulatory documents within the period prescribed by the Accounts Chamber ( recognized as an administrative offense ). System administrators are often tried for pirated "leftist": for 3 years now for installing unlicensed programs (March 2005), for illegal installation of Autodesk Ink and Microsoft Windows XP (June 2009).

Former system administrator of the Black Sea Fleet CJSC ABS Russia was convicted of blocking the work network (October 2016). As it was established by the UFSB investigators, in the summer of 2015, he made unlawful access to the company's telecom network, as a result of which information was blocked both by the company and a number of others, including those related to the military-industrial complex. As a result, the entire network of enterprises was paralyzed, which resulted in the inoperability of the main programs used by its employees in the performance of their work functions.

Of the people I knew no one sat down, but they lost their jobs at the moment. The best example was the case when the CIO, the main administrator and the deputy of the CIO in a state-owned company clicked on an urgent letter from the minister. This was their last working day both in this state company and in the state apparatus in general.

In the electric power industry, we were taken to a "safety net" contract - every day we have to send one XLS to the electricity exchange. I clicked such a report - they remove the company from the stock exchange. You can warm the air, the yard of losses immediately, simply because "not filed." Also castrated by the method of separation, but does not reach the courts. We were told about previous losers when they signed a contract.

In retail, the fall of fiscal registrars, processing or logistics at the peak of sales is kapets. Half of the IT specialists will immediately go looking for a job - they will not even understand who is at the end of the peak who and what. There were examples. There was also a pair of reinsurance contracts: saving the ass, if everything crashed - X-team, to leave.

Once we were looking for the admin server. He simply did not know where she was, although he had been working for more than a year.

The production is even tougher, there is a process control system. At hazardous production, if there is a malfunction of the sensors, it can, for example, jam the turbine. That is why IT specialists extremely do not like to take on the support of production equipment - there a bug can mean the death of a worker. Although, of course, more often this is a profaned procedure - for example, I know the case of injuries on an escalator, when the tape broke in the subway (they missed the service schedule, the emergency brake did not work).

Even for personal data, good fines have now gone, so too many support contracts go.

Perhaps, when the next customer asks me why our calculator gives out such strange numbers, I will show him this post. And I will explain that if he is ready to take responsibility for himself - no question, it can be cheaper. But usually the responsibility is still on us. Or we insure. This understanding, as a rule, removes all questions.

Links


Source: https://habr.com/ru/post/319908/

More articles:


All Articles