Survival Instructions for a full-time system administrator

One in the field - CIO!

From time to time I am invited to small companies in order to “sort out” with their sole system administrator. The list of complaints about the work of their staff specialist is always about the same: it performs tasks slowly, often “nothing works”, and even there was such a case ...
At the same time, the level of claims to the work of a system administrator often does not correlate with his level of technical competence, but mainly due to the lack of minimum managerial competencies for system administrators, which are so necessary to work in a company in which you are the only person in IT.

Having seen enough of the completely undeserved suffering of full-time system administrators, I decided to write this small instruction, which I hope will help them avoid conflicts with management and remove such notions as “stress tolerance” from system administrators' vacancies.

So let's start from the very first:
')

1. When you get a job, learn about the targets in your work.

Usually, the employer, hiring a system administrator for work, expects that everything will work “well” with him now. But what he puts into this concept is often covered in darkness and in most cases it ends with the fact that a system administrator from an IT specialist turns into a whipping boy in each case when he doesn’t feel as good as he wanted.
While still applying for a job, negotiate with your employer what expectations he has of your work: starting from things close to your body (your work schedule, availability on weekends and evenings, how you will have your vacation), ending with business issues ( which IT services are critical, what is their maximum idle time, what is the maximum data loss allowed, etc.). All these parameters should be measurable (work from 9 to 18, but by phone up to 22 is available, the Internet should, in which case, be repaired in 2 hours, etc.). Having received this information, do not be lazy to fix it in writing - this will greatly help you in the future, but in the meantime, starting to work:

2. Backup and align backup scheme with manual

Set up automatic backup creation and regularly check the correctness of their creation. Reconcile the backup scheme with the manual once again and do it in writing - in the future it will help you to answer the questions “why our base was copied only once a day, not every hour”. In case the management wants a greater depth of storage or frequency of creation of backup copies - designate necessary investments into the equipment. And yes, do not store backups on the same disk arrays as the master data, for:

3. Assess the risks and do not rely on the reliability of hardware, software and communication channels

It is this expensive brand (besides, the only) server will fail at the most inappropriate moment and deprive you of sleep for a couple of days, forcing you frantically to think of what can be done in this situation. Conduct a small analysis of operational risks: what will you (the organization) do if the server / computer / printer fails, the Internet channel fails, any software error or fire in the server room occurs? How much time will it take to recover? Is this time in line with business expectations (see clause 1)? What can be done to reduce downtime and negative consequences? Then again, document your findings and submit them to the management:

4. Align operational risks and disaster recovery plans with management.

Carry your conclusions from the previous paragraph to the management, so that they take operational risks for granted or allocate money to eliminate them or minimize negative consequences. Coordinate not only operational risks, but also disaster recovery plans - in the event of an accident, this will greatly simplify your life, because there will be no “unexpected situations” for you and you will not have to wait until “Petr Petrovich comes in contact” to buy a replacement power supply unit in the failed server. After agreeing on operational risks, you are likely to be allocated money for a small upgrade. And here the most important thing:

5. Plan any changes in IT infrastructure in advance, but, most importantly, create both a change plan and a change rollback plan in case something goes wrong.

The phrase "yes there is 5 minutes" when administering servers, in my opinion, is akin to the phrase "guys, look how I can" while driving a car - once started with such a phrase, a big fail at the end waits. No matter how large or small upgrades you plan to carry out, they need to be planned, even if you and only you will carry them out. When planning, it is necessary to take into account which services will be affected by your changes, what steps need to be done in the process of changes, how to check that the upgrade was successful and, most importantly, what you will do if something goes wrong. Well, after you have understood everything for yourself, again go to the management and:

6. Coordinate the time of any work related to downtime or possible service downtime in the IT infrastructure with management and always justify the need for such work.

Worse than the admin who intentionally harms, there are only admins who are harmed for the best reasons. You decided to blow the server from the dust on the last day of reporting, you decided to update the system on the servers in the middle of the project session, and you are doing the migration of mail to the new mail system exactly when the whole company is waiting for exactly the most valuable and important letter . The list can be continued, but the meaning is the same - always convey to the leadership your plans, so that it at least understands the need for those minor inconveniences that you sometimes can’t do without. If you do not have enough experience to make any changes, then do not hesitate to seek the advice of professionals:

7. Clearly define your area of ​​competence and incompetence and do not be afraid to discuss this issue with management.

Once upon a time I thought that a professional is someone who knows and knows everything. With age came the understanding that a professional is one who knows the limits of his competence and incompetence. There are professional network administrators, professional storage system engineers, professional technical support specialists, etc., but there are no people who know everything about everything. If the zoo from the technologies you serve, then the attempt to serve all of this alone (even if in connection with small volumes) causes great respect from colleagues, but it is unlikely to ever be appreciated by the employer, but to ask you will be for all this to the fullest. Indicate to management what systems you are fully familiar with, and where your knowledge may not be enough - let them decide whether to leave things as they are, hire another specialist or use the services of expert technical support from an external IT company. In general, take the rule and:

8. Always share responsibility for the decisions made, especially if it is not your sole decision.

You have a task to buy equipment, you chose a model, requested an invoice, but at the last moment the purchasing department put a veto on it - it’s cheaper on the Yandexmarket! Ok, no problem, but in this case, let the purchasing department be responsible for the purchase of this equipment. I know a couple of companies where the purchasing department has been waiting for the delivery of purchased laptops “at the lowest price” for 2 years. Similarly, with all other decisions, it is best to approve them with the manager, showing possible alternatives in advance, and receiving confirmation from him about the correctness of the decision made. And yes, just in case:

9. Agree on all decisions in writing.

Teach everyone that all decisions are agreed in writing - a simple e-mail will help you when the proceedings begin (not “if”, but “when”). It is necessary to coordinate all decisions without exception, so that people do not have a defensive reaction when you suddenly send a letter of approval for no reason at all, although before that all decisions were made orally. A very common situation is when your colleagues ignore written agreements - in this case, coordinate the decision with them orally, and after the conversation send them a letter like “Dear Pal Palych, sending the list of decisions agreed with you: ...” in the case of proceedings Palyu at least there will be questions why he still didn’t react to your letter. In general, written approvals are one of the procedures that you will need to introduce in order to properly cope with your work:

10. Create clear and convenient rules of interaction with you. Do everything so that users do not experience anxiety.

In small organizations there is a very common way to solve problems in the computer: go to the system administrator, take him by the hand, bring him to your computer, show the problem, do not let go until the problem is resolved. Naturally, no thoughtful, systematic, focused work on the current maintenance and configuration of servers is possible with this approach. But usually, users come to this way of solving problems empirically, because if you just say that you have a problem, then the admin will come incomprehensibly when, and having come, not the fact that he will do, but it’s necessary to work and time does not wait, and this all makes the users worry and, ultimately, makes them go to such funny, first look, deeds. Do not make your users worry: give them dates when you can help them, indicate terms when the problem will be fixed, and in cases where you have a lot of serious work planned in the coming days, write them letters in advance that you have reception rooms watches from so many to so many (of course, with the indication of the reason). Ask them to submit non-urgent tasks in writing and promise to work them out within certain time frames. And most importantly: keep your promises - the trust and love of users you will be provided.

Well, after all of the above, do not forget to do the following:

11. Regularly check backups

You all know the old joke that there are two types of sysadmins: those that do not back up yet, and those that already do. I would change this joke, adding that there is a third type - which also checks the created backups. Don't get me wrong, but in our business losing data is the height of lack of professionalism. No matter how many days a day you have been busy at work, always take time to check that you have created and backed up data correctly and that you can restore these same data.

12. Repeat the cycle: “removal of needs - risk analysis - coordination - alignment” at least once a year

Small business is changing very quickly: just yesterday, the company wrote out 3 bills a week, and today they are discharged 20 pieces per minute. Just yesterday, all they needed was a daily backup of the data, and today the loss of information in 5 minutes is already critical for business. Your task is to always be in the forefront of all changes in business and stay ahead of them.


Well, as an afterword I would like to add that in any organization the formation of a structural unit begins with its head. So if you are the only IT-aware person in your company, then I assure you that you are already a CIO, regardless of what you have written in your workplace, they will ask you as a manager, not as an executive. I hope that the above instructions will allow you to take a different look at your daily work, it is easier to join a new role for you and make your work as calm and predictable as possible.

Ivan Kormachev
Company "IT Department"
www.depit.ru