Rules of life of the sysadmin
We all make mistakes in our lives. Miscalculations in a professional career sometimes affect personal life, and vice versa. If the delivery, which a saleswoman in a shop didn’t give you enough in a human error, doesn’t incur anything critical after buying bread, then, for example, miscalculation of an engineer when calculating and designing a structure entails not only financial costs, but can also have irreparable consequences.
Since we are a hosting company, I would like to draw your attention to the errors associated with this particular service industry. And how, if not to the data center engineers - those people who support non-stop equipment operation, do not know all the flaws of their working moments. Let's try together with you to sort out possible options for errors in the difficult work of the system administrator. After all, it will help to raise the level of support, and thus not to attack us with you on the same rake.
')
I think many administrators will agree that sometimes it is not so easy to make the necessary server settings, and it is also quite difficult to carry out the necessary technical support of the equipment 24/7. Presentation of exactly how the system commands work, the availability of knowledge to “fix” something, when a breakdown / failure occurs, or how to monitor equipment performance, as well as set up a backup correctly, or how to create a smart script that will correctly work - all these skills and not only make you a good system administrator. Knowing these things and adhering to the basic rules will help you keep the system running, and also make your users a little bit happier.
Most of these "commandments" you probably have heard many times already. Perhaps some of them you had a chance to check on your "skin". All this practical knowledge was obtained thanks to many years of experience in system administration, and they helped many of us to think reasonably even when it was really “hot”. Here we go…
Do not forget to check whether you are ready to cancel the actions that are going to be done at the moment? Even for the simplest changes, you should consider being able to go back to the original version. There are many ways to leave the retreat and, if necessary, to return to the starting point. Make backup copies of the files you are going to edit. Pay attention to any problems that you encounter in the process. Be sure to check the changes on the test version and only after that release them to the world.
Also, before moving on, make sure that the current edits are successful. In general, such work should be planned in advance. And remember, an extra pair of eyes will always help you notice problems that you have missed.
Never implement global change just before going away for a few days. Always give innovations the opportunity to show themselves and only then go with peace of mind for the weekend.
Always set yourself a goal to thoroughly understand the situation and determine the true cause of the problem. Do not rush to conclusions - check everything consistently and slowly. Even then, when it wasn’t immediately possible to establish a “diagnosis” - don’t give up and don’t try to stop, but continue to analyze the problem. If you need to - ask for help from colleagues or those professionals who will help you find what the problem is. And remember, the one who is looking for - always finds!
Test your disaster recovery plan in practice to feel comfortable and confident if you have to activate it. If you do not do this, then at least two things can happen: first, you will not be sure that disaster recovery works at all, and second, you can act uncertainly while taking emergency steps to correct the situation.
Is it bad to be prepared for possible interruptions? You can become a real superhero in the eyes of your own clients if you reanimate the work of their project or equipment in two clicks, without any dancing with a tambourine. Yes, someone can say that it comes with experience. But do we have a few free years? Hosting clients, as a rule, cannot wait even an hour, and there can be no talk of years. Therefore, we advise you to calculate the possible recovery options in advance and thereby protect several of your nerve cells from sometimes “angry” and ... once again very “angry” customers.
How easy it is to make a mistake. Always check your scripts, even if you work with them throughout your career. And especially check those scripts that someone else used besides you. Check scripts with or without parameters. Check the scripts for common mistakes that could be made by someone else. Just check the scripts and they will never let you down!
Automate everything that you need to do more than three times, as well as something that is quite complex in execution. Do not give yourself an additional opportunity to make a mistake, but rather entrust this task to the scripts, since the probability of an error immediately approaches zero. After all, you will also save a lot of time and effort, as you will immediately no longer need to search for someone else to do this routine work for you.
Document everything that you do regularly. After all, your actions may not be obvious to others. Always remember that your colleagues are not telepaths, and do not know how to read each other’s thoughts. If the question at the request of the client could not be resolved during your shift, then please find time to leave simple and understandable notes to this ticket, so that your colleagues would not have any additional questions, and they did everything correctly. Why, you can call and clarify - you say. And in some difficult situations most likely you will be right. But no one will deny that a few words in the “notes” to the ticket can save you from an unnecessary call, when you, for example, decided to devote this free time to your beloved person. Would she or he be glad that on your day off, which was supposed to be your only, you are still distracted by work? Think about it before you leave the shift.
Also do not forget to add comments to your scripts. After all, the script that you regularly use now, after a year… the second “idle” can become so mysterious even for its author. And perhaps it’s not worth sacrificing readability for the sake of brevity, as it is possible that someone else besides you will have to work with this code. Do not use abbreviations that only you understand. Notes should be clear even to your grandmother :) You should not play in the lonely fighter of the invisible front, if you work in a team. Yes, system administrators are, as a rule, “loners” by their very nature, but the ability to work in a team is a very important quality for working in a modern company, remember this.
Understanding one’s own mistakes in one’s own thinking may be the only way to eliminate them in the future. Pay attention to what you are doing is not as it should be (according to the opinion of the "great luminaries"), especially if you repeat this "error" more than once.
Yes, from the side you know better, but no one else knows you better than you. Well, except that you work together with your mom, which is very unlikely :) Just remember this and then there will be less “errors”.
To be a little bit "sick in the head" is ridiculous - you will say. But when really big projects “fall” because the administrator or the whole technical support team simply didn’t check again what had already been checked many times before. Never blindly trust what the data center engineers or hosting company administrators write to you (if you rent equipment through them) in response to your request. Take some time and check the server yourself that everything is done exactly as you need.
As a rule, a couple of minutes of time now will help save you forever in the future. If, God forbid, the server "falls" and the project owners start losing, as they like to say - "millions", then who do you think whose heads can "fly"? Yes, sooner or later the servers fall, but a really competent sysadmin will calculate not only possible errors on his part, but also think through all sorts of errors of those “smart” guys in the data center (hosting provider), who are people too and can be banal.
Sometimes clients even do not realize how much effort it takes to “kick” the data center engineer or to “send” him right where he needs to ensure high uptime of the equipment. Yes, administrators often “save the world” imperceptibly for ordinary users, but we don’t pretend to the laurels of the “greats”, but simply do our work. Do it efficiently, and do not care if someone considers you paranoid, if at the same time you are a real guru of your business.
Not every problem will knock on the door. Find some free time in your schedule and check all systems. It is possible to set up an additional monitoring system, because not all parameters need to be monitored independently.
And the fact that there is no way without your attention - check it personally. Sometimes a person may notice what the machine will miss, and vice versa. Whatever they say, but the right balance of both “technologies” is capable of doing great things, do not forget about it.
In matters of security, there is never such a thing as “a lot”, but the efforts that you put into ensuring it must be commensurate with the data that you protect. Watch for critical parameters, test vulnerabilities, run only those services that are really needed.
Always be alert and ready for any signs of hacking or compromising the system. Remember - God protects you, and I think, you would not have refused, that it would be in such “spy battles” that the great would be in your team. But for this you need to be ready for anything, because in such "games" no one ever observes the rules.
Analysis of the logs can warn you about a possible problem before it starts to threaten the work of the hosting and the whole system. Check the log files for errors and notifications.
Also, do not spare funds for specialized software (software) to analyze this kind of files or try to develop your own, which is quite a costly undertaking. I think you wonderfully understand that it may take a lot of time for a detailed analysis of log files, but people have not yet learned to live forever, because good software will be by the way and not only for such tasks.
You need to backup everything and always. There is an opinion that system administrators, and not only them, are divided into 3 categories: those who do not make backups, those who already make backups and those who already make and check the possibility of recovery.
It is not enough just to back up data, you also need to test the possibility of their recovery. Remember, the replicated error is still an error. Check backups to make sure they are normal, and believe me, your sleep will only be better.
Some system administrators are a little arrogant, as they are real pros of their field. However, this does not mean that you should look down on people, much less do so in relations with your clients. Treat customers the way you would like them to treat you.
Yes, they may not know what the command line is, they may not even know how to properly connect via FTP and ask about it again. But do not rush to send them to / for ... (insert your own version), but try again to explain the same thing, but in other words or give a link to the information where it is described in an accessible form. And it doesn’t matter that you have already provided a “carriage” of links - give one that will help the client understand. Be patient and calm. Also remember, that client who is understood here - will never leave you, even to where it will be a little cheaper.
Always keep your users up to date with all the events. Perhaps the next scheduled work in the data center will not affect the uptime of the equipment, but you still notify them to show that work on improving the quality of service is constantly being carried out and this does not always entail downtime.
Also, if the "trouble" you have not passed, you should not try to hide something from the client using the standard unsubscribe. Give the user the most detailed answer and keep him informed of all events until the problem with his hosting is resolved. Do not wait until he writes to you again and asks the same question, but get ahead of him and notify about the whole course of action. Do not be afraid to appear incompetent in some matters, but rather show your concern. Believe me, for some customers this is equally important.
Be yourself, be easier and users will be with you. After all, no one asks you to become a superhero, and why do you need that league of supernatural people. Start with simple things - help some online project to avoid downtime and quickly raise the server, if God forbid, it "fell." Solve the problems of your customers qualitatively and quickly, and better - do everything to keep them as small as possible and, believe me, the number of your “army of fans” will only grow.
Movement is not only life, but also development. If you do not move forward, then you are not just standing, but slowly “returning” back ... to the past. Yes, you and I can be old-fashioned, and some even like the old school. But please do not do this with technology. Never stop searching for and learning new solutions. Always be in trend, especially in those issues that directly relate to your work.
Remember, great are not born, they become. Isn't it awesome exactly when you tell your client about a new product on the web technologies market that will help improve the work of his online project. And if you also introduce it, how do you think, to whom will he again turn for advice and order a support service without any questions?
Harmony should be in everything. Do not get hung up on one job, because besides her there are other delights of life. Let your work activity be that which helps to realize not only children's dreams, but also to realize the plans of an adult life. Work in pleasure and do not forget to share it with others. You'll see, the moment will come when they will reciprocate.
But to enjoy what you are doing, and even in the company of wonderful people, this is such an incredible thrill. Yes, it may not come immediately, but who said that being a sysadmin is easy. Do not stop there and everything will turn out. Believe in what you are doing and your work will definitely be appreciated.
Once, one of the Nobel Prize winners said: “Only the one who does nothing is not mistaken! Do not be afraid to make mistakes - be afraid of repeating mistakes! ”Remember this and you will succeed.
Since we are a hosting company, I would like to draw your attention to the errors associated with this particular service industry. And how, if not to the data center engineers - those people who support non-stop equipment operation, do not know all the flaws of their working moments. Let's try together with you to sort out possible options for errors in the difficult work of the system administrator. After all, it will help to raise the level of support, and thus not to attack us with you on the same rake.
')
I think many administrators will agree that sometimes it is not so easy to make the necessary server settings, and it is also quite difficult to carry out the necessary technical support of the equipment 24/7. Presentation of exactly how the system commands work, the availability of knowledge to “fix” something, when a breakdown / failure occurs, or how to monitor equipment performance, as well as set up a backup correctly, or how to create a smart script that will correctly work - all these skills and not only make you a good system administrator. Knowing these things and adhering to the basic rules will help you keep the system running, and also make your users a little bit happier.
Most of these "commandments" you probably have heard many times already. Perhaps some of them you had a chance to check on your "skin". All this practical knowledge was obtained thanks to many years of experience in system administration, and they helped many of us to think reasonably even when it was really “hot”. Here we go…
Return point
Do not forget to check whether you are ready to cancel the actions that are going to be done at the moment? Even for the simplest changes, you should consider being able to go back to the original version. There are many ways to leave the retreat and, if necessary, to return to the starting point. Make backup copies of the files you are going to edit. Pay attention to any problems that you encounter in the process. Be sure to check the changes on the test version and only after that release them to the world.
Also, before moving on, make sure that the current edits are successful. In general, such work should be planned in advance. And remember, an extra pair of eyes will always help you notice problems that you have missed.
The end of the week is not the time for innovation.
Never implement global change just before going away for a few days. Always give innovations the opportunity to show themselves and only then go with peace of mind for the weekend.
Get to the truth
Always set yourself a goal to thoroughly understand the situation and determine the true cause of the problem. Do not rush to conclusions - check everything consistently and slowly. Even then, when it wasn’t immediately possible to establish a “diagnosis” - don’t give up and don’t try to stop, but continue to analyze the problem. If you need to - ask for help from colleagues or those professionals who will help you find what the problem is. And remember, the one who is looking for - always finds!
Disaster recovery
Test your disaster recovery plan in practice to feel comfortable and confident if you have to activate it. If you do not do this, then at least two things can happen: first, you will not be sure that disaster recovery works at all, and second, you can act uncertainly while taking emergency steps to correct the situation.
Is it bad to be prepared for possible interruptions? You can become a real superhero in the eyes of your own clients if you reanimate the work of their project or equipment in two clicks, without any dancing with a tambourine. Yes, someone can say that it comes with experience. But do we have a few free years? Hosting clients, as a rule, cannot wait even an hour, and there can be no talk of years. Therefore, we advise you to calculate the possible recovery options in advance and thereby protect several of your nerve cells from sometimes “angry” and ... once again very “angry” customers.
Without testing anywhere
How easy it is to make a mistake. Always check your scripts, even if you work with them throughout your career. And especially check those scripts that someone else used besides you. Check scripts with or without parameters. Check the scripts for common mistakes that could be made by someone else. Just check the scripts and they will never let you down!
Automation
Automate everything that you need to do more than three times, as well as something that is quite complex in execution. Do not give yourself an additional opportunity to make a mistake, but rather entrust this task to the scripts, since the probability of an error immediately approaches zero. After all, you will also save a lot of time and effort, as you will immediately no longer need to search for someone else to do this routine work for you.
Documenting
Document everything that you do regularly. After all, your actions may not be obvious to others. Always remember that your colleagues are not telepaths, and do not know how to read each other’s thoughts. If the question at the request of the client could not be resolved during your shift, then please find time to leave simple and understandable notes to this ticket, so that your colleagues would not have any additional questions, and they did everything correctly. Why, you can call and clarify - you say. And in some difficult situations most likely you will be right. But no one will deny that a few words in the “notes” to the ticket can save you from an unnecessary call, when you, for example, decided to devote this free time to your beloved person. Would she or he be glad that on your day off, which was supposed to be your only, you are still distracted by work? Think about it before you leave the shift.
Also do not forget to add comments to your scripts. After all, the script that you regularly use now, after a year… the second “idle” can become so mysterious even for its author. And perhaps it’s not worth sacrificing readability for the sake of brevity, as it is possible that someone else besides you will have to work with this code. Do not use abbreviations that only you understand. Notes should be clear even to your grandmother :) You should not play in the lonely fighter of the invisible front, if you work in a team. Yes, system administrators are, as a rule, “loners” by their very nature, but the ability to work in a team is a very important quality for working in a modern company, remember this.
Pay attention to yourself
Understanding one’s own mistakes in one’s own thinking may be the only way to eliminate them in the future. Pay attention to what you are doing is not as it should be (according to the opinion of the "great luminaries"), especially if you repeat this "error" more than once.
Yes, from the side you know better, but no one else knows you better than you. Well, except that you work together with your mom, which is very unlikely :) Just remember this and then there will be less “errors”.
Paranoids still in fashion?
To be a little bit "sick in the head" is ridiculous - you will say. But when really big projects “fall” because the administrator or the whole technical support team simply didn’t check again what had already been checked many times before. Never blindly trust what the data center engineers or hosting company administrators write to you (if you rent equipment through them) in response to your request. Take some time and check the server yourself that everything is done exactly as you need.
As a rule, a couple of minutes of time now will help save you forever in the future. If, God forbid, the server "falls" and the project owners start losing, as they like to say - "millions", then who do you think whose heads can "fly"? Yes, sooner or later the servers fall, but a really competent sysadmin will calculate not only possible errors on his part, but also think through all sorts of errors of those “smart” guys in the data center (hosting provider), who are people too and can be banal.
Sometimes clients even do not realize how much effort it takes to “kick” the data center engineer or to “send” him right where he needs to ensure high uptime of the equipment. Yes, administrators often “save the world” imperceptibly for ordinary users, but we don’t pretend to the laurels of the “greats”, but simply do our work. Do it efficiently, and do not care if someone considers you paranoid, if at the same time you are a real guru of your business.
Initiative
Not every problem will knock on the door. Find some free time in your schedule and check all systems. It is possible to set up an additional monitoring system, because not all parameters need to be monitored independently.
And the fact that there is no way without your attention - check it personally. Sometimes a person may notice what the machine will miss, and vice versa. Whatever they say, but the right balance of both “technologies” is capable of doing great things, do not forget about it.
Security
In matters of security, there is never such a thing as “a lot”, but the efforts that you put into ensuring it must be commensurate with the data that you protect. Watch for critical parameters, test vulnerabilities, run only those services that are really needed.
Always be alert and ready for any signs of hacking or compromising the system. Remember - God protects you, and I think, you would not have refused, that it would be in such “spy battles” that the great would be in your team. But for this you need to be ready for anything, because in such "games" no one ever observes the rules.
Log files
Analysis of the logs can warn you about a possible problem before it starts to threaten the work of the hosting and the whole system. Check the log files for errors and notifications.
Also, do not spare funds for specialized software (software) to analyze this kind of files or try to develop your own, which is quite a costly undertaking. I think you wonderfully understand that it may take a lot of time for a detailed analysis of log files, but people have not yet learned to live forever, because good software will be by the way and not only for such tasks.
Backup
You need to backup everything and always. There is an opinion that system administrators, and not only them, are divided into 3 categories: those who do not make backups, those who already make backups and those who already make and check the possibility of recovery.
It is not enough just to back up data, you also need to test the possibility of their recovery. Remember, the replicated error is still an error. Check backups to make sure they are normal, and believe me, your sleep will only be better.
Time is precious for everyone
Some system administrators are a little arrogant, as they are real pros of their field. However, this does not mean that you should look down on people, much less do so in relations with your clients. Treat customers the way you would like them to treat you.
Yes, they may not know what the command line is, they may not even know how to properly connect via FTP and ask about it again. But do not rush to send them to / for ... (insert your own version), but try again to explain the same thing, but in other words or give a link to the information where it is described in an accessible form. And it doesn’t matter that you have already provided a “carriage” of links - give one that will help the client understand. Be patient and calm. Also remember, that client who is understood here - will never leave you, even to where it will be a little cheaper.
Informing
Always keep your users up to date with all the events. Perhaps the next scheduled work in the data center will not affect the uptime of the equipment, but you still notify them to show that work on improving the quality of service is constantly being carried out and this does not always entail downtime.
Also, if the "trouble" you have not passed, you should not try to hide something from the client using the standard unsubscribe. Give the user the most detailed answer and keep him informed of all events until the problem with his hosting is resolved. Do not wait until he writes to you again and asks the same question, but get ahead of him and notify about the whole course of action. Do not be afraid to appear incompetent in some matters, but rather show your concern. Believe me, for some customers this is equally important.
Simplicity is the key to success
Be yourself, be easier and users will be with you. After all, no one asks you to become a superhero, and why do you need that league of supernatural people. Start with simple things - help some online project to avoid downtime and quickly raise the server, if God forbid, it "fell." Solve the problems of your customers qualitatively and quickly, and better - do everything to keep them as small as possible and, believe me, the number of your “army of fans” will only grow.
Live and learn
Movement is not only life, but also development. If you do not move forward, then you are not just standing, but slowly “returning” back ... to the past. Yes, you and I can be old-fashioned, and some even like the old school. But please do not do this with technology. Never stop searching for and learning new solutions. Always be in trend, especially in those issues that directly relate to your work.
Remember, great are not born, they become. Isn't it awesome exactly when you tell your client about a new product on the web technologies market that will help improve the work of his online project. And if you also introduce it, how do you think, to whom will he again turn for advice and order a support service without any questions?
Balance in everything
Harmony should be in everything. Do not get hung up on one job, because besides her there are other delights of life. Let your work activity be that which helps to realize not only children's dreams, but also to realize the plans of an adult life. Work in pleasure and do not forget to share it with others. You'll see, the moment will come when they will reciprocate.
But to enjoy what you are doing, and even in the company of wonderful people, this is such an incredible thrill. Yes, it may not come immediately, but who said that being a sysadmin is easy. Do not stop there and everything will turn out. Believe in what you are doing and your work will definitely be appreciated.
Once, one of the Nobel Prize winners said: “Only the one who does nothing is not mistaken! Do not be afraid to make mistakes - be afraid of repeating mistakes! ”Remember this and you will succeed.
Source: https://habr.com/ru/post/278235/
All Articles