IT infrastructure reanimation
Greetings, Habrazhiteli.
I inherited a “legacy” IT infrastructure of a budget organization, very powerful, with a margin for performance, but absolutely not manageable. What we have:
All this is seasoned with an indefinite number of hardware and virtual servers (more than fifty). In view of the fact that there is no documentation (the notes of the administrators do not count) it is impossible to clothe the network parameters in any specific numbers.
How did that happen? It's very simple - the average administrator's salary is 12 thousand rubles, hence a very large staff turnover and a pathological reluctance to work normally for those who still wished to stay working. The second point - there were many expensive projects with very correct goals, which were never brought to a logical conclusion. Third, the skills and computer skills of the majority of staff are extremely low.
')
Initially, when the trees were bigger and the grass is greener, and the number of ports only passed for 1000, everything was conceived beautifully and competently:
Infrastructure
The network was built and built only on the equipment of Cisco Systems. At the core of the network is an optical switch, the highways are routed only with optics, the access level is exclusively for Cisco-controlled L3 switches. On all ports of the access level, port security was labeled with the MAC address; when another computer was connected, the port dropped. This was abandoned at the moment when the average employee could afford a laptop. Naturally, he put him on to work, stuck and craved to work, but he got a drop in the port, nerves, etc., plus the painful craving for some employees to change places also did not contribute to the development of this practice.
Network logic
Virtual network segments were created with a linkage of these to a specific territorial location. This practice has sunk into oblivion after the bright idea came to issue vlan to the port in divisions and isolate them from the rest of the subnets. But the people who started it, left and it turned out that the vlans and the ACL managed to spawn, but they didn’t have time to deploy in the enterprise and kill the old logic. It turned out a mess.
Car park
99.9% of all machines are Windows. How to steer? Of course the domain! No sooner said than done. Raised, set up, started an org. structure, group policies were hanged, users were brought in and received a megaproser (sorry for my French, but the organizational measures accompanying the technical solution were merged into the toilet). The average computer was used by several people during the working day. Those who were too lazy to remember the login and password, he pasted it on the monitor, the rest even wrote down a piece of paper (by the way, there were printed cards with credentials that were lost even faster than handwritten) did not bother themselves and went to the machines with the “shared” password. Someone saw an incomplete session, sat down, worked, saved documents, the next day already working with his profile matt yelled "Where are my documents that I did yesterday, the My Documents folder is empty." As a result, the real domain works only in one department, where for a long time very educated people work in terms of IT.
Information Services
As such, in our organization they are not. It is possible with a certain stretch to include sites, business cards units on the third-level domains. Here is a separate story. There was a hosting - hacked. They created a new "more secure", dragged the most important sites to it - they hacked it again. Created a third - the story repeated. As a result, we have 4 or 5 hosting sites plus sites of those divisions where there are own developers.
Nowadays
All these problems over time and the increase in the number of subscribers only multiplied, very literate people came, tried to patch holes, gained experience and left without regret for more adequate money, and all their undertakings overgrown with mud and only created problems due to the need for certain skills of admins for their operation. A lot of problems added a project to deploy a WiFi network (60 points). Netbooks, tablets, phones got into the network. The management’s reproaches are constantly heard that it slows down the network, in fact it turns out that the local network is working hardly at 10% of its capacity, and everything rests on the provider’s channel width (90 Mbps).
What to do?
For myself, he outlined three stages of work in order to clear up this chaos and at the same time not to tear the navel, and most importantly * om
The first stage: "Control all and all"
Screwed the collection of statistics on SNMP from the equipment. Used Cacti. The tasks are solved by 100%, beautiful graphics, the leadership likes.
I installed and configured the ticket system (OTRS), set up departments in queues, escalation and all that. Open sloppiness became less. Again, reports for management seem to be liking too.
It is planned to raise a normal Call-center, dispatch service so that not a single call goes nowhere and the admins do not blink their eyes, saying “I don’t know anything, I don’t remember, they could call”, and at the same time remove the personal contact of the admins and staff.
A transparent proxy and a tough Gestapo are planned (I hate Vkontakte! It gives more than 50% of traffic).
As soon as I understand in the smallest detail what is happening in my network, you can safely move on to the second stage.
The second stage: "Sanitary"
Cremation of all stillborn, who died by his own death, but who is in a state of zombies, or hellish agonizing. In other words, removing unnecessary Vlans, transferring hardware servers to virtuals. From the released iron to collect some cluster, have not yet decided which one. The goal of this stage is to make from the fact that there is a system that is as simple as possible and convenient in terms of maintenance. Moreover, this should be done as imperceptibly as possible for users, because the internal resistance of the organization to everything new is simply colossal in strength.
The third stage: “We can’t do that”
If a couple of intelligent administrators can manage all of these (enikeyshchiki do not count), then it will be possible to focus on creating a single information space, with a portal and real information services. But it seems to me that it will not be soon.
On this I want to complete my narration, thank you for your attention, waiting for the criticism on the merits.
If the community is interested, I will publish specific solutions to the challenges we face.
I inherited a “legacy” IT infrastructure of a budget organization, very powerful, with a margin for performance, but absolutely not manageable. What we have:
- 4000 hosts on the network
- Access Level Switches: Cisco Catalyst 35xx (~ 2500 ports)
- In the center is a Cisco Catalyst 6500 optical switch
- Cisco wireless access points of various models (~ 1500 clients daily) managed by a Cisco 44xx controller
- Internet Releases Cisco 38xx Router
- More than 50 vlans are created in the network.
All this is seasoned with an indefinite number of hardware and virtual servers (more than fifty). In view of the fact that there is no documentation (the notes of the administrators do not count) it is impossible to clothe the network parameters in any specific numbers.
How did that happen? It's very simple - the average administrator's salary is 12 thousand rubles, hence a very large staff turnover and a pathological reluctance to work normally for those who still wished to stay working. The second point - there were many expensive projects with very correct goals, which were never brought to a logical conclusion. Third, the skills and computer skills of the majority of staff are extremely low.
')
Initially, when the trees were bigger and the grass is greener, and the number of ports only passed for 1000, everything was conceived beautifully and competently:
Infrastructure
The network was built and built only on the equipment of Cisco Systems. At the core of the network is an optical switch, the highways are routed only with optics, the access level is exclusively for Cisco-controlled L3 switches. On all ports of the access level, port security was labeled with the MAC address; when another computer was connected, the port dropped. This was abandoned at the moment when the average employee could afford a laptop. Naturally, he put him on to work, stuck and craved to work, but he got a drop in the port, nerves, etc., plus the painful craving for some employees to change places also did not contribute to the development of this practice.
Network logic
Virtual network segments were created with a linkage of these to a specific territorial location. This practice has sunk into oblivion after the bright idea came to issue vlan to the port in divisions and isolate them from the rest of the subnets. But the people who started it, left and it turned out that the vlans and the ACL managed to spawn, but they didn’t have time to deploy in the enterprise and kill the old logic. It turned out a mess.
Car park
99.9% of all machines are Windows. How to steer? Of course the domain! No sooner said than done. Raised, set up, started an org. structure, group policies were hanged, users were brought in and received a megaproser (sorry for my French, but the organizational measures accompanying the technical solution were merged into the toilet). The average computer was used by several people during the working day. Those who were too lazy to remember the login and password, he pasted it on the monitor, the rest even wrote down a piece of paper (by the way, there were printed cards with credentials that were lost even faster than handwritten) did not bother themselves and went to the machines with the “shared” password. Someone saw an incomplete session, sat down, worked, saved documents, the next day already working with his profile matt yelled "Where are my documents that I did yesterday, the My Documents folder is empty." As a result, the real domain works only in one department, where for a long time very educated people work in terms of IT.
Information Services
As such, in our organization they are not. It is possible with a certain stretch to include sites, business cards units on the third-level domains. Here is a separate story. There was a hosting - hacked. They created a new "more secure", dragged the most important sites to it - they hacked it again. Created a third - the story repeated. As a result, we have 4 or 5 hosting sites plus sites of those divisions where there are own developers.
Nowadays
All these problems over time and the increase in the number of subscribers only multiplied, very literate people came, tried to patch holes, gained experience and left without regret for more adequate money, and all their undertakings overgrown with mud and only created problems due to the need for certain skills of admins for their operation. A lot of problems added a project to deploy a WiFi network (60 points). Netbooks, tablets, phones got into the network. The management’s reproaches are constantly heard that it slows down the network, in fact it turns out that the local network is working hardly at 10% of its capacity, and everything rests on the provider’s channel width (90 Mbps).
What to do?
For myself, he outlined three stages of work in order to clear up this chaos and at the same time not to tear the navel, and most importantly * om
The first stage: "Control all and all"
Screwed the collection of statistics on SNMP from the equipment. Used Cacti. The tasks are solved by 100%, beautiful graphics, the leadership likes.
I installed and configured the ticket system (OTRS), set up departments in queues, escalation and all that. Open sloppiness became less. Again, reports for management seem to be liking too.
It is planned to raise a normal Call-center, dispatch service so that not a single call goes nowhere and the admins do not blink their eyes, saying “I don’t know anything, I don’t remember, they could call”, and at the same time remove the personal contact of the admins and staff.
A transparent proxy and a tough Gestapo are planned (I hate Vkontakte! It gives more than 50% of traffic).
As soon as I understand in the smallest detail what is happening in my network, you can safely move on to the second stage.
The second stage: "Sanitary"
Cremation of all stillborn, who died by his own death, but who is in a state of zombies, or hellish agonizing. In other words, removing unnecessary Vlans, transferring hardware servers to virtuals. From the released iron to collect some cluster, have not yet decided which one. The goal of this stage is to make from the fact that there is a system that is as simple as possible and convenient in terms of maintenance. Moreover, this should be done as imperceptibly as possible for users, because the internal resistance of the organization to everything new is simply colossal in strength.
The third stage: “We can’t do that”
If a couple of intelligent administrators can manage all of these (enikeyshchiki do not count), then it will be possible to focus on creating a single information space, with a portal and real information services. But it seems to me that it will not be soon.
On this I want to complete my narration, thank you for your attention, waiting for the criticism on the merits.
If the community is interested, I will publish specific solutions to the challenges we face.
Source: https://habr.com/ru/post/163027/
All Articles