Three years of those. support, 152 health facilities, 3740 pcs
In 2012-2013, a large infrastructure project was launched, part of which was the creation of an information network segment (SCS, LAN, protection of transmission channels, centralized management) in 152 health care facilities (medical institutions).
In the total amount, more than 50,000 different devices (PCs, servers, switches, firewalls, information protection systems, printers, UPS, etc.) were installed, configured, connected, enabled, activated, mounted and launched.
The scheme of the segment of the unified public health information system (EGISZ) of this project:
')
Here, for example, only a short excerpt of the delivery set to one of the hospitals.
Equipment manufacturers were identified in the TOR in advance, the requirements for equipment and service and technical support were also quite harsh, but fair.
As for the system units, they were:
ICL RAY B314.2 Workstation (Intel Celeron G530 (2.4 GHz) / 2GB / HDD500GB) - 3740 pcs.
Most of the servers were:
ICL teamRAY 2062–2U server (2U, 1xQC, 8 GB, RAID 512, 3x1TB, DVD (teamRAY 2062–2U), there were 155 of them.
This article will be more about the PC material, as the server equipment was installed in the telecommunications racks and served by system administrators, but the workstations are subject to more stringent user treatment.
Examples of installed server and switching equipment:
Not without such neighborhoods:
Let's return to the system blocks.
On each system unit, there is a serial number on the top, seals are installed on the back and inside of the components to control the opening of the case and replace the components.
The service warranty for this equipment is three years, it has already expired. And everything became almost history, a stepping stone in the development of the infrastructure of the Ministry of Health.
Since, by the nature of my activity, I had to take the engineering baton on the first line of support for PC data, then I’ll tell you about the PCs themselves and their testing, how the technical support went, how the system unit went through all the stages of warranty acceptance and repair, and in the final phase, he returned back to the health care facility already in working condition.
At once I want to express my gratitude to the engineering staff (system administrators, heads of IT departments, etc.) of health facilities with whom they had to communicate, work through the issues of setting up, deploying and operating the equipment.
At one of the stages of the project, I even had to interact several times with each engineer of the health care facility. I liked very much that in many cities the engineers of adjacent health care facilities know each other, working together and helping to solve many problems arising from the operation of this complex system.
Further, according to the text, the responsible officer in the health care facility responsible for IT equipment will be referred to as a health care engineer to make it easier to understand. The system unit will also be called simply the PC health care facility.
Description and composition of the PC health care facility.
In fact, the composition of the PC health facilities, motherboards and HDDs could vary by manufacturers, but their main characteristics and HDD capacity remained unchanged for all PCs.
Find two differences:
The software is all licensed, of course, the native OS Win7Pro64bit, the performance index of the PC LPU:
Device Manager:
Work HDD installed by the manufacturer:
Replaced under warranty HDD new ones were put like this (in principle, it was important that the volume of the new drive was not less than the previously installed one):
Only the processor and the integrated video card remained unchanged for the PC of the health care facility - Intel Celeron G530 (2M Cache, 2.40 GHz).
Data from HWMonitor:
Data from HDTune (on the example of two HDDs, since at first there was one regular HDD manufacturer, then the other as part of a PC health facility):
Problems and solutions.
How did the service support for this project work (on the example of the PC of health facilities):
1. The engineer of the health care facility receives information from the health care facility employee (for which the PC is listed) that a particular health care facility PC does not function. This information is analyzed, a solution is being worked out if it can be corrected on its own (reinstalling the software and carrying out maintenance, not all engineers of the health care facilities could carry out on-site maintenance of the health care center PC).
Regarding the shipping seals on the health facility PC - these shipping seals were qualified by a qualified health care facility engineer to remove them for maintenance (PT) of the health facility PC without losing the warranty.
THAT can be carried out independently by trained personnel, according to the manufacturer's instructions, or for a fee, by an authorized service center.
In this case, the warranty may be refused if:
- the product has been modified or unprofessional repair has been performed on its own;
- the product has no or damaged label with a serial number;
- warranty labels (seals) or manufacturer’s markings on the component parts of the product are missing or damaged;
- there are mechanical damages of the case, components or separate components, traces of corrosion;
- and so on, in accordance with the manufacturer’s warranty conditions specified in the product certificate.
I’ll say right away that they didn’t do any special work with the LPU, since the engineering staff is already loaded heavily with them.
2. If the health facility PC does not work properly and this was confirmed by the health facility engineer, the health facility engineer sent us a request for warranty service support indicating the name of the health facility, the full number of the health care facility PC, and a description of the problem with the health facility PC. contact details.
3. This application is already from our side after receiving it was registered in the HelpDesk system (based on Spiceworks). Each application was assigned its own ticket number, the person responsible for this application was appointed and the deadline was planned. The same information on the actions on the ticket was sent to the initiator - the engineer of health facilities.
4. The person responsible for the ticket registered the number of the health facility PC, checked whether it was a new health facility PC or was already under repair (there were several PCs that came to the service 3-4 times), sent instructions to the health facility as a health facility PC to deliver to us for further with him action to repair.
5. The staff of the health care facility (or our employees, if possible) delivered the repair facility PC already to our warranty department, where the health care center PC received, checked and registered the ticket person, the relevant acts on the transfer of the health care facility equipment were issued to the health care facility.
6. Already we have the PC of the health care facility inspected by warranty engineers (external, internal inspection for compliance with the configuration), cleaned, checked for the stated problem. If the repair by an engineer could be done on site (resetting the BIOS, replacing the battery, removing the cable, removing a lot of dust, reinstalling the OS, testing the memory and HDD without errors), then the PC of the health care facility was put into operation if the replacement of functional parts was necessary (power supply, motherboard, HDD, memory strip, etc.), then the PC of the health care facility was sent for warranty repair to a specialized SC, which worked directly with the manufacturer of this equipment.
7. In the specialized SC, the non-working functional parts were already replaced with new ones, and then the PC of the health care facility returned to our warranty department.
8. The person responsible for the ticket received the Health Unit's PC back from the SC, checked its operation and initiated the issue back to the Health Facility.
9. Ticket closed after receiving a PC health facility in the health facility.
Complicated? Why do we need the first line of such technical support?
According to the project contract. Moreover, at this stage in the first line up to about a quarter of all tickets can be closed without the involvement of a specialized SC.
Some hospitals immediately worked directly with the SC, bypassing the first line of support.
What the owners of the personal computer of the hospital did in the hospital: they changed the HDD for others, inserted DVD-ROMs, changed the memory bars, turned off the fans and the dust sensor, and so on. All this was recorded in the ticket and for some tickets even the guarantee was denied, and even the SC could provide an official act on such a problem.
So, it was and is in operation 3740 PCs of health facilities - physically (approximately) about 500 PCs of health care facilities during these 3 years came to the guarantee. Those. 5% per year from about the total number of PCs.
They carried out maintenance work - cleaning (although it was not the responsibility of the engineers), checking cable connections, and diagnosing a faulty unit.
Further, the PC was sent to the service center (SC) to replace the faulty unit or to the health care facility back if the PC worked regularly after testing.
What changed in the SC (descending): HDD, power supplies, motherboards, memory bars, sat buttons and cables, only 2 times the processor, and that with the motherboard together.
A small potpourri of faults.
Faulty memory bar:
Blue screen when loading:
A dust sensor is installed at the bottom of the case:
A special StopDust software is installed for the sensor.
Only most of the PC health facilities at the first opening of the case on the fans give this real picture:
Problems with HDD:
Errors in SMART (it is clear that such a PC health care facility was able to work 3569 hours or 446 working days):
The problems that even we had to put up with:
1. A good SC of this manufacturer is very difficult to pick up - first they worked with one SC, but they took everything for a long time and gave it away from the repair - they said that there were problems with the supply and with the manufacturer ICL.
Then they worked with another SC, engineers work there promptly and the period of rotation of the PC health care facilities became less critical.
2. Problems with the health facility PC - Many health care center PCs in the health facility simply do not start from scratch, even if they were left in a warehouse at the health facility, etc., because the entire PC fleet was not immediately turned on and put into operation at the health facility. And how they are stored in warehouses in hospitals sometimes ... This is a separate song.
3. The built-in dust sensors in the PC of the health care facility do not particularly help, as the engineers of the health care facility can conduct maintenance, but this is not their responsibility, so this sensor must be used as it were, but its data is not particularly used.
4. Information, training and constant communication with health care facilities on problematic issues - observe repair time, if a PC is delivered for 300 km, clarify all aspects of the situation with documents and acts of acceptance, control and inform all involved persons by ticket. Especially, when a local engineer was changing at the hospital, and the new one was not aware of the situation at all, it all started in a new way.
And now, for sweets, some applications from health facilities (complaints to the health care facilities, some even deserve a bash!):
- “Sat does not turn on (turns on for 3 seconds and turns off)”;
- “the system does not start, when trying to reinstall, it displays a message that the hard disk will soon fail and is not installed”;
- "The Security Council does not turn on, the smell of burning";
- "The Security Council does not turn on, when you try to turn on, a crackling sound is heard";
- “The SAT turns on, but there is no power to the mouse and there is no video signal to the monitor”;
- “after switching on after a certain period of time it turns off or“ freezes ””;
- "does not turn on, but when it turns on it makes noise";
- “falls in the BSOD indicating the damage to the RAM”;
- “it slows down in the process of loading and freezes during operation, a blue screen happens”;
- “is overloaded after a short work”;
- “does not turn on (turns on for 3 seconds and turns off) the similar problem was last time, before the occurrence of such a problem another one appeared, the image swam, sinned first on the monitor and checked, now I sin on the video card”;
- “Our Security Council broke down, it DOES NOT INCLUDE”;
“The monitor is out of order, its malfunction has appeared, the image is twitching. And one more system unit, the cooler on the processor slows down and does not work ”;
- “There is a malfunction of the power supply on the computer. Apparently the power supply unit gives out voltage to the device case ";
“The system unit has broken, the download fails, it does not even reach the BIOS loading”;
- “the reboot is constant, it does not even reach the BIOS. Will go to the service for the fifth time ";
- “hard drive failed. ("Pours") ";
“First, the USB outputs on the front panel disappeared, then something with a video input, it does not connect with any monitor, even downloads are not visible”;
- “The system unit turned off during operation, the smell of burning PCB appeared. Our assumptions, short circuit in BP ";
- "reboots (without reaching the load of the operating system)";
- “we have another warranty computer, with a floating error. Video card artifacts that manifest themselves day after day ”;
- “! Vindous reported a failure (inevitable equipment failure) of hdd at this ARM ”;
- “PC does not start. No signal to monitor. It turns itself off. ";
- "Periodically, the PC does not turn on until you disconnect from 220V";
“The SB has failed, it does not load, and judging by a number of signs, the reason for this behavior in the motherboard”;
- “there was a problem while setting up the new SAT. After I put him on the domain, he stopped seeing me as an administrator and rejects all passwords, writes that he needs a raise ”;
- “when working with SAT, a knock is heard, which leads to a hang”;
- “It turns on only after entering the BIOS, the language in the BIOS is not switched, the indicators on the front panel of HDD_LED and Power_Led are confused”;
- Does not turn on the PC serial number: 1212, symptoms, as in the previous time.
Eventually.
Most of the PCs of health care facilities work, either as a standard without repair, or after maintenance and service.
But the backup data of the LPU PC in the LPU is basically not done (although the main data on the servers must also be in cloud solutions), so after replacing the HDD, the SC engineers had to do disk cloning before repair (if possible) or in the LPU empty PC health care facility, but working.
Technically, HDDs on PCs were needed only to run the OS and simple operation, although a special was installed there. Software to protect data. The main work with data in RMIS through the cloud, server and browser.
In general, the places of use of PC health facilities are very different, as it turned out, the treatment of users with them in each health care facility was also different.
And how happy the rebuilt PC, which is ready for further work, is:
In fact, the guarantee support for this “park” ended in 2016, now the components of the IT infrastructure are being updated at the health facilities and the introduction of more modern solutions.
Although, part of this park continues to work in 2018, although it is already supported by local system administrators.
In the total amount, more than 50,000 different devices (PCs, servers, switches, firewalls, information protection systems, printers, UPS, etc.) were installed, configured, connected, enabled, activated, mounted and launched.
The scheme of the segment of the unified public health information system (EGISZ) of this project:
')
Here, for example, only a short excerpt of the delivery set to one of the hospitals.
Equipment manufacturers were identified in the TOR in advance, the requirements for equipment and service and technical support were also quite harsh, but fair.
As for the system units, they were:
ICL RAY B314.2 Workstation (Intel Celeron G530 (2.4 GHz) / 2GB / HDD500GB) - 3740 pcs.
Most of the servers were:
ICL teamRAY 2062–2U server (2U, 1xQC, 8 GB, RAID 512, 3x1TB, DVD (teamRAY 2062–2U), there were 155 of them.
This article will be more about the PC material, as the server equipment was installed in the telecommunications racks and served by system administrators, but the workstations are subject to more stringent user treatment.
Examples of installed server and switching equipment:
Not without such neighborhoods:
Let's return to the system blocks.
On each system unit, there is a serial number on the top, seals are installed on the back and inside of the components to control the opening of the case and replace the components.
The service warranty for this equipment is three years, it has already expired. And everything became almost history, a stepping stone in the development of the infrastructure of the Ministry of Health.
Since, by the nature of my activity, I had to take the engineering baton on the first line of support for PC data, then I’ll tell you about the PCs themselves and their testing, how the technical support went, how the system unit went through all the stages of warranty acceptance and repair, and in the final phase, he returned back to the health care facility already in working condition.
At once I want to express my gratitude to the engineering staff (system administrators, heads of IT departments, etc.) of health facilities with whom they had to communicate, work through the issues of setting up, deploying and operating the equipment.
At one of the stages of the project, I even had to interact several times with each engineer of the health care facility. I liked very much that in many cities the engineers of adjacent health care facilities know each other, working together and helping to solve many problems arising from the operation of this complex system.
Further, according to the text, the responsible officer in the health care facility responsible for IT equipment will be referred to as a health care engineer to make it easier to understand. The system unit will also be called simply the PC health care facility.
Description and composition of the PC health care facility.
In fact, the composition of the PC health facilities, motherboards and HDDs could vary by manufacturers, but their main characteristics and HDD capacity remained unchanged for all PCs.
Find two differences:
The software is all licensed, of course, the native OS Win7Pro64bit, the performance index of the PC LPU:
Device Manager:
Work HDD installed by the manufacturer:
Replaced under warranty HDD new ones were put like this (in principle, it was important that the volume of the new drive was not less than the previously installed one):
Only the processor and the integrated video card remained unchanged for the PC of the health care facility - Intel Celeron G530 (2M Cache, 2.40 GHz).
Data from HWMonitor:
Data from HDTune (on the example of two HDDs, since at first there was one regular HDD manufacturer, then the other as part of a PC health facility):
Problems and solutions.
How did the service support for this project work (on the example of the PC of health facilities):
1. The engineer of the health care facility receives information from the health care facility employee (for which the PC is listed) that a particular health care facility PC does not function. This information is analyzed, a solution is being worked out if it can be corrected on its own (reinstalling the software and carrying out maintenance, not all engineers of the health care facilities could carry out on-site maintenance of the health care center PC).
Regarding the shipping seals on the health facility PC - these shipping seals were qualified by a qualified health care facility engineer to remove them for maintenance (PT) of the health facility PC without losing the warranty.
THAT can be carried out independently by trained personnel, according to the manufacturer's instructions, or for a fee, by an authorized service center.
In this case, the warranty may be refused if:
- the product has been modified or unprofessional repair has been performed on its own;
- the product has no or damaged label with a serial number;
- warranty labels (seals) or manufacturer’s markings on the component parts of the product are missing or damaged;
- there are mechanical damages of the case, components or separate components, traces of corrosion;
- and so on, in accordance with the manufacturer’s warranty conditions specified in the product certificate.
I’ll say right away that they didn’t do any special work with the LPU, since the engineering staff is already loaded heavily with them.
2. If the health facility PC does not work properly and this was confirmed by the health facility engineer, the health facility engineer sent us a request for warranty service support indicating the name of the health facility, the full number of the health care facility PC, and a description of the problem with the health facility PC. contact details.
3. This application is already from our side after receiving it was registered in the HelpDesk system (based on Spiceworks). Each application was assigned its own ticket number, the person responsible for this application was appointed and the deadline was planned. The same information on the actions on the ticket was sent to the initiator - the engineer of health facilities.
4. The person responsible for the ticket registered the number of the health facility PC, checked whether it was a new health facility PC or was already under repair (there were several PCs that came to the service 3-4 times), sent instructions to the health facility as a health facility PC to deliver to us for further with him action to repair.
5. The staff of the health care facility (or our employees, if possible) delivered the repair facility PC already to our warranty department, where the health care center PC received, checked and registered the ticket person, the relevant acts on the transfer of the health care facility equipment were issued to the health care facility.
6. Already we have the PC of the health care facility inspected by warranty engineers (external, internal inspection for compliance with the configuration), cleaned, checked for the stated problem. If the repair by an engineer could be done on site (resetting the BIOS, replacing the battery, removing the cable, removing a lot of dust, reinstalling the OS, testing the memory and HDD without errors), then the PC of the health care facility was put into operation if the replacement of functional parts was necessary (power supply, motherboard, HDD, memory strip, etc.), then the PC of the health care facility was sent for warranty repair to a specialized SC, which worked directly with the manufacturer of this equipment.
7. In the specialized SC, the non-working functional parts were already replaced with new ones, and then the PC of the health care facility returned to our warranty department.
8. The person responsible for the ticket received the Health Unit's PC back from the SC, checked its operation and initiated the issue back to the Health Facility.
9. Ticket closed after receiving a PC health facility in the health facility.
Complicated? Why do we need the first line of such technical support?
According to the project contract. Moreover, at this stage in the first line up to about a quarter of all tickets can be closed without the involvement of a specialized SC.
Some hospitals immediately worked directly with the SC, bypassing the first line of support.
What the owners of the personal computer of the hospital did in the hospital: they changed the HDD for others, inserted DVD-ROMs, changed the memory bars, turned off the fans and the dust sensor, and so on. All this was recorded in the ticket and for some tickets even the guarantee was denied, and even the SC could provide an official act on such a problem.
So, it was and is in operation 3740 PCs of health facilities - physically (approximately) about 500 PCs of health care facilities during these 3 years came to the guarantee. Those. 5% per year from about the total number of PCs.
They carried out maintenance work - cleaning (although it was not the responsibility of the engineers), checking cable connections, and diagnosing a faulty unit.
Further, the PC was sent to the service center (SC) to replace the faulty unit or to the health care facility back if the PC worked regularly after testing.
What changed in the SC (descending): HDD, power supplies, motherboards, memory bars, sat buttons and cables, only 2 times the processor, and that with the motherboard together.
A small potpourri of faults.
Faulty memory bar:
Blue screen when loading:
A dust sensor is installed at the bottom of the case:
A special StopDust software is installed for the sensor.
Only most of the PC health facilities at the first opening of the case on the fans give this real picture:
Problems with HDD:
Errors in SMART (it is clear that such a PC health care facility was able to work 3569 hours or 446 working days):
The problems that even we had to put up with:
1. A good SC of this manufacturer is very difficult to pick up - first they worked with one SC, but they took everything for a long time and gave it away from the repair - they said that there were problems with the supply and with the manufacturer ICL.
Then they worked with another SC, engineers work there promptly and the period of rotation of the PC health care facilities became less critical.
2. Problems with the health facility PC - Many health care center PCs in the health facility simply do not start from scratch, even if they were left in a warehouse at the health facility, etc., because the entire PC fleet was not immediately turned on and put into operation at the health facility. And how they are stored in warehouses in hospitals sometimes ... This is a separate song.
3. The built-in dust sensors in the PC of the health care facility do not particularly help, as the engineers of the health care facility can conduct maintenance, but this is not their responsibility, so this sensor must be used as it were, but its data is not particularly used.
4. Information, training and constant communication with health care facilities on problematic issues - observe repair time, if a PC is delivered for 300 km, clarify all aspects of the situation with documents and acts of acceptance, control and inform all involved persons by ticket. Especially, when a local engineer was changing at the hospital, and the new one was not aware of the situation at all, it all started in a new way.
And now, for sweets, some applications from health facilities (complaints to the health care facilities, some even deserve a bash!):
- “Sat does not turn on (turns on for 3 seconds and turns off)”;
- “the system does not start, when trying to reinstall, it displays a message that the hard disk will soon fail and is not installed”;
- "The Security Council does not turn on, the smell of burning";
- "The Security Council does not turn on, when you try to turn on, a crackling sound is heard";
- “The SAT turns on, but there is no power to the mouse and there is no video signal to the monitor”;
- “after switching on after a certain period of time it turns off or“ freezes ””;
- "does not turn on, but when it turns on it makes noise";
- “falls in the BSOD indicating the damage to the RAM”;
- “it slows down in the process of loading and freezes during operation, a blue screen happens”;
- “is overloaded after a short work”;
- “does not turn on (turns on for 3 seconds and turns off) the similar problem was last time, before the occurrence of such a problem another one appeared, the image swam, sinned first on the monitor and checked, now I sin on the video card”;
- “Our Security Council broke down, it DOES NOT INCLUDE”;
“The monitor is out of order, its malfunction has appeared, the image is twitching. And one more system unit, the cooler on the processor slows down and does not work ”;
- “There is a malfunction of the power supply on the computer. Apparently the power supply unit gives out voltage to the device case ";
“The system unit has broken, the download fails, it does not even reach the BIOS loading”;
- “the reboot is constant, it does not even reach the BIOS. Will go to the service for the fifth time ";
- “hard drive failed. ("Pours") ";
“First, the USB outputs on the front panel disappeared, then something with a video input, it does not connect with any monitor, even downloads are not visible”;
- “The system unit turned off during operation, the smell of burning PCB appeared. Our assumptions, short circuit in BP ";
- "reboots (without reaching the load of the operating system)";
- “we have another warranty computer, with a floating error. Video card artifacts that manifest themselves day after day ”;
- “! Vindous reported a failure (inevitable equipment failure) of hdd at this ARM ”;
- “PC does not start. No signal to monitor. It turns itself off. ";
- "Periodically, the PC does not turn on until you disconnect from 220V";
“The SB has failed, it does not load, and judging by a number of signs, the reason for this behavior in the motherboard”;
- “there was a problem while setting up the new SAT. After I put him on the domain, he stopped seeing me as an administrator and rejects all passwords, writes that he needs a raise ”;
- “when working with SAT, a knock is heard, which leads to a hang”;
- “It turns on only after entering the BIOS, the language in the BIOS is not switched, the indicators on the front panel of HDD_LED and Power_Led are confused”;
- Does not turn on the PC serial number: 1212, symptoms, as in the previous time.
Eventually.
Most of the PCs of health care facilities work, either as a standard without repair, or after maintenance and service.
But the backup data of the LPU PC in the LPU is basically not done (although the main data on the servers must also be in cloud solutions), so after replacing the HDD, the SC engineers had to do disk cloning before repair (if possible) or in the LPU empty PC health care facility, but working.
Technically, HDDs on PCs were needed only to run the OS and simple operation, although a special was installed there. Software to protect data. The main work with data in RMIS through the cloud, server and browser.
In general, the places of use of PC health facilities are very different, as it turned out, the treatment of users with them in each health care facility was also different.
And how happy the rebuilt PC, which is ready for further work, is:
In fact, the guarantee support for this “park” ended in 2016, now the components of the IT infrastructure are being updated at the health facilities and the introduction of more modern solutions.
Although, part of this park continues to work in 2018, although it is already supported by local system administrators.
Source: https://habr.com/ru/post/428546/
All Articles