The crash of the satellite Express-MD1 (July 4, 2013)

"Proton" with Express MD1 and Express AM44 on board

Our satellite network experienced an uncounted number of minor failures and satellite anomalies. Express-2 (correction engine failure), Express-AM11 (depressurization), KazSat-1 (control system failure), Express-AM2 (failure of the solar cell turning system), NSS-703 (full development of the “working body” of exact correction engines) ... And yes, Express MD1.

It was a normal working day on July 4, 2013. Routine: we looked at the current state of the network and channels, worked with the plans for the next extensions and changes, solved some minor problems (something is happening on a large network). As time went by for dinner, I decided to go pour some tea. And then out of the corner of my eye I saw a bunch of fallen out accidents on the screen of the monitoring system.
')
The simultaneous occurrence of a large number of accidents or accidents in many areas may well be, for example, a shower at one of the central stations. But here I see that the canals collapsed at the same time, landing both in Ulan-Ude and in Vladivostok.

All carriers first sagged, then almost recovered, and then very quickly disappeared completely. This is a very characteristic picture for the case when the satellite loses its orientation. Therefore, dialing the number of the RSCC on-duty service, I was already preparing to hear the worst.

A little about the network

Before continuing the story, it is worth explaining that satellites are hand-made products made from virtually manual, having a reservation of all vital systems and worth a lot of money. Nevertheless, everything that can break will break.
All accidents and problems with satellites can be very conditionally divided into two parts: disposable (but which, nevertheless, can lead to partial loss of working capacity or functionality) and ineradicable ones - those that lead or can lead to complete loss of working capacity (up to loss of or the death of the entire apparatus).

A typical disposable accident is a failure of a unit or system for which a regular reserve is provided. It is clear that the backup options are not limitless. And if, for example, the main and backup transmitters consistently fail, the trunks served by these transmitters will stop working. But still this is only a partial loss of functionality, because the remaining trunks, working through other transmitters, will continue normal operation.

The most common accidents that lead (or may lead) to total failure are problems with engines and orientation systems, problems with the power supply system, and problems with the satellite control system. And the hallmark of almost all such accidents is the "twisting of the satellite." In normal condition, the satellite is directed exactly to Earth. Position tracking is the task of the stabilization system, which determines the position of the satellite relative to the Earth and the stars. The same system controls the gyroscopes, which, in fact, ensure the constant stabilization of the satellite in three planes. Imagine now that the food "flashed" briefly on board. What happens next? That's right, the gyros will start to slow down. And the satellite at the same time, according to the third law of Newton, will begin to turn, and almost unpredictable. And this in itself is almost a catastrophe: the “command link” is lost (the antenna turns away from the Earth), the power supply from the solar batteries stops (they constantly monitor the Sun) ...

It is clear that when "turning" all services also stop working. And not even because the antennas are now not looking where they need to, but because in such emergency situations the satellite automatics first of all disconnects the entire payload, trying to save battery power as much as possible.

Again July 4, 2013

RSCC confirmed the existence of problems with the satellite, but there was no specifics (official confirmation of the accident and its alleged causes were announced only three days later). According to unofficial channels, my guess was confirmed about the loss of satellite orientation.

But no one has yet had any information about the prospects for restoring control over the satellite: too little time has passed, and it will take at least several hours to assess the causes, possible consequences and plan of possible actions.

We began to slowly collect all the information in one place: stations and their channels, equipment at these stations and its capabilities, the ability of antennas to turn from position 80 to other positions, possible touchdown points, availability of necessary equipment on them, possible transition scenarios, lane and energy required to turn on the Express-MD1 channels ... Fortunately, we had only one transponder on this satellite, however, there were 12 satellite communications earth stations that landed on two central stations: Vladivostok and Ulan- de. And we had to figure out where and how to stuff these channels into the capacity we have on the other sides.
We have a fairly large satellite network. And the capacity for it, we rent not "per channel", and take from the owners of the boards rather large pieces. Therefore, almost always we have a certain amount of capacity: the band, which is currently not occupied by channels. But firstly, such “surplus” is not so much (dear!), And secondly, almost always this free capacity is not a single unit, but pieces of strips between working carriers. A typical transponder download picture looks like this:



Each stick is one carrier (or one channel if modems with a correlation multiplex work on the line), whose width (and height) depend on the speed of the channel, the modulation used and the codec, the size of the antennas. Is it possible to shove this trunk, say, a third of what is already there? Well, if you really try ... But at least for this you need to collect all the voids in one place. Further, if we have several trunks on this satellite, and there are at least some empty space in other trunks, it is possible to transfer several carriers / channels from the trunk to the trunk (again collecting all the empty space into pieces for use). But it is possible to shuffle the channels along the trunks, if this allows the equipment of both JSSs to be made - here, too, there are limitations.

And if it does not, then maybe there is another station that looks at the same satellite, which has no such restrictions, and where can you land the channel (with its subsequent return on the ground)? Or maybe a little more band can be won in this or in the neighboring trunk, due to the “band exchange for energy”? Or maybe there is a way to quickly lead one ZSSS with a large number of channels - from this satellite to some other, and in its place drag the victims? And if it is possible, what kind, how and where to land, whether the necessary equipment will be found at the new site, whether there will be a capacity on the ground to turn on / return services ... All these options needed to be invented, sorted, analyzed and calculated, thought out all multi-moves like if we do this, it will turn out like this, ”and“ if it doesn’t work out like this, then like this, it’s like that ”. In short, we had something to do.

It took us more than a day of almost uninterrupted work on all these “tag games”. And finally, by the evening of July 5, we had the first version of the plan of salvation. According to this plan, the main part of the channels was diverted to Express-AM33 (with the UFD Ulan-Ude turnaround from MD1 to AM33). At one of the points there were two antennas (C / Express-MD1 and Ku / Yamal-301), this direction was diverted to Yamal with simultaneous re-grounding from Ulan-Ude to Irkutsk. Another station was diverted to the NSS-9 with re-landing from Vladivostok to Khabarovsk. And one more was disconnected completely (the traffic was transferred to a very timely-coming relay, the commissioning of which was expected just a few days ago). At the same time, at several stations it was necessary to change the modems in order to work in the correlation compaction mode. At the same time, a test turn was made by Ulan-Ude on Express-AM33. It was necessary to make sure that at least “in the center” we have a turn and further work on Express-AM33 will pass without surprises and carefully “unfolded rakes”.

On the 6th, PACS launched Emergency Express AM2, which was stationed at the same point. This was done primarily for the salvation of TV broadcasting. Express AM2 itself was disconnected at one time due to problems with solar panels (yes, the one with which we also had to escape). Therefore, it turned on for a very short time and at a lower power. Nevertheless, it was clearly better than the complete lack of communication, so we returned back Ulan-Ude for a while. And they began to prepare the unused Moscow antenna at that time, in order to ensure the temporary landing of the ZSSS channels remaining at the 80E position (Express-AM22), after the Ulan-Ude ZSSS will change over to the Express-AM33. Well, yes, the best option is to turn all ZSSS at once. But you have to somehow get to them ...

July 8: a bit of math and roulette

By July 8, the development, preparation and assembly of ground connections for re-grounded channels was completed, the available human resources were recalculated (everyone who saw the satellite dish was put under the gun), their routes were determined (for Siberia, where half The points are rotational settlements, this is very important). People ran for the tickets. The longest trip (taking into account the return time) was two weeks. At the same time, work was done on clearing and releasing the band on the satellites to which the stations had to be transferred: assembling all voids into one piece, transferring from trunk to barrel, changing channel parameters, changing frequencies — and again assembling voids into one piece. Many iterations - only because all the permutations, movements and reconfigurations had to be done with minimal interruptions in the work of roaming channels. Clients using these channels are certainly not very interested in the fact that a nearby satellite was killed.

And most importantly - plans and instructions. Search for a satellite, onboarding the ZSSS on board and reconfiguring the equipment are simple but still specific tasks. And they require not only knowledge and experience, but also instruments and tools. In fact, it turned out that a number of stations will have to be turned by people who have something, but not enough (there is experience, but no instruments, or vice versa). Unbearable task? Not at all: in this case, it was very helpful that the ZSSS were already set up, only to another board. That is, it was not necessary to fumble around the sky in search of a satellite, it was only necessary to very accurately shift the antenna to the required number of degrees (well, and then tune out more precisely, under our guidance and on the working channel). You won’t believe it, but for an accurate “bias” even a relatively large antenna using this technique, the usual roulette wheel is quite enough! Well, clear instructions: what and how to measure, where to put the risks, how to turn the antenna on these risks ... Everything is important here, right down to the order of release and tightening the nuts. For each ZSSS, its own manual was prepared for the type of antenna used and the installed equipment: with pictures / photos, offset signs, target designations (at least for checking or “if something goes wrong”), a label of changeable configuration parameters, etc.

July 9–15: recovery

The last channel was restored on July 15th. A lot of writings, a lot of nerves, by the end of the day, the throat is already tearing away from constant telephone conversations. And to some people - all night on the plane, from the plane to the helicopter, then to the bus station, then blindly adjust the antenna - and again to the bus station, the helicopter, to another station ...

PS "I told you: damn place"

A little mysticism. The point of 80 degrees east is definitely unhappy for Russia. Judge for yourself: the last satellite that worked in this orbital position from and to (and even much more) without accidents and adventures is the Soviet Horizon. So what is next:

• Express-2, launched at this point to replace Horizon, is the failure of exact correction engines, the impossibility of stabilizing a satellite in orbit. The satellite itself has worked for a very long time in inclined orbit mode (ZSSS, working through such satellites, must necessarily have an auto-tracking system behind the satellite);
• Express-AM2, which came to replace Express-2, is the failure of the control mechanism of solar batteries, the inability of the satellite to operate around the clock under load. Emergency replacement for the “small satellite” Express MD1;
• the planned replacement of the Express MD1 should have been made a full-fledged heavy satellite of the Express AM4. Failure of the accelerating unit, the threat of a collision with Express-MD1 (then still working), loss of the satellite;
• after the unsuccessful withdrawal of Express-AM4, Express-AM4P was launched after him. The explosion of the 3rd stage of the launch vehicle at launch.
• And finally, Express MD1 - loss of orientation, termination of work.