Network duel

Introduction

I want to tell you an almost detective story about a network duel. I apologize for the long introduction, it seemed to me that without him it would not be clear why this all happened.
One side I did was administering about 10 years ago, in an organization where there were about 150 computers and then everything was in Windows (2000). Later, I left the administration, and although now at work I perform the functions of an administrator (also with Windows), but more in support so that it does not fall than in moving forward. In my life I turned away from Microsoft and now I use Ubuntu Linux. Ubuntu and at the user level, because it is far from being a student - you need to work on the computer, there is not enough time for diving into the depths with all the desire.
I will talk about the other side later, but for now around the battlefield.

A year ago, my half got a job as an IT manager in one of the many hotels in our resort town. At that moment, two guys worked there (although one of them soon left) and I was confused by the association with IT Crowd. Although it is just a laugh - in fact, she understands the hotel automation software, and in 2003 she successfully passed a couple of MS administration exams. Nevertheless, she has no great experience in administering any kind of male attitude to this.

In the fall, the second one decided to leave the company and went to Moscow to continue professional growth. He left behind a workstation with Gentoo (it passed half of the inheritance), a server with Gentoo and FreeBSD, a Windows domain (with two DCs), one Cisco 1810 and promised to support remotely as far as possible. (A terrible wish to the director of the company: “Yes, so that your Gentushnik will quit your job!”).

The workstation after pulling out the personal video card, and maybe something else did not want to load, and then I was at home working with it, I got acquainted with Gentoo and eventually put it in order, although after a while I forgot everything. I myself really do not like to do half the work, I keep it as much as I can. Only when the house is brought is, well, nowhere to go.
')
Gradually, the company's relationship with the gentushnik disappeared, instead of it, a young guy began to work, who had previously worked as a safety net and part-time job, conventionally Vova. Over time, Vova began to somehow cope with the administration of one, and recently came another young guy.

Aggravation

In each company there are some features that old-timers get used to, and the young and energetic want to win, but more often give way to the next. In this company it is a regular recurring ban of their IP addresses by hosting their own site. From time to time, half approached me with a question - what to do, and left with an answer - see the logs of your squid. What for some reason could not be done: the ban was removed / the log was not at the right time, etc ...

On Monday this week, half again wanted to see the logs, and it turned out that they could not access the Gentoo server, where squid works. In this case, Vova went somewhere for 5 days and promised to be available. Half come from work and share experiences - I say, well, call, find out. Hours at 8 pm calls, reports the problem. Vova calls back at half past nine, says - and I can go in, with this and such password. Half tries - goes into the tunnel, then comes to the right server - everything turns out and the password is the same that she typed. I didn’t pay much attention to it, I wondered if I didn’t switch the layout or I was wrong in the name. On Tuesday he goes to work - he tries the same operation there - he fails to connect, complains again. I say, okay, figure it out - anything happens, maybe there is a verification through Kerberos, and it breaks.
Wednesday is again a question for me how to get to that server. In parallel with working questions in the background on ICQ, I say, well, look for the Gentoo server bucket, look at the configuration in it, the / etc / shadow file (in the weak hope that there was a bucket, it became at night and contains more or less relevant data and it will show the date of the shadow change or you can see if Kerberos is really configured there). At first, it looks like a bykap is not located, and then all the same hurray, we found it - on the server with FreeBSD. So far, they thought what to do with it / copy / not copy and how to look - its connection to FreeBSD falls off and it is impossible to enter anymore, the password does not fit. I say, well, call Vova, ask for a password, let him try, adjust. Vova is not available. Well, be patient, you will understand later. At the same time, sluggishly answering other questions - why does my Security log on the domain controller clear out? Well, I don’t want to dream and have no time, so the traditional answer is "understand, under what conditions it happens, and then we will analyze ...". Well, already some of her vague doubts began to torment, maybe it's Vova?

I say - well, if in doubt, deny just access to the internal network until he arrives, maybe his password has gone somewhere and someone in his place is with you now. She commissioned to do this to a new guy and went home. VPN access, which was configured on Cisco, was limited through the WebVPN group in AD. The guy carefully takes screenshots of the composition of this group before and after the changes and sends half to the mail at 17:55.
Then housework, and at 10 o'clock in the evening, though I don’t want to do it all, but I think, well, now I’ll quickly help with my business. We try to connect and look again, what is there now with FreeBSD, why now it’s not possible to go there. And lo and behold, now it’s not possible to enter the VPN. At the same time, I am sure that everything is fine with the configuration of the VPN client, nothing has changed in it. Somehow there was a feeling that this is simply not the wrong setting to explain, and someone was offended by the fact that they restricted the entrance to the VPN. At the same time, he himself did not suffer from this in any way - he still walked in just as easily, just apparently he had to get a spare loginopole from somewhere, and this was hurt. Half call Vova - he still does not respond. He calls a new guy, asks to try - and he has the same nonsense.

Fighting

I say - well, it smells bad, you have to go to the site and understand. Gathered - I found an old compact with Ubuntu and took a flash drive, went there. While we were driving, some kind of plan was forming in my head, but in general it was somehow scary to go and do something quickly in an unknown network for me, and even with such an OS zoo. The first thought is to turn off the Internet, but if I turn it off, I can only count on myself and I’ll not solve those problems I have not encountered. For example, I didn’t deal with Cisco — I quickly found how to reset the password before I left it, wrote it on a USB flash drive, but I understood that this alone would not help, I would have to change its configuration. But if the Internet is not turned off and show at least some activity, he will notice and can go for anything. Therefore, the first is to turn off.

We go into her office, she immediately behind the computer - I'll see. I say - do not touch anything, do not give yourself away. I managed only to move the mouse. We went straight to the server room, there is a rack there, a lot of things crammed into it. For a network with 50 computers, even too much. When I administered the network from 150 I didn’t allow myself to unwind the manual so much. Some kind of network KVM (I did not see such, I understood the essence only by connections), and UPS with signals via Ethernet, something else interesting. The camera stands, writes everything that happens. As half said later, she writes with sound. I understand wires - it turned out that both optics and Cisco are part of a radio network - a backup channel. That is, it seems like two channels, and if Cisco falls, there will be no Internet. I ask - is there somewhere else 3G-modems? There are, but without simok. Well, I mean again without the Internet, well, and it seems only these two paths lead to the network. After a minute of hesitation, after all, the volitional decision — to put Cisco in the safest state — off. Half of them became very nervous “what if”, and I have nothing to lose, the work is not mine. It was already about 23:30.

Then I insert the CD with Ubuntu into the “Gentoo” server, it does not boot - you need to change the boot parameters in the BIOS. Hurray, there is no password on the BIOS, otherwise I was horrified to take the server out of the rack, and then drive it out of it or look for a jumper in it. Ubuntu is loaded, I stand and laugh - someone will watch the video, as I’ve gotten into the coochacker's lair with the most Dummy’s Linux, I’m stuck.

Well, nevertheless, the system booted up, I saw the RAID (again, hooray), the flash drive was working (not hooray that USB 1.1, well, okay). I look at / etc / shadow - well, so far nothing suspicious, the date does not confuse - until the moment when half could not enter. The content is of course difficult to verify, I start copying / etc to the USB flash drive and study further. In / var / log, too, nothing catches the eye, some files are empty (like sshd.log), which is strange, but you never know - the log can be disabled. Archiving more logs for the history and directory / root. I find one interesting thing - the IP-address is visible in the lastlog file: 87.245.163.145. There is no Internet, check and there is no possibility to compare with something, intrigue remains. Then I go to the Windows domain controller, look, and what about the groups and users. The first is the Security magazine - and a surprise: the last event was “cleaned up at 00:32” by the user, by the way that I entered. The next event is right behind it - Process Tracking Audit, “c: \ windows \ system32 \ wscript.exe is shutting down”. In fact, it was only at that moment that I found confidence that, after all, these were not computer failures or adjustment errors. Before that, one could doubt and think, well, maybe some kind of misunderstanding, misunderstanding.

In Windows, I have more experience (this is how it happened), I quickly check Startup, the registry, group policies (domain, domain controllers), I finally find usrlogon.vbs in NETLOGON, and its launch is assigned simply in the user's profile as favorites. It could probably be all put, but apparently too lazy. The script itself is also clean-free - the security log is cleared through WMI. The creation date of the script is tonight, 21 with something. Rename it immediately away from sin. I look through net user login time of administrative users. And it turns out that here the admin and went to 23 with something. That is, it was on the network at the moment when we came and chopped off the Internet. Another user with admin rights also logged on in the evening when there was no one at work.

I start searching for files that have been modified in the last 24 hours - I find c: \ program files (x86) \ qip \ <UIN of my half> \ config.ini. That is, it is clear that he read our logs with her and therefore could promptly disable her sessions and servers. For example, in the process of such a dialogue:

[09:59] : xxx 20 00:36
[10:00] : ?
[10:00] : yyy- 131
[10:13] : xxx?
[11:21] :
[11:21] : ' -

I simply did not pay attention to this. Perhaps, on the last day, he already wanted to write something and write to me himself, since he reached out in config.ini. There were even less interesting finds in files, such as in “Recent Documents” test.wsh - someone was probably training, working on something else. To search for interesting later, launched the archiving of profiles of important users.

Time is ticking away, and on such a late night lark thinks badly and wants to sleep. Therefore, I already wanted to bring it to a certain point. It was really easier to disable the screws on the Gentoo server to tighten the nuts in the domain on the Gentoo server, and it seemed easier to disable FreeBSD where the bycaps were located. With Windows, it's easier for me again - to filter the Administrators groups, change the password to the remaining admins.

Returning to the Gentoo server - right there from Ubuntu I decided to change / etc / shadow, remove the passwords in it in order to install my own after reboot. I look through the list of users more attentively - I see aimsniff - beauty! locate aimsniff, I find a config in which a filter is set, on which ports to intercept messages and where to write (in MySQL database). Yes, there is such a base, the dates are fresh. I stop the aimsniff, remove from the launch.
Then I am overloading the system in the normal mode, in Gentoo - it successfully loads - hurray, nothing is broken. I reset root / toor passwords, disconnect any other suspicious ones.

Everything seems to end, about 4 in the morning. There was a philosophical question - whether to turn on cisco so that someone could go inside, but people could use the Internet from the inside, or not turn on the channels and switch in the morning, etc. Without knowing the opponent, it is difficult to predict his behavior, but the smart one would probably understand that I still found a scythe on a stone, I would have hidden it, and if I went further to cisco, then quietly. I made a bet on it - they turned on cisco, checked on the workplace - the Internet works, as it worked - nothing has changed. With a sense of satisfaction retired. In the morning I didn’t go there, I had to do my job, but it turned out that the calculation was wrong - he apparently didn’t believe something and continued active work. The Internet has stopped working. And the young guy who continued to put the network in order today, said that while he was managing something through the network KVM, someone was connecting to him. I said that now you can’t use network KVM now, you need to disconnect and switch to normal ones or just physically connect what you need. Later in the afternoon the Internet was already connected without cisco, and time will tell what will happen next.

But probably the detective story should not end without revealing. Probably the laziest people have already seen what IP I brought, it refers to the network:

Owner Name: Network for Kabeljnyy Internet Krasnoznamenska
From IP: 87.245.163.128
To IP: 87.245.163.159
Address: Russia, Moscow region, Krasnoznamensk, ul. Krasnokazarmennaya, d. 23

In /root/.history (which he didn’t have time to clear, didn’t expect us to go to the site at a late hour) I saw “ping moneybox.sktv.ru”, and sktv.ru is again Krasnoznamensk. According to the presence of the “:> lastlog” commands, in the same place it can be assumed that the IP address is real, and not some intermediate one that you should not hide.
What role did Vova play in this and whether he contacted this former administrator (conditionally Roma), when it was necessary to change the password, you will probably still know.

Some details of this story I wrote for those who can get into a similar situation, suddenly it is useful. I understand perfectly well that the reverse side can also make stronger moves, but physical access is physical access and, if we act carefully, it is possible to win. Like this.