Once again, the benefits of traffic analyzers
This week, a very informative article on the use of a microscope for a high-quality fiber-optic connection has been published. She reminded me of the number of tools in each specialty, which, despite the obvious, seemingly usefulness, are far from being used by everyone. In system administration, one such tool is a traffic analyzer . Under the cut is a typical story illustrating its usefulness (experienced specialists will not find anything new there - the story is designed for beginners).
Let's start, according to tradition, from afar. The customer decided to upgrade their Microsoft Active Directory forest and domain from 2008 to 2012 R2. In fact, the need was only to upgrade to 2008 R2, but given the complexity of such projects in a large environment (and the customer had more than 1000 Windows servers alone in dozens of geographically distributed locations), Service Owner decided to switch immediately to 2012 R2 . Moreover, the actual server build at that time was already Windows Server 2012 R2.
In order to increase the functional level, you must first migrate all domain controllers to the new OS. The process is quite simple from the point of view of Windows. All difficulties arise in those locations where integration of something third-party with the Active Directory environment is implemented. That is almost everywhere :)
Listing all the problems of that migration is material for several articles. Now, we are only interested in one medium-sized location - two controllers, one thousand users, two EMC Celerra NAS devices (of course, hundreds of servers, databases and applications, but we will not talk about them). In addition to shared resources, NAS were used to store user profiles. When there are two controllers in one site, the migration process is significantly simplified - we can migrate one controller and, if something went wrong, it can always be extinguished - the second one remains (it is important to note that by this time it has already passed Migration of several locations and no particular problems were expected).
')
So, day X has arrived and one of the controllers has been removed from the domain. We rearranged the OS and re-elevated the role on it. Immediately it became clear that this time there were no problems. Users who received a new controller as a Logon Server have lost access to their profiles and shared folders. Instead, they saw a sad message:
We extinguished the problem controller, created a separate artificial site for it and added its IP address with a c / 32 mask, transferred one test client and started testing (yes, you could start with this, but to save time and because of low risks, Service Owner allowed enable the controller immediately in the live site after the end of the working day). Recently there was an article about full-stack administrators . This is, without a doubt, very cool, if you have the knowledge and rights on all devices to solve the problem yourself. Most often, the company has a fairly rigid division of teams into areas of responsibility and you technically cannot check the NAS settings while working in the Active Directory support team. It is clear that once a problem appeared after changing your infrastructure component, then problems, by default, are on your side. How to find the cause of your troubles and get arguments for a request for some action from the other team?
An invaluable tool will be traffic analyzer. Here I am a little cunning - one of the important differences between Windows 2008 and Windows 2012 R2 is the new version of SMB protocol , so I guessed what the problem would be. My favorite tool in such cases is Wireshark (do not count for advertising). Quick install, launch capture, attempt to access the shared folder and what do we see with the packet exchange logs?
Ioctl Response, Error: STATUS_INVALID_DEVICE_REQUEST shows us that an SMB session between the user and the NAS device is not established. Considering that everything works with the old controller, I received confirmation of my guess - a problem in the new version of SMB. In general, NAS devices in the customer's environment should support the new version of SMB (in other locations, everything was fine), so the next idea was to search whether it was necessary to update the firmware on them. Bingo! The vendor forum confirms that the old Celerra firmware version does not support the updated SMB. The information is sent to the NAS support team along with packet exchange logs, links to the vendor's site and a request to update the firmware. The following weekend, the firmware is updated and the tests confirm - now everything works.
As an afterword. When I recommend my friends to use a traffic analyzer to study a problem, the most frequent reason why a person does not want to do this is to think that it is very difficult. This is not true! In most cases, in order to understand what is happening, it’s enough to look at the packet exchange log and sometimes read the KB article about how the protocol you are interested in is arranged. It is very simple. And it can save you a lot of time.
Let's start, according to tradition, from afar. The customer decided to upgrade their Microsoft Active Directory forest and domain from 2008 to 2012 R2. In fact, the need was only to upgrade to 2008 R2, but given the complexity of such projects in a large environment (and the customer had more than 1000 Windows servers alone in dozens of geographically distributed locations), Service Owner decided to switch immediately to 2012 R2 . Moreover, the actual server build at that time was already Windows Server 2012 R2.
In order to increase the functional level, you must first migrate all domain controllers to the new OS. The process is quite simple from the point of view of Windows. All difficulties arise in those locations where integration of something third-party with the Active Directory environment is implemented. That is almost everywhere :)
Listing all the problems of that migration is material for several articles. Now, we are only interested in one medium-sized location - two controllers, one thousand users, two EMC Celerra NAS devices (of course, hundreds of servers, databases and applications, but we will not talk about them). In addition to shared resources, NAS were used to store user profiles. When there are two controllers in one site, the migration process is significantly simplified - we can migrate one controller and, if something went wrong, it can always be extinguished - the second one remains (it is important to note that by this time it has already passed Migration of several locations and no particular problems were expected).
')
So, day X has arrived and one of the controllers has been removed from the domain. We rearranged the OS and re-elevated the role on it. Immediately it became clear that this time there were no problems. Users who received a new controller as a Logon Server have lost access to their profiles and shared folders. Instead, they saw a sad message:
We extinguished the problem controller, created a separate artificial site for it and added its IP address with a c / 32 mask, transferred one test client and started testing (yes, you could start with this, but to save time and because of low risks, Service Owner allowed enable the controller immediately in the live site after the end of the working day). Recently there was an article about full-stack administrators . This is, without a doubt, very cool, if you have the knowledge and rights on all devices to solve the problem yourself. Most often, the company has a fairly rigid division of teams into areas of responsibility and you technically cannot check the NAS settings while working in the Active Directory support team. It is clear that once a problem appeared after changing your infrastructure component, then problems, by default, are on your side. How to find the cause of your troubles and get arguments for a request for some action from the other team?
An invaluable tool will be traffic analyzer. Here I am a little cunning - one of the important differences between Windows 2008 and Windows 2012 R2 is the new version of SMB protocol , so I guessed what the problem would be. My favorite tool in such cases is Wireshark (do not count for advertising). Quick install, launch capture, attempt to access the shared folder and what do we see with the packet exchange logs?
NegotiateProtocol Request
NegotiateProtocol Response
SessionSetup Request
SessionSetup Response
TreeConnect Request Tree:
TreeConnect Response
Ioctl Request
Ioctl Response, Error: STATUS_INVALID_DEVICE_REQUEST
Ioctl Response, Error: STATUS_INVALID_DEVICE_REQUEST shows us that an SMB session between the user and the NAS device is not established. Considering that everything works with the old controller, I received confirmation of my guess - a problem in the new version of SMB. In general, NAS devices in the customer's environment should support the new version of SMB (in other locations, everything was fine), so the next idea was to search whether it was necessary to update the firmware on them. Bingo! The vendor forum confirms that the old Celerra firmware version does not support the updated SMB. The information is sent to the NAS support team along with packet exchange logs, links to the vendor's site and a request to update the firmware. The following weekend, the firmware is updated and the tests confirm - now everything works.
As an afterword. When I recommend my friends to use a traffic analyzer to study a problem, the most frequent reason why a person does not want to do this is to think that it is very difficult. This is not true! In most cases, in order to understand what is happening, it’s enough to look at the packet exchange log and sometimes read the KB article about how the protocol you are interested in is arranged. It is very simple. And it can save you a lot of time.
Source: https://habr.com/ru/post/280280/
All Articles