Once again about the benefits of backups or the story of my failure
It is always nice to remember how we managed to solve some non-trivial task. But it happens the other way around. The task was seemingly understandable, but it could not be solved. Below I will tell a sad story from my practice. She is sad because I still do not know how to solve this problem. The story, once again, will remind you how important it is to pay due attention to backing up and carefully considering your actions. Well, if someone from the readers tells me what could have been done to achieve a better result, then it will be just great.
So, there was a system administrator and he had an Active Directory domain. Since the infrastructure was small and inherited a long time ago, there was only one domain controller with Windows Server 2003 installed on it. There were not enough resources, so a rather large number of applications were used on the same server that the administrator himself used and some of his colleagues. Well, the cherry on the cake - backups in this company has never been done. What could go wrong? Details under the cut.
Nothing can stay the same forever and the technical management has decided that it is time to switch to a more recent version of Active Directory, which means you need to update the domain controller. How to switch to a more recent version of Windows Server in your domain, if you have only one controller, on which there is a lot of additional software and for which there are no backups? Of course - in-place upgrade! During the update, something went wrong and the process was interrupted with an error. After that, errors from various systems and applications fell down, and our hero, realizing that the old controller was in trouble, decided to correct the situation by adding a new controller (it’s a pity that he didn’t start with this). A new server with Windows Server 2008 was raised and he tried to enter it into the domain in order to make it a controller. Having received the next error message, the administrator realized that it was time to seek outside help.
This is where our acquaintance with him begins. A small remark - in the process of finding a way to help him, several people participated, but in order to simplify the narration, I will not focus on this attention, especially since it will not add any technical details.
')
For ethical reasons, I cannot show screenshots from someone else’s work environment. Therefore, to write the story, I (now already knowing what had broken down) reproduced the situation in the test lab. For those who wish, at the end of the article will be told how you can do the same if you want to try your hand at solving this problem. Thus, our patients will be the domain Test.local , Windows Server 2003 controller TESTDC , Windows Server 2008 server TESTNEWDC .
When trying to enter a new server into the domain, the following error was issued:
As usual, problems with Active Directory start with DNS. Looking into the DNS management snap-in at TESTDC we saw an empty server with no zones. So obviously it should not be, since this is the only controller, in combination was the only DNS server.
So, task number one is to restore DNS functionality. Without it, a working domain is out of the question. Just in case, they specified at the customer that all zones were Active Directory integrated. So finding their files on the server itself will not succeed. DNS data must be loaded from the Active Directory partitions of the database. But it seems that this was a problem. Looked into the System and Application logs to see what happens when the DNS service starts. So it is - in the logs were filled with errors Event ID 4000 and Event ID 4007 , characteristic of problems with the operation of AD integrated zone:
It was a very unpleasant sign, since the impossibility of downloading information from the Active Directory database of the only available controller hinted at the severity of the incident. However, there was hope that it would turn out to make it all work with your hands. When trying to create a zone again, the idea was confirmed that the DNS could not get anything from Active Directory. He could not even get a list of domain controllers on which the relevant sections are stored:
As a workaround, it was decided to try to replace the native DNS zone with a local stub. It is clear that it will not contain all the necessary information, but the controller can be forced to update the DNS records for Active Directory, and everything else could wait for now. So, the local main zone test.local was created and dynamic updates were enabled in it. The easiest way that Microsoft offers for registering DNS records of a domain controller is to restart NetLogon service on it. This was done. As a result, a local zone was obtained with the necessary entries:
Here we took a pause. The customer checked the functionality of applications and systems in their environment. Everything worked. users can use their accounts regularly. The customer was slightly relieved from the heart - at least people could get back to work.
But the main problem was not solved. We are back to trying to enter a new server in the domain. For the introduction to the domain, the domain administrator’s test.local \ administrator account was used, which successfully logged into the old controller. Error again. True, this time, the other. Now "Login Failure: The target account name is incorrect".
Remembering the problem of downloading information from Active Directory, here we had an idea to try instead of a domain account a local administrator account from the old testdc \ administrator controller. It is clear that, in fact, this is the same account, because when creating a domain, the local built-in administrator of the first controller becomes a built-in administrator of the domain, but, nevertheless, it worked. The server was successfully entered into the domain and its account appeared in the Active Directory database:
It's time to try to make it a controller. Error again. The process of obtaining the controller role via dcpromo ended with the message “Access denied”:
In Active Directory, the logs of the old controller often encountered the error message Event ID 40960 - again the problem with the account, this time with the record of the controller itself:
In general, thoughts about this possibility appeared at the stage of detecting errors with DNS , but I really didn’t want to believe in it.
So, it finally became clear what went wrong during the in-place upgrade — the data on the domain account of the controller that was stored locally on it was lost. As a result, we had a partially workable domain, but there was no domain controller. There was only a server on which the Active Directory database was stored. Since he did not remember his computer password, for the domain he, in fact, was nobody. Above was a link to a Microsoft article that offers a method for solving this problem:
The problem is that for this you need to have a second working controller, but it was not there.
In general, since we have a problem with the discrepancy between the login-password pairs stored locally and in the Active Directory database, then to solve it, we need to be able to somehow change this data.
With the local version of the account, everything is simple. There is a great utility from Joeware - machinepwd . It allows you to set an arbitrary computer recording password (and if this record is not yet broken, then not only locally, but also in the AD database).
Harder to write in AD. Since controller accounts are critical to this service, it protects them from any encroachment. We tried the following:
The last hope was to get the current controller password from Active Directory. After that, you could set the same local password value. There are many tools for extracting this information (for example: Mimikatz DC sync ). However, even with administrator access, you can retrieve the password in the clear only if the Store passwords using reversible encryption setting has been enabled prior to its last shift. In our case, it was turned off. Since the password belonged to a computer account, there was no chance to pick it up using the dictionary with the NTLM hash in hand.
It is unpleasant to admit it, but I never figured out how to remedy the situation in this situation. The customer was forced to lift the domain on holidays (in principle, there was no rush - with the local DNS zone, users did not even notice that something was wrong, so that he could calmly prepare for this operation). It did not even work out the migration tools — they required a trusting relationship between the old and the new domains to transfer the passwords, but a live domain controller is needed to establish these relationships. It is clear that the man himself angry Pinocchio and did not so all that was possible, but still I was hurt.
I hope you enjoyed this story with a bit of a sad ending. As an afterword, a few reminders to novice Active Directory administrators:
PS If you want to get a test environment with the same problem, then all you need is to raise a Windows Server 2003 domain controller, download the machinepwd utility, the link to which I gave above, disable the network connection on the controller, stop the KDC service and, with the help machinepwd, set a new computer password.
PPS If you know how to fix the connection between the controller and the domain in such a situation, please share it with the audience. Your feat will not be forgotten!
UPDATE: As a result of the discussion in the comments, it turned out that I was wrong - manually changing the computer password via MachinePWD is not a complete emulation of what happened. In the case of a password change through this utility, the controller can be revived simply by stopping the KDC and running netdom resetpwd without shamanism. Unfortunately, I could not find the original image of the server to continue the experiments. It turned out very disappointing.
So, there was a system administrator and he had an Active Directory domain. Since the infrastructure was small and inherited a long time ago, there was only one domain controller with Windows Server 2003 installed on it. There were not enough resources, so a rather large number of applications were used on the same server that the administrator himself used and some of his colleagues. Well, the cherry on the cake - backups in this company has never been done. What could go wrong? Details under the cut.
Nothing can stay the same forever and the technical management has decided that it is time to switch to a more recent version of Active Directory, which means you need to update the domain controller. How to switch to a more recent version of Windows Server in your domain, if you have only one controller, on which there is a lot of additional software and for which there are no backups? Of course - in-place upgrade! During the update, something went wrong and the process was interrupted with an error. After that, errors from various systems and applications fell down, and our hero, realizing that the old controller was in trouble, decided to correct the situation by adding a new controller (it’s a pity that he didn’t start with this). A new server with Windows Server 2008 was raised and he tried to enter it into the domain in order to make it a controller. Having received the next error message, the administrator realized that it was time to seek outside help.
This is where our acquaintance with him begins. A small remark - in the process of finding a way to help him, several people participated, but in order to simplify the narration, I will not focus on this attention, especially since it will not add any technical details.
')
For ethical reasons, I cannot show screenshots from someone else’s work environment. Therefore, to write the story, I (now already knowing what had broken down) reproduced the situation in the test lab. For those who wish, at the end of the article will be told how you can do the same if you want to try your hand at solving this problem. Thus, our patients will be the domain Test.local , Windows Server 2003 controller TESTDC , Windows Server 2008 server TESTNEWDC .
DNS
When trying to enter a new server into the domain, the following error was issued:
As usual, problems with Active Directory start with DNS. Looking into the DNS management snap-in at TESTDC we saw an empty server with no zones. So obviously it should not be, since this is the only controller, in combination was the only DNS server.
So, task number one is to restore DNS functionality. Without it, a working domain is out of the question. Just in case, they specified at the customer that all zones were Active Directory integrated. So finding their files on the server itself will not succeed. DNS data must be loaded from the Active Directory partitions of the database. But it seems that this was a problem. Looked into the System and Application logs to see what happens when the DNS service starts. So it is - in the logs were filled with errors Event ID 4000 and Event ID 4007 , characteristic of problems with the operation of AD integrated zone:
It was a very unpleasant sign, since the impossibility of downloading information from the Active Directory database of the only available controller hinted at the severity of the incident. However, there was hope that it would turn out to make it all work with your hands. When trying to create a zone again, the idea was confirmed that the DNS could not get anything from Active Directory. He could not even get a list of domain controllers on which the relevant sections are stored:
As a workaround, it was decided to try to replace the native DNS zone with a local stub. It is clear that it will not contain all the necessary information, but the controller can be forced to update the DNS records for Active Directory, and everything else could wait for now. So, the local main zone test.local was created and dynamic updates were enabled in it. The easiest way that Microsoft offers for registering DNS records of a domain controller is to restart NetLogon service on it. This was done. As a result, a local zone was obtained with the necessary entries:
New DC
Here we took a pause. The customer checked the functionality of applications and systems in their environment. Everything worked. users can use their accounts regularly. The customer was slightly relieved from the heart - at least people could get back to work.
But the main problem was not solved. We are back to trying to enter a new server in the domain. For the introduction to the domain, the domain administrator’s test.local \ administrator account was used, which successfully logged into the old controller. Error again. True, this time, the other. Now "Login Failure: The target account name is incorrect".
Remembering the problem of downloading information from Active Directory, here we had an idea to try instead of a domain account a local administrator account from the old testdc \ administrator controller. It is clear that, in fact, this is the same account, because when creating a domain, the local built-in administrator of the first controller becomes a built-in administrator of the domain, but, nevertheless, it worked. The server was successfully entered into the domain and its account appeared in the Active Directory database:
It's time to try to make it a controller. Error again. The process of obtaining the controller role via dcpromo ended with the message “Access denied”:
In Active Directory, the logs of the old controller often encountered the error message Event ID 40960 - again the problem with the account, this time with the record of the controller itself:
In general, thoughts about this possibility appeared at the stage of detecting errors with DNS , but I really didn’t want to believe in it.
Accounts
So, it finally became clear what went wrong during the in-place upgrade — the data on the domain account of the controller that was stored locally on it was lost. As a result, we had a partially workable domain, but there was no domain controller. There was only a server on which the Active Directory database was stored. Since he did not remember his computer password, for the domain he, in fact, was nobody. Above was a link to a Microsoft article that offers a method for solving this problem:
- Stop the KDC service on the damaged controller
- Run the admin command netdom resetpwd / server: <PDC.domain.com> / userd: <Domain \ domain_admin> / passwordd: *
- Reboot controller
The problem is that for this you need to have a second working controller, but it was not there.
In general, since we have a problem with the discrepancy between the login-password pairs stored locally and in the Active Directory database, then to solve it, we need to be able to somehow change this data.
With the local version of the account, everything is simple. There is a great utility from Joeware - machinepwd . It allows you to set an arbitrary computer recording password (and if this record is not yet broken, then not only locally, but also in the AD database).
Harder to write in AD. Since controller accounts are critical to this service, it protects them from any encroachment. We tried the following:
- The easiest way is to reset the computer account via Active Directory Users and Computers (if you do a Reset, the password is reset to its default value, while creating a new computer - COMPUTERNAME $). The operation gave an access error.
- nltest . This tool allows you to manage secure channel properties (the same login-password pair) for computers in Active Directory. Generally speaking, it allows you to reset the password on both sides. Error finding domain I_NetLogonControl failed: Status = 1355 0x54b ERROR_NO_SUCH_DOMAIN
- dsmod . A utility for working with a computer account in the Active Directory itself buz. Error Internal Error .
- admod . Another utility for working with objects in AD. Unwilling error to perform .
- ktpass An interesting tool that allows you to generate keytab files (a kind of offline Kerberos token). One of the features of this utility is the ability to change the password of the account for which you are creating a keytab file. Access error again.
The last hope was to get the current controller password from Active Directory. After that, you could set the same local password value. There are many tools for extracting this information (for example: Mimikatz DC sync ). However, even with administrator access, you can retrieve the password in the clear only if the Store passwords using reversible encryption setting has been enabled prior to its last shift. In our case, it was turned off. Since the password belonged to a computer account, there was no chance to pick it up using the dictionary with the NTLM hash in hand.
Summarizing
It is unpleasant to admit it, but I never figured out how to remedy the situation in this situation. The customer was forced to lift the domain on holidays (in principle, there was no rush - with the local DNS zone, users did not even notice that something was wrong, so that he could calmly prepare for this operation). It did not even work out the migration tools — they required a trusting relationship between the old and the new domains to transfer the passwords, but a live domain controller is needed to establish these relationships. It is clear that the man himself angry Pinocchio and did not so all that was possible, but still I was hurt.
I hope you enjoyed this story with a bit of a sad ending. As an afterword, a few reminders to novice Active Directory administrators:
- You can use a single domain controller only if you have a VERY good reason (as a rule, lack of money for another Windows Server license). You get a single point of failure, problems with which completely disable your infrastructure.
- No matter how many controllers you have, but especially if there is only one, ALWAYS make backup copies. Information carriers are now so cheap that there can be no reason to save on this.
- Do not install applications on a domain controller unless absolutely necessary. This is not the same server on which you can put something at the same time.
- Once again, I repeat - never start the in-place upgrade procedure without having a backup copy of the server. This procedure in itself carries more risks than migration, so there is no need to expose yourself even more, losing even the possibility of reversing the changes.
PS If you want to get a test environment with the same problem, then all you need is to raise a Windows Server 2003 domain controller, download the machinepwd utility, the link to which I gave above, disable the network connection on the controller, stop the KDC service and, with the help machinepwd, set a new computer password.
PPS If you know how to fix the connection between the controller and the domain in such a situation, please share it with the audience. Your feat will not be forgotten!
UPDATE: As a result of the discussion in the comments, it turned out that I was wrong - manually changing the computer password via MachinePWD is not a complete emulation of what happened. In the case of a password change through this utility, the controller can be revived simply by stopping the KDC and running netdom resetpwd without shamanism. Unfortunately, I could not find the original image of the server to continue the experiments. It turned out very disappointing.
Source: https://habr.com/ru/post/309132/
All Articles