How does ntds.dit work?

All Active Directory data is stored in the database in the ntds.dit file. The vast majority of applications interact with the directory through the DSA layer implemented in ntdsa.dll. In turn, functions from ntdsa.dll do not work directly with ntds.dit, their functionality is limited by the needs of the directory service and they cannot give us an idea about the internal structure of the Active Directory database. However, ntds.dit is nothing more than a JET Blue database. Each windows version (starting with Windows 2000) has everything you need to work with this database.

In the article below I will try to highlight the following questions:

• What is the structure of the database?
• How does the data in ntds.dit get "tree"?
• How is group membership implemented?
• What is the format of the replPropertyMetaData attribute and with what accuracy are the time stamps stored in the replication metadata?

What you need to look into ntds.dit?

At a minimum: esent.dll
For "comfortable" use from various programming languages ​​over ESENT API there are various "wrappers". I used Meneged Esent in conjunction with C #. There are many examples on the project site - so I will try to concentrate on the contents of ntds.dit later.
')
Also note that the JET Blue DB has a parameter like PageSize. By default, it is equal to 4096 (for those versions of esent.dll that I came across). So in ntds.dit the page size is 8192 and this parameter should be correctly set before opening the database.
The ESENT API has several versions. These versions are incompatible! So ntds.dit from Windows Server 2008 R2 on windows xp will not open, only on Windoes 7. (I did not check backward compatibility)

Question one: What is the structure of the database?

The GetTableNames function will return a list of tables in the database:
datatable - the main table with all directory data
hiddentable
link_table - according to an article on the technet table with information about related attributes (for example MemberOf)
quota_rebuild_progress_table
quota_table
sdpropcounttable
sd_table - according to the article on the technet table contains information about the inherited access rules for each directory object.

Let us dwell on the datatable table. The list of its columns has the form (in the screenshot the first few lines):

Dead end? Not!
The names of most columns have the ATT format <one Latin letter> <numeric code>. So this digital code is the unique identifier of the Active Directory attribute. The table has one row in which the value of the digital code of the column coincides with its value. If we derive all the values ​​of the Text type from this line, we will see that this is the definition of the attribute “attributeID”.



There is only one "undeciphered" attribute ATTc0 - this is a link to "objectClass".
Please note - all columns storing attributes of the Active Directory directory are entered as Multi-value regardless of the scheme.

Question two: How does the data in ntds.dit get "tree"?

The name of the column "DNT_col" provokes to suggest that it is somehow related to the distinguishedName of the object (as it turned out later - they are equal)
The values ​​of the DNT_col column begin with 1, and the line with DNT_col = 1 corresponds to an interesting object with the attribute value name = "$ NOT_AN_OBJECT1 $"
The string with DNT_col = 2 contains the attributes of the object named "$ ROOT_OBJECT $" (Not to be confused with the RootDSE)
With DNT_col = 6, objectClass definitions begin.
Again a dead end? Again, no!
Let's go the other way.
Searching the ATTm589825 column (name) for values ​​of the type Text “Administrator” returned an entry with DNT_col = 3841 and PDNT_col = 1951
Searching the ATTm589825 column (name) for values ​​of the Text “Users” type (the container in which the standard administrator account is located) returned an entry with DNT_col = 1951 and PDNT_col = 1944
Here it is a connection! The PDNT_col column contains the DNT_col identifier of the parent object.
In the line with DNT_col = 1944 (PDNT_col = 1943) - there was a second-level domain object (ntds.dit was taken from the second-level test domain controller)
In the line with DNT_col = 1943 (PDNT_col = 2) - the object of the first level domain.


If you have ever wondered: “Why do not objects from cn = Configuration, dc = contoso, dc = com appear when connecting to dc = contoso, dc = com?” Then you should pay attention to the column NCDNT_col - it contains links on the naming context object. (moreover, the first-level domain object has no naming context - is this why it is not visible?). For Objects in the naming contexts "dc = contoso, dc = com" and "cn = Configuration, dc = contoso, dc = com", the value of this column is different.

No less interesting is how from these numbers a complete distinguishedName of the object is obtained? Which object attribute is a relative unique name (RDN) and is used to build a full distinguishedName.
The RDNtyp_col column contains the attribute identifier (attributeID) containing the RDN.
The column CNT_col contains the number of objects associated with the current one. This column is used by the Link Cleaner mechanism.
If the bit in the OBJ_col column is set to 1, then this string describes the directory object. Otherwise, it is a phantom object.

How is SubTree search implemented?
It is clear that with such a structure, for searching OneLavel it is enough to search among the records with PDNT_col equal to the search base DN, but how to search by SubTree? Bypass all branches? No, it is too difficult.
Look at the values ​​in the column with the speaker name Ancestors_col. It is striking that the deeper the object is located in the hierarchy, the longer the value in this column. Each nesting level adds 4 bytes to the length. This is nothing more than a list of parent objects in the order of precedence.
And the list starts with dn = 2, i.e. with "$ ROOT_OBJECT $"
Who is involved in keeping Ancestors_col up to date? According to the document, when an object is moved to a new place, only its PDNT_col is changed, and the Ancestors_col value is updated by the SDProp mechanism along with the recalculation of the new object access rules.

Question three: How is group membership realized?

Why did the server 2003 have the opportunity to include a practically unlimited number of users in a group, and at the same time it is not possible to write more than 1200 values ​​to a string or numeric multi-value attribute?
Yes, because “member” and “memberOf” are defined as links. There are no columns in the datatable table corresponding to these attributes. The values ​​of these attributes are implemented as matches in a separate link_table table.
Perhaps that is why when you delete a group (without using RecucleBin), we lose information about its members - the object of the remote group receives a new identifier in DNT_col when moving to the container of remote objects (and the connection of the group and its member is based on this identifier)


The names and type of the columns hint, and the analysis of the data they contain confirms that:
link_DNT — Contains the dn identifier (corresponding to DNT_col from datatable) of the object whose link is described (for example, a group object)
backlink_DNT - Contains the dn identifier of the object associated with it (for example, a user object)
link_ncdnt — Contains the dn identifier of the naming context in which the members of the link are located.
link_usnchanged - Contains the last modified USN
link_data - I saw this field filled only for NTDS Settings object links. Yes! NTDS Settings objects have links to objects of directory partitions. Apparently, these connections define the replication parameters of various directory partitions.
link_deltime — Contains the time at which the link was deleted.
link_metadata - contains metadata necessary for replicating the changes made. I saw the filled values ​​of this column only on Windows 2012. Christoffer Andersson writes that the column contains the structure DS_REPL_VALUE_META_DATA and here I totally disagree with it. The DS_REPL_VALUE_META_DATA structure is used at a higher level — it is returned by calls to ntdsa.dll. The data contained in this column are smaller than what is needed for the DS_REPL_VALUE_META_DATA structure. While this column remains a mystery to me. In principle, the replication metadata for attribute-relationships can be obtained from the built-in attribute msDS-ReplValueMetaData, but this does not reveal the internal mechanism for processing and storing this data.

Question four: What is the format of the replPropertyMetaData attribute and with what accuracy are the time stamps stored in the replication metadata?

This attribute stores binary replication metadata:

What is stored in the first 8 bytes of this structure, I have not figured it out yet.
Attention is drawn to the fact that in the replPropertyMetaData structure only replicable attributes with values ​​are listed. For example, you will not see the logonTimestamp attribute identifier in replPropertyMetaData.
There is an interesting point: in the replPropertyMetaData structure for Active Directory security objects (users, groups and computers), the objectSid attribute is present.
By the way, in order to decipher the value of this attribute - it is not necessary to strain so much - there is a built-in attribute msDS-ReplAttributeMetaData - it is nothing but an parsed value of the replPropertyMetaData.

The time stamps in Active Directory are 64-bit unsigned integers, indicating the number of seconds elapsed since 00:00 on January 1, 1601. This leads to an interesting nuance: if you edit one attribute of the same object on two domain controllers for 1 second (for example, when batch processing a large number of users), you can get an unexpected result.
According to the article on technet , the attribute version wins from the controller with a lower GUID.

Information used:

technet.microsoft.com/en-us/library/cc772829%28v=ws.10%29.aspx
blogs.chrisse.se/2012/02/11/how-the-active-directory-data-store-really-works-inside-ntds-dit-part-1
blogs.chrisse.se/2012/02/15/how-the-active-directory-data-store-really-works-inside-ntds-dit-part-2
blogs.chrisse.se/2012/02/20/how-the-active-directory-data-store-really-works-inside-ntds-dit-part-3
blogs.chrisse.se/2012/02/28/how-the-active-directory-data-store-really-works-inside-ntds-dit-part-4
gexeg.blogspot.ru/2009/12/active-directory.html
www.ntdsxtract.com

All experiments were performed on ntds.dit from Windows Server 2012 Eng

UPD . For a snack, a small utility that I wrote to study ntds.dit (maybe someone will do).