An Enterprise Administrators Blog

Ensuring replication is running smoothly across an Active Directory forest is a primary Administration task.     Best practice recommends the ongoing monitoring of Active Directory replication using SCOM or  another enterprise management tool, monitoring can also be performed using repadmin (repadmin /showrepl * /csv) together with a little scripting to interpret the output.   Typical symptoms of replication issues are objects not being available in one or more site(s), for example a recently created user account, a printer or a directory share is not visible for a subset of users.

A good starting point to confirm there are issues is in the Directory Services Event Logs on the Domain Controllers   Deciding what actions to take are dependent upon the specific events that are being logged, however to troubleshoot, it is important to have a thorough understanding of Active Directory replication.

Most people reading this are already aware Active Directory Forests use sites to manage replication traffic between Domain Controllers.   Sites are interconnected by site links which determine the routing cost for replication of directory data (forest partitions, domain partitions and SYSVOL) between different sites.  The replication  routing algorithm is calculated for the forest by the KCC.

The KCC decides the replication partners for all the Domain Controllers in the forest.    For inter-site replication bridgehead servers are chosen by the KCC or the Administrator can specify a Domain Controller within a site as a preferred bridgehead server which will manage the inter-site replication.  Sites and bridgehead servers allow the replication traffic to be managed and reduce the amount of directory replication traffic across the network.  (More detailed information: Active Directory Replication Topology)

The forest inter-site topology can be analysed using the command:

the command outputs a matrix of sites belonging to the Active Directory forest.  For each site to site relationship the matrix contains 3 values separated by colons, the cost of replication between the two sites, the replication interval in minutes and any site link options that have been set.   A simple example output is:

C:\>repadmin /showism
==== TRANSPORT CN=IP,CN=Inter-Site Transports,CN=Sites,CN=Configu
ration,DC=lab,DC=net CONNECTIVITY INFORMATION FOR 3 SITES: ====
0,    1,    2
Site(0) CN=LAB-Site1,CN=Sites,CN=Configuration,DC=lab,DC=net
0:0:0, 10:15:0, 10:30:0
All DSAs in site CN=ADP-ADSN,CN=Sites,CN=Configuration,DC=lab,
DC=net (with trans& hosting NC) are bridgehead candidates.

Site(1) CN=LAB-Site2,CN=Sites,CN=Configuration,DC=lab,DC=net
10:15:0, 0:0:0, 20:30:0
All DSAs in site CN=ADP-Intranet,CN=Sites,CN=Configuration,DC=la
b,DC=net (with trans & hosting NC) are bridgehead candidates

Site(2) CN=LAB-Site3,CN=Sites,CN=Configuration,DC=lab,DC=net
10:30:0, 20:30:0, 0:0:0
1 server(s) are defined as bridgeheads for transport CN=IP,CN=Int
er-Site Transports,CN=Sites,CN=Configuration,DC=lab,DC=net &
site CN=LAB-Site3,CN=Sites,CN=Configuration,DC=lab,DC=net:
Server(0) CN=testlabdc2,CN=Servers,CN=LAB-Site3,CN=Sites,CN=Co
nfiguration,DC=lab,DC=net

C:\>

The output shows there are 3 configured sites for the lab.net forest, they are listed in the output as Site(0), Site(1) and Site(2).  Each site has 3 sets of replication information, one for each of  the sites in the forest.  For example the replication configuration between Sites(2) (LAB-Site3) and Site(0) (LAB-Site1) is 10:30:0 which translates to a cost of 10 and a replication interval of 30 minutes.    Also of note on Site(2) in the above example is a preferred bridgehead server has been configured for the site, (testlabdc2).

Domain Controller replication partners can be identified both using Gui and command line tools.    Using the Active Directory Sites and Services MMC snap-in, expand the Site node and expand the node with the name of the Site you wish to view.    Under the site the Domain Controllers belonging to the site are listed.  Expanding a Domain Controller and selecting it’s NTDS Settings displays it’s replication partners.

From the command line, using nslookup a list of Domain Controllers belonging to a site can be listed on a domain by domain basis, (of course this does assume that all DC’s are registering their SRV records correctly).  To do this:

which produces an output similar to:

C:\>
_ldap._tcp.LAB-Site1._sites.dc._msdcs.lab.net SRV service location
priority       = 0
weight         = 100
port           = 389
svr hostname   = testlabdc1.lab.net
_ldap._tcp.LAB-Site1._sites.dc._msdcs.lab.net SRV service location
priority       = 0
weight         = 100
port           = 389
svr hostname   = testlabdc2.lab.net
testlabdc1.lab.net       internet address = 172.24.24.10
testlabdc2.lab.net       internet address = 172.24.24.12

To view a Domain Controllers replication partners along with the last attempted replication timestamp and replication status use the following:

DNS is an important component in Active Directory replication , Domain Controllers register SRV records in DNS.   Each Domain Controller in a forest registers a CNAME record dsaGuid._msdcs.forestName where the dsaGuid is the GUID visibile in the NTDS Settings object for the Domain Controller in the AD Sites and Services MMC plug-in.   If DNS related errors are logged in the Directory Services Event Log, run the following which verifies both the CNAME and A records for the Domain Controller are registered correctly in DNS.

If errors are reported, restart the Netlogon service, this will re-register the missing records.   If the dcdiag still throws up an error check the DNS configuration thoroughly.  Ensure the configured DNS settings on the DC are correct

Repadmin provides a quick summary switch, /replsummary that returns a quick replication status summary either environment wide or from the perspective of one or more Domain Controllers by using a wildcard as the Domain Controller name.

Both replication source DSA and replication target DSA information is returned, an example output is:

C:\>repadmin /replsummary testlabdc2
Replication Summary Start Time: 2010-01-24 15:56:03

Beginning data collection for replication summary, this may take a
while:
....

Source DSA       largest delta    fails/total  %% error
testlabdc1           06m:27s       0 /   3    0
testlabdc3           06m:27s       0 /   6    0
testlabdc4           06m:27s       0 /   5    0

Destination DSA  largest delta    fails/total  %% error
testlabdc3           06m:27s       0 /  14    0

C:\>

If there are no replication issues, the output of repadmin /replsummary should (as above) report no errors.

If problems are reported, it is possible to gain more information using Repadmin - Each Domain Controller maintains it’s own Update Sequence Number (USN) which is incremented each time an update to Active Directory is committed.   When replication is initiated with a partner the USN is provided to the partner, the partner compares the USN to the USN it saved during the last replication with the source Domain Controller and from this changes that have occurred since the last replication are determined.

Using the /showutdvec flag, the USN’s currently stored for the replication partners of a Domain Controller can be obtained.

for example

C:\>repadmin /showutdvec  testlabdc4 dc=lab,dc=net
Caching GUIDs.
....
LAB-Site1\testlabdc1   @ USN  16608532 @ Time 2010-01-24 16:27:11
LAB-Site1\testlabdc2   @ USN    307126 @ Time 2010-01-24 16:27:27
LAB-Site2\testlabdc3   @ USN 297948217 @ Time 2010-01-24 16:19:34
LAB-Site3\testlabdc4   @ USN 245646728 @ Time 2010-01-24 16:19:36

C:\>

By running the same command on a Domain Controller that is not experiencing replication issues, the number of missing updates can be obtained by comparing the USN values.

The /queue option displays the inbound replication requests a Domain Controller has to issue to be up to date with it’s replication partners, the returned value for this should be zero.

It is possible to view the update status of individual directory objects using the /showobjmeta switch.   Either a single or a wildcard can be specified for the target Domain Controller.   The output details the originating and local USN along with the originating Domain Controllers GUID and timestamp.  The originating DC/USN/timestamps are provided on an attribute level basis   Use the following format:

To test replication, repadmin offers several possibilities, the /replicate switch replicates a specified directory partition to a destination Domain Controller, the /replsingleobj switch replicates a single object between two Domain Controllers and the /syncall synchronizes a target Domain Controller with all of it’s replication partners.

C:\>repadmin /replicate testlabdc1 testlabdc3 dc=lab,dc=net
Sync from testlabdc3 to testlabdc1 completed successfully.

C:\>repadmin /replsingleobj testlabdc1 testlabdc3 cn=stuart,ou=dsu
sers,dc=lab,dc=net
Successfully replicated object cn=stuart,ou=dsusers,dc=lab,dc=net
to testlabdc1 from <GUID=99305007-2290-489b-9550-20827ba0664d>.

C:\>repadmin /replsingleobj testlabdc1 testlabdc3 ou=dsusers,dc=la
b,dc=net
Successfully replicated object ou=dsusers,dc=lab,dc=net to testlab
dc1 from <GUID=99305007-2290-489b-9550-20827ba0664d>.

C:\>repadmin /replsingleobj testlabdc1 testlabdc3 dc=lab,dc=net
Successfully replicated object dc=lab,dc=net to testlabdc1 from
<GUID=99305007-2290-489b-9550-20827ba0664d>.

C:\>repadmin /syncall testlabdc3
CALLBACK MESSAGE: The following replication is in progress:
From: 25fdc051-6ff6-4922-bc02-0b77a4652bfc._msdcs.lab.net
To  : 99305007-2290-489b-9551-20827ba0664d._msdcs.lab.net
CALLBACK MESSAGE: The following replication completed successfully
From: 25fdc051-6ff6-4922-bc02-0b77a4652bfc._msdcs.lab.net
To  : 99305007-2290-489b-9551-20827ba0664d._msdcs.lab.net
CALLBACK MESSAGE: The following replication is in progress:
From: b0870af5-ab82-4372-9e39-0a9772a5e47c._msdcs.lab.net
To  : 99305007-2290-489b-9551-20827ba0664d._msdcs.lab.net
CALLBACK MESSAGE: The following replication completed successfully
From: b0870af5-ab82-4372-9e39-0a9772a5e47c._msdcs.lab.net
To  : 99305007-2290-489b-9551-20827ba0664d._msdcs.lab.net
CALLBACK MESSAGE: SyncAll Finished.
SyncAll terminated with no errors.

C:\>

when a problem with Active Directory replication has been identified, using repadmin, nslookup, dcdiag and the Directory Services Event Log can help identify the scope and the source of the issue.  Microsoft KB’s provide step by step actions to resolve many replication problems (search the Microsoft Knowledge Base for Event IDs found in the Directory Services Event Log) but hopefully this posting has provided information to increase understanding of Active Directory replication and how to use the tools to understand your environment.