Adding new domain controllers to my lab

Hi all,

This was supposed to be not so hard I was told.  It turned out very hard.  Our trip into the mountains this long weekend was canceled mid-way through, so I am writing this up.  I think it will be helpful next time I have to upgrade domain controllers, and maybe someone else might be helped.


I had two Win2K12 domain controllers – Logan and Logan2.  I wanted to add somethings like Exchange 2016 to my lab, and I thought I needed to updated my domain as well.  I was worried about how many licenses I have from MSDN.  So I tried a while back to upgrade my DC’s.  It did not work but the Veeam Restore worked great. I think it failed for the same reasons that this migration type upgrade failed the first time.

I have been successful now.  I want to share the outline of how I did it.  But, the first time it did not work.  I had to reset, and try again and the outline below is the one from the second time when it worked.  After it I will talk about the first time when things failed and also share my reset outline.

This upgrade also includes removing the old domain controllers so there is also new DNS settings for the lab.

I would also like to thank Kev Johnson, Simon Sparks, and Dan Barr as they all were a big help. Very smart guys and a lot of help in a painful situation.

New DC’s

  • Test to confirm things healthy.
    • Make sure you can access tools like User and Computers
      • On the DC’s
      • And on desktops
    • At the command line – netdom query fsmo looks right
    • Use the AD Replication report to confirm things are in fact working (find it here – This is the single best tool to check your replication.  It has cumulative errors, and last status so very useful).
    • Old server is logan, new is dc01 and dc02.
  • New domain controllers
    • The idea being this is easier than upgrades.
    • Update template – took a while.  It was Windows 2016. Did MS Updates, VMware Tools, and Virtual Hardware. Before I did all the updates I could not deploy Win2K16 templates.  This was the solution.
    • deploy two DCs
      • dc01 – .93
      • dc02 – .94
      • Memory to 6 GB
      • Add 10 GB to each drive C:
      • BTW, took a while to personalize.  Maybe 15 minutes or so.  I waited until I saw proper FQDN  and IP in vSphere Client. Definitely longer than expected but it did work.
    • Promote dc01 to DC, including DNS
      • GC, DNS, along with AD
      • Leave in place in Network Settings for DNS, and add .93 to the other setting.
      • Assign tabs and notes as necessary.
    • Check replication
      • report – success, no failures, all DC seen.
      • used at CLI repadmin /replsummary,  to make sure all dc’s seen in each section
    • Test
      • Does users and computes work on new dc?
    • make dc01 have all the FSMO roles (this can help)
    • check, confirm
      • netdom query fsmo
    • do NTP on DC01 – this can help.
    • confirm things
      • used repadmin /replsummary, all dc’s seen in each section, no errors.
      • users and computers opens right
      • replication report – no failures, all dc’s seen.
    • install DHCP to dc01
    • Configure DHCP on dc01
    • disable DHCP on logan
    • quick DHCP test
      • cipher – aok – can renew a DHCP address
    • IP major work – remove .8, .10 and go with .93,94
      • lefroy – VCSA
      • mwfw01 – firewall
      • vSphere hosts
        • tembo
        • oesa
        • sam
        • victoria
      • VMs
        • broker
        • secbrok
        • vum
        • ias
        • tmcc
        • sql01
        • veeamone
        • vao01
        • vbr01
      • DHCP – refresh, and spot confirm
        • cipher
        • various
    • test
      • log in on various VMs, and PCs
      • access the web via iPad, laptop
      • access sites and domains on DC, and desktop
      • access users and computers on DC, and on desktop
      • The AD replication reports shows everything healthy.
    • Deploy dc02
      • power on
      • add dc and dns
      • check
        • used repadmin /replsummary, all dc’s seen in each section
      • Use the AD replication report, no errors and all seen?
      • Can we access user and computers
        • on dc01
        • dc02
    • DHCP
      • pull out .8 from scope, and add .94
    • demote logan
    • DNS
      • confirm logan not in DNS
      • tweak dc01 to have 94 and, and dc02 to have 93 and
      • make sure .8 is not in forwarders on dc01
      • make sure and are forwarders on both dc01, dc02
    • delete logan
    • promote domain
    • Backup
      • disable ad job
      • do new job, ADDC
      • After a week, delete ad job, and backups

What went wrong the first time

The first time I did not have the AD Replication Report so I thought I had good replication but I did not.  I did use the repadmin / replsummary and it did not show the issue – it only shows connections for the most part. I also did not leave in the DNS settings and right away changed them to both .93 and .94. I also did not turn off the existing DHCP at the right time.
I saw errors like:

With some investigation I determined that with no replication the SYSVOL partitions where not populated so that explains the issues.

The outline at the top did in fact work for me – it is the new and improved one!

Reset time

I did not want to restore from backups as I already had logan2 out of the picture safely.

  • Demote new domain controllers and back to sort of normal.
    • DNS working on Logan
    • DHCP working on Logan
    • disable dc01 dhcp
    • Test
      • cipher – got IP renewed
      • internet access – new IP and to internet
    • transfer roles back to logan
      • scheme
      • naming
      • RID
      • PDC
      • Infrastructure
      • confirm done! netdom query fsmo
    • Do health check – this is very minor help in this case.
    • Do AD report – it needs to look right – no errors and successful connections.
    • demote dc2 – problems, could not talk to domain so need cleaning after. No sysvol after all.
    • demote dc1 – problems, could not talk to domain so need cleaning after. No sysvol after all.
    • power off dc2
    • power off dc1
    • metadata clean up – this article helps with metadata clean up very nicely. Confirm nothing seen of the old DC’s in users and computers
    • clean up DNS, make sure dc1, dc2 not in there.
    • Test (need to renew DHCP)
      • cipher
      • internet
      • Use users and computers on desktop – yes!
    • AD report again.  Only one DC and it crashes, but no replication in place with only one DC so no problem I think.
    • delete both dc
    • change dns (to just .8) in the custom spec for the dc’s so that when I redeploy they are correctly configured to work with the current DNS.


  • 10/28/18 – this did not work.  I guess it worked fine for the the first DC, and it looks like it worked for the second.  But something is wrong in the system partition of the first DC.  So everything looks good but it isn’t.  This has been so surprising and fustrating I am going to redo the domain in my lab.  Which is turning out to be a lot of work but I suspect it will actually work!
  • 10/12/18 – The zone transfer issue is misdirection.  This is Active Directory integrated to zone transfer not in use. Thanks Matt!
  • 10/12/18 – I heard from a smart Microsoft guy who said the process I used is correct.  My issue came out of DNS issues.  Primarily in the _msdcs files.  I was supposed to delete them, and stop / start netlogon and they would be created correctly.  I did not do that but rather inspected each one and made sure it had the correct value.  Many were wrong actually. In addition all three zones (my domain, _msdcs, and reverse zone) all had a zone transfer to a non-existing DNS server. I could not change it and have the change persist through a reboot. So this may be my issues with the failed domain controller migration.

Any questions and comments are welcome. I record this for all to see so I can find it in the future easy when I need to upgrade my DC’s again.  But I also hope that there may be something in here that helps other people.


=== END ===

Leave a Reply