Warning: tech content (and Windows tech content at that) follows.
I don’t blog about work much, and only then, in general, tech terms- this is one of those: mainly for discussion with a few people I know who read here and may offer an explanation.
I’ll start by saying it’s fixed. Rebuilding a Windows Domain Controller is not hard, if other DCs exist. That, in itself, is a lesson why you never, ever, only have one DC in a Windows domain.
First, I’ll set the scene: a (planned but very lengthy) power outage, combined with a desire to maintain service as long as possible, are probably the cause, along with a bit of VMWare cleverness.
The symptoms: Logon failures, unexpected password prompts, account creation failing. One DC with a clock an hour ahead of the others. These symptoms are expected together- Kerberos expects times to be matched fairly closely.
You’d expect the fix to be easy: fix the duff time, reboot the lot, and off it should go. Here’s the but- one DC (on VMWare) kept on advancing it’s clock, each time it started. Some client PCs would pick up time from that server too. As another DC was Infrastructure Master, this one would be refused from talking to the rest of the domain properly.
After much mucking about checking NTP servers, hardware clocks, and VMWare settings (There’s a setting to sync the HW clock to the ESX host at boot), and checking the Daylight Savings patches (which would seem obvious for a 1 hour difference), no good reason was found for the problem: with increasing calls from users and my patience wearing thin, I decided to rebuild, so stopped the time service, synched the clock manually, and ran our old friend
dcpromo
and demoted the server out, renamed it, changed it’s IP, then built a new VMWare Server 2003 R2 box, patched it, and did
dcpromo
again to do the reverse. A bit of tweaking to add DNS and RADIUS and all is well. It seems extreme, but had I done it earlier I’d probably not have the headache I do. What a nice welcome back to work that wasn’t.
My theory is as follows: The large VMWare cluster this DC ran on was running on reduced capacity, and may have even stopped, as the power ran out. At that point the DC would have been paused, and got confused when it was re-awakened several hours in the future. What I don’t understand, is why it would not be told what the correct time was, even with the updating service disabled. Any ideas? *nudges LeeH-W, Sublimeproduct and Andy*
The obvious stuff was OK- hardware clock, ESX Host clock (been there, done that) NTP servers, Timezone, Daylight saving.
Lessons for next time? Just take the bloody lot down, instead of trying to keep stuff going.
Oh, and if any non-sysadmin types have made it this far, this is why we hate power failures, patches, and reboots.
[edit] Read the comments. I’ve solved the mystery.
Most bizarre and one I hope not to come across. Logic would dictate that the first dc to respond would be the place to *set* the time for the domain so it should have fell into line but saying that I’ve seen some machines that if they are so far out they cannot pickup the time and behave like a child in the naughty corner. But at dc? Hmmm, weird…
Your logic is correct- the holder of the PDC Operations Master sets the time for the domain. The problem is the secure channel that the time is set over relies on Kerberos, so if the time is way, way out, the channel doesn’t establish, so the time doesn’t get set, *and* the DC concerned couldn’t determine which other DC was ops master.
Imagine my joy to find the time going wrong on the new machine today, but this time with help from a colleague the mystery is solved.
The VMWare hosts are actually now ESXi, not ESX Server. The ESXi hosts, like their ESX predecessors, are synched to NTP, and a guest virtual machine gets it’s initial clock setting (the equivalent of a real PCs real-time clock) from the ESXi host….
BUT, ESXi seems to read the NTP, and apply local time as UTC, where ESX Server did not. As this has been the first time the ESXi hosts and most of the guests (including the DC) have been booted since the start of BST, we hadn’t seen When the guest boots, it reads the RTC from ESXi, then adds an hour for BST. D’oh.
I’ll be looking for a solution, but for now turning off NTP and setting the ESXi host’s clock properly has it fixed. Damn you, VMWare.
Right. This transpires it’s a “feature” of ESXi and Windows working together.
Windows, for legacy reasons, assumes the PC clock is set to local time. Most operating systems- OSX, Unix, Linux, Solaris assume the clock is set to UTC, and then have a local offset to cater for local time.
ESXi doesn’t, and assumes whatever it receives from an NTP server is UTC- even if that isn’t the case.
More info here.
A booting Windows server reads local time from the ESXi host, which now has the wrong time. It boots with the wrong time, and can’t talk Kerberos. Nice.
It looks like the fix is to manually set ESXi to local time at each clock change, checking them for excessive drift, and sync the Windows boxes with NTP. It’s a bodge, because two operating systems choose to interpret time in a broken way, unlike Ubuntu or anything else Unix-ish, or indeed, VMWare ESX. Idiots.