Warning: tech content (and Windows tech content at that) follows.<\/p>\n
I don’t blog about work much, and only then, in general, tech terms- this is one of those: mainly for discussion with a few people I know who read here and may offer an explanation.<\/p>\n
I’ll start by saying it’s fixed. Rebuilding a Windows Domain Controller is not hard, if other DCs exist. That, in itself, is a lesson why you never, ever, only have one DC in a Windows domain.
\n
\nFirst, I’ll set the scene: a (planned but very lengthy) power outage, combined with a desire to maintain service as long as possible, are probably the cause, along with a bit of VMWare cleverness.<\/p>\n
The symptoms: Logon failures, unexpected password prompts, account creation failing. One DC with a clock an hour ahead of the others. These symptoms are expected together- Kerberos<\/a> expects times to be matched fairly closely. <\/p>\n You’d expect the fix to be easy: fix the duff time, reboot the lot, and off it should go. Here’s the but- one DC (on VMWare) kept on advancing it’s clock, each time it started. Some client PCs would pick up time from that server too. As another DC was Infrastructure Master, this one would be refused from talking to the rest of the domain properly.<\/p>\n After much mucking about checking NTP servers, hardware clocks, and VMWare settings (There’s a setting to sync the HW clock to the ESX host at boot), and checking the Daylight Savings patches<\/a> (which would seem obvious for a 1 hour difference), no good reason was found for the problem: with increasing calls from users and my patience wearing thin, I decided to rebuild, so stopped the time service, synched the clock manually, and ran our old friend again to do the reverse. A bit of tweaking to add DNS and RADIUS and all is well. It seems extreme, but had I done it earlier I’d probably not have the headache I do. What a nice welcome back to work that wasn’t.<\/p>\n My theory is as follows: The large VMWare cluster this DC ran on was running on reduced capacity, and may have even stopped, as the power ran out. At that point the DC would have been paused, and got confused when it was re-awakened several hours in the future. What I don’t<\/em> understand, is why it would not<\/strong> be told what the correct time was, even with the updating service disabled. Any ideas? *nudges LeeH-W, Sublimeproduct and Andy*<\/p>\n The obvious stuff was OK- hardware clock, ESX Host clock (been there, done that) NTP servers, Timezone, Daylight saving.<\/p>\n Lessons for next time? Just take the bloody lot down, instead of trying to keep stuff going. <\/p>\n Oh, and if any non-sysadmin types have made it this far, this is why we hate power failures, patches, and reboots. <\/p>\n [edit] Read the comments. I’ve solved the mystery.<\/p>\n","protected":false},"excerpt":{"rendered":" Warning: tech content (and Windows tech content at that) follows. I don’t blog about work much, and only then, in general, tech terms- this is one of those: mainly for discussion with a few people I know who read here and may offer an explanation. I’ll start by saying it’s fixed. Rebuilding a Windows Domain … Continue reading Time Machine<\/span>
\n
\ndcpromo<\/code>
\nand demoted the server out, renamed it, changed it’s IP, then built a new VMWare Server 2003 R2 box, patched it, and did<\/p>\n
\ndcpromo<\/code><\/p>\n