Krusty Ruffle Babbles!

Ouch!! Today I was installing and setting up kerberos, trying to set things up so that I could log into any of the computers here with SSH and no passwords. Anyway, there came a point where I decided to reboot the server, it was the first time I had rebooted in a while, and when it came back up it was totally screwed…

DNS was timing out, SSH with an IP address was timing out, and logging in on the physical console seemed to be totally non-functional. Eventually I gave up and went to smoke a cigarette, when I came back I was logged in, but then sudo -i took nearly five minutes to switch to root, a quick tail /var/log/syslog showed many tens of thousands of ‘time went backwards’ messages being logged per second. I turns out that there is a bug that has been around for a while from the looks of it. I followed the suggestions there, adding: “xen.independent_wallclock=1″ to /etc/sysctl.conf in each DomU and “clock=jiffies” in the DomU’s .cnf file in the “extra” section. I also installed ntp on each domU and set them to sync with the NTP server that runs on dom0.

Unfortunately this caused me to encounter another problem, it goes like this: whenever you try to create a domU it dies with the following: ‘Error: Device 0 (vif) could not be connected. Hotplug scripts not
working.!’ and stuff was still running extremely slow. I found various references to this, mostly old stuff, and none of them helped me at all. The weird thing was, it didn’t happen to all of the domU machines, only the one that had DNS/DHCP running on it. I was very close to just doing a complete reinstall, but I’ve been working on this server for a few weeks now, and I really don’t want to lose all of that effort just like that, especially if I didn’t figure out the problem and I might just run nto it again…

Eventually I realized that it was only happening to this one domU, which is normally the first one to start, but since it was not running when I rebooted it always end up being the last one to start. I decided to stop all of the domU machines and try rebooting again, this way I could see if this problem would happen on this same domU, or if it was only the final one to start that would get bit… When it came back up everything started right up and it all worked beautifully!?!? WTF?!

Oh well, I’m going to go with it for now, I guess I’ll find out if it’s really fixed or not the next time I reboot…

To sum up… This is what I changed:

1. on every domU, as root do:

   echo 'xen.independent_wallclock=1' >> /etc/sysctl.conf

2. in each of the /etc/xen/<domU>.cnf add: clock=jiffies to the ‘extras’ line, (inside the quotes).
3. On every domU: Install ntp:

   apt-get install ntp

   And edit the config file, changing ntp.ubuntu.com to whatever your local NTP server is:

   vim /etc/ntp.conf 

   Restart NTP:

   /etc/init.d/ntp restart

4. Shutdown all of the domU’s:

   xm shutdown <name.of.domU> 

   Or, if that doesn’t work:

   xm destroy <name.of.domU> 

   Then reboot the physical machine….

I can’t, and definitely won’t, promise that this will work. It’s just what seems to have done it for me, after an entire day of googling and cussing, and I’ve not had time, or energy really to test it fully yet…

I’m just hoping for the best, and thinking that I may just start always shutting down all of the domU’s before rebooting from now on