Krusty Ruffle Babbles

Alright! It took me a while, but I’ve finally gotten this Xen thing up and running on my server. It took a lot of searching around for bug fixes and information, but I think the end result is working alright so far. Of course I haven’t done anything beyond getting the Dom0 up and running, and set up four DomU virtual server for different intended purposes so there’s still a lot to go on that. I’ll walk through the setting up of the Doms today, then start tackling each DomU after that. I’ve reached a point now that I’m doing things, then posting about it the day after I actually get it all worked out so I doubt that I’ll be posting daily or anything, but I am hoping to get this whipped out as quickly as possible so that I can get on with other things As in previous articles about this server I am doing this through a SSH connection and switching to a root shell ( sudo -i ) right off the bat, though I do advise that anytime you need to restart the machine you should have actual access to the physical console attached to it..

Okay, to get things started we just need to install the Xen server package, like this: apt-get install ubuntu-xen-server This will pull in all the dependencies that we need. An important note here! If you have been following the rest of these walk throughs then when this installs it will ask what to do about the menu.lst that we modified in an earlier step. You must tell it to install the package maintainers version, the first time through I told it to keep the altered version and I could not get grub to see the xen kernel. After purging everything that was installed with this, and reinstalling with the ‘use package maintainers version’ it all worked fine, and it kept the changes that I had already made.

The next thing is the problem with TLS (Thread-Local Storage), if you don’t do: mv /lib/tls /lib/tls.disabled You will get a message during boot that warns about emulating threads and this being slow, or something along those lines. I have looked into this as much as possible, and I think that this is a problem with the way init was built. I seem to remember that I read something saying that to get rid of it you could build init without static linked libraries. Unfortunately I can’t find that now…? At any rate, if you use the above command it will remove the error, but I think this is not needed anymore, Ubuntu 8.04 does come with libc6-xen, which is a xen friendly build of glibc so I think this error only happens during init because it was built statically against the non xen friendly glibc. I could be wrong, but I’m leaving it in place and I’m not seeing any slowdowns, though I have not done any actual benchmarking to check it out…

At this point we can reboot, and, assuming we watch the machines console, we will see a LOT more text flying across the screen than before. After this reboot we need to do several things to get ready to start installing virtual machines. First we will edit the xen-tools config file by doing:

vim /etc/xen-tools/xen-tools.conf

Nothing really needs to be changed here, but it makes the command line much easier and shorter if we set some defaults. To build the first virtual machine I changed it so that the following were the only uncommented options:

##
#  /etc/xen-tools/xen-tools.conf
###

##
#
# If you don't wish to use loopback images then you may specify an
# LVM volume group here instead
#
##
lvm = system-volume

#
##
#
#
# install-method = [ debootstrap | rinse | rpmstrap | copy | tar ]
#
#
install-method = copy

#
##
# Disk and Sizing options.
##
#
size = 5Gb # Disk image size.
memory = 256Mb # Memory size
swap = 1Gb # Swap size
# noswap = 1 # Don't use swap at all for the new system.
fs = ext3 # use the EXT3 filesystem for the disk image.
dist = hardy # Default distribution to install.
image = sparse # Specify sparse vs. full disk images.

#
# Networking setup values.
##
#
# Uncomment and adjust these network settings if you wish to give your
# new instances static IP addresses.
#
gateway = 192.168.1.1
netmask = 255.255.255.0
broadcast = 192.168.1.255

##
# Misc options
##
#
# Uncomment the following line if you wish to interactively setup
# a new root password for images.
#
passwd = 1

#
# If you'd like all accounts on your host system which are not present
# on the guest system to be copied over then uncomment the following line.
#
accounts = 1

#
#
# Default kernel and ramdisk to use for the virtual servers
#
kernel = /boot/vmlinuz-`uname -r`
initrd = /boot/initrd.img-`uname -r`

#
# A mirror suitable for use when installing the Dapper release of Ubuntu.
#
mirror = http://us.archive.ubuntu.com/ubuntu/

#
# Filesystem options for the different filesystems we support.
#
ext3_options = noatime,nodiratime,errors=remount-ro
ext2_options = noatime,nodiratime,errors=remount-ro
xfs_options = defaults
reiser_options = defaults

#
# Uncomment if you wish newly created images to boot once they've been
# created.
#
boot = 1

#
# If you're using a newer version of the Xen guest kernel you will
# need to make sure that you use 'xvc0' for the guest serial device,
# and 'xvdX' instead of 'sdX' for serial devices.
#
# You may specify the things to use here:
#
# serial_device = tty1 #default
serial_device = xvc0
#
disk_device = xvda #default
# disk_device = sda
#

This is a trimmed down version, I removed a ton of comments, and tried to leave only the parts that explained the settings I used. A couple of notes are needed here:

1. I tried to use reiserfs as the filesystem type, this did not work no matter what other options I used with it. As soon as I changed it to ext3 it installed and started up perfectly.
2. The man page says that you can use an auto generating ip address by using: eg: ip=192.168.1 without the last octet and then specifying a beginning number in /etc/xen-tools/ips.txt, eg: echo 201 > /etc/xen-tools/ips.txt I could not get this to work and trying to broke all kinds of stuff. I advise just entering the ip addresses manually on the command line.
3. There are a couple of bugs that need to be taken care of. There is a forum thread that explains the bugs, and how to fix them, it also links to the bug reports on launchpad. I would suggest taking care of those before continuing.
4. I also have a problem with the console on the virtual machines, and on the first console of the physical machine. I get no kind of login prompts on either and cannot login with them. This is not a problem for me as I access all of the virtual machines through SSH, and on the physical console I can change to a new VT with <control><alt><F2> and log in from there, but I thought it was worth mentioning, I couldn’t work out a solution to this, but I didn’t try real hard on it either… I think it’s probably because I set it up to use xvc0 instead of tty1 on the config file above…
5. [EDIT] If you want to have login consoles on the virtual machines it is apparently necessary to comment out the line that says: serial_device = xvc0 which will leave consoles using the default tty1 for a console. If you choose to do this you will be able to watch the virtual machine start up and get a login screen for it by doing xm create /etc/xen/xen1.yourdomain.name -c to start it up. However, this did not give me a login screen for the first console on the physical machine, maybe it would have if I had started off that way?[/EDIT]
6. [EDIT] If you do not uncomment this line:

   passwd = 1

   Then your virtual machine will have an enabled root account with no password! If you have already made some with no passwords then you need to log into them and run:

   passwd root

   This will prompt you for the new password, twice. Assuming you type the same thing both times it will confirm success and you will now have a password on the root account of your virtual machine.

If you use this config you will need to be sure that you:

1. Make the lvm=system-volume contains the correct value for the LVM group that you want to use.
2. Pay attention to your memory usage. I have 1.25Gb in my machine, I tried using 256MB on four vitual machines and the Dom0, but could not do it, there is apparently some overhead involved. It took me a while of playing around with settings after they were all set up to get my memory usage figured out and I used different amounts on all of the virtual machines…

One more thing to set up before we can build our first virtual machine, we need to set up the firewall to forward things to the virtual machines we are about to create. The ip-masquerading section of the firewall page at help.ubuntu.com explains this beautifully. Keep in mind though, the instructions for UFW will work with out adding the rules in /etc/ufw/before.rules I think this part can be skipped if you only have only one network card and aren’t planning on using this machine, or any of it’s virtual machines, to forward network traffic to anywhere but the virtual machines. If you have UFW enabled and don’t enable port forwarding you will not be able to reach the virtual machines from anywhere else on the network.

Now we can create the first virtual machine. I named mine after what I wanted to run on that particular server, (like dns.mydomain.name, ldap.mydomain.name, etc…), here I will use generic names that you can change to whatever you would like, you will also need to change the xxx in the ip to whatever you need for each one. Create the first virtual machine by typing:

 xen-create-image --hostname=xen1.yourdomain.name --ip=192.168.xxx

This might take a couple minutes. If you set the boot option in the config file then this machine will start up once it’s created and you’ll be able to see it in the list with xm list and attach to it’s console with xm console xen1.yourdomain.name You can exit that console with <control><5> if you’re using SSH to do this.

Now that the first virtual machine is up and running we log into it with: ssh username@{ip address of new virtual machine}

[EDIT] If you used tty1 for the serial device you can log in through the console by doing: xm console xen1.yourdomain.name [/edit]

As soon as we’re logged in we switch to root with sudo -i and do:

 apt-get update && apt-get upgrade --yes

If you don’t like the TLS error when starting your virtual machines you can do:

 mv /lib/tls /lib/tls.disabled

[EDIT] You may want to install the ubuntu-standard package, this will give your virtual machines a lot of basic command line functionality that is missing by default, like man pages. This can be done with:

apt-get install ubuntu-standard

I suggest doing this here as it will save one more step in other new virtial machines that will be installed by copying this one. [/edit]

Now we need to exit out of there and shut down the virtual machine. Back on Dom0 we need to type:

 xm shutdown xen1.yourdomain.name

[EDIT] If you used tty1 for the console you can shut down the virtual machine by typing halt or using the shutdown command on the machines console. [/edit]

After a minute or so we check to make sure that worked with:

 xm list

Assuming it shutdown properly, (if it’s still in the list wait another minute ant try listing it again), we now have several things to do:

1. Enable kerberos login in OpenSSH by following the instructions on the advanced OpenSSH page at help.ubuntu.com. This step is optional, but I’m planning on using it, so I went ahead and enabled it now.
2. Mount the root disk volume for the virtual server we just created by doing:

    mkdir /mnt/xen && mount /dev/system-volume/xen1.yourdomain.name-disk /mnt/xen

   That last bit will need to be adjusted to whatever you named your LVM volume for xen, and whatever you named the actual virtual machine itself.

3. Copy the OpenSSH config file, the /etc/issue and /etc/issue.net files over to the virtual machine, that is, if you’ve used custom issue files and you want the same config for OpenSSH on all of the virtual machines:

    cp /etc/ssh/sshd_config /mnt/xen/etc/ssh/ && cp /etc/issue* /mnt/xen/etc/

4. [EDIT] I forgot to do the syslog stuff on this the first time around, to have the virtual machines use the network logger that we set up in an earlier step of this series type: vim /mnt/xen/etc/syslog.conf And add:

    # Log to the network server
   *.* @{ip of network syslog server}

   This causes all syslog logs to be sent to the network server. [/EDIT]

5. Edit the xen-tools config. We are now going to copy new virtual machines from the mounted root drive of the system we just set up. This will save us from having to set up OpenSSH, and immediately updating on each one. I will post the differences in my config file below:

    vim /etc/xen-tools/xen-tools.conf

These are the changes I made in my xen-tools.conf:

###
#  /etc/xen-tools/xen-tools.conf
##

#
##
#
#
# install-method = [ debootstrap | rinse | rpmstrap | copy | tar ]
#
#
install-method = copy

#
# If you're using the "copy", or "tar" installation methods you must
# need to specify the source location to copy from, or the source
# .tar file to unpack.
#
# You may specify that with a line such as:
#
# install-source = /path/to/copy
# install-source = /some/path/img.tar
#
#

install-source = /home/xen

#

As you can see, all I really did was change the install method to copy (instead of debootstrap) and add an install source so that xen-tools would know where to copy it from. Now we can install new virtual machines by doing:

 xen-create-image --hostname=newname.yourdomain.name --ip=192.168.xxx

Of course you will need to edit that line as above, and change the ip for each machine installed this way. I created three more this way, making Dom0 and four DomU virtual machines. Once you are done installing machines you need to:

1. Unmount the first virtual machines root drive with: umount /mnt/xen
2. start that machine with: xm create /etc/xen/xen1.yourdomain.name (Adjust for actual name of virtual machine)
3. You may want to reboot, just to make sure everything goes smoothly when restarting.

One last note. If you followed earlier steps in this series then this setup will cause the DomU virtual machines to show that they could not find an encrypted device while starting up, this is due to the information for the encrypted root drive being built into the initrd image on Dom0. The errors just go right on by, but have no effect on the start up of the virtual machine.

This process leaves me with 5 servers on one machine, I just hope that it’s strong enough to run them, considering that I have a very small network to govern I think I’ll be okay, but if you have a bigger network you would need bigger machines. Anyway, I think I’m going to tackle DHCP/DynamicDNS on a virtual machine next, though it will probably take ma a couple of days