Howto Ubuntu Intrepid Ibex 8.10 Xen Dom0

Intro

So... much to my dismay Ubuntu quit supporting Xen as a Dom0. The default kernel intrepid ships with is supposed to support being a domU, but that is just because of the addition of paravirt_ops into the standard linux kernel.

Anyway, they aren't supporting xen anymore because they are going with KVM as their server virtualization technology, and virtualbox as their destkop virt tech. Not that I am opposed to KVM, it just seems like this is a bit more complicated than xen-create-image... especially since I am not familiar with it.

As it so happens there are a couple branches of 2.6.27 linux xen kernels, and while they are not official, they seem to be working well. I think that the plan is to integrate dom0 functionality into the paravirt_ops, which basically means creating the backend drivers, but since that is not completed yet, we will need to use one of these unstable branches to get a functioning dom0 with a recent kernel (although they seem very stable to me -- the one I'm using is from Novell Suse).


Howto:

Install a fresh Ubuntu Server 8.10 AMD64. I opted for lvm, but that is certainly not necessary. Luckily Xen 3.3.0 is in the intrepid repos, unfortunately it doesn't have a compatible linux kernel, so we have to download, compile, and install one.

Note: MAKE SURE ITS AMD64, apparently there are build problems with 32bit. Besides there really is no point to have a 32bit dom0 -- it can't see more than like 3gb of ram, and you can always make 32bit domUs.


sudo apt-get install ubuntu-xen-server build-essential libncurses5-dev gawk mercurial
mkdir -p /home/#user#/build/linux-2.6.27-xen
cd /usr/src/
sudo hg clone http://xenbits.xensource.com/ext/linux-2.6.27-xen.hg
cd linux-2.6.27-xen.hg
make O=/home/#user#/build/linux-2.6.27-xen/ menuconfig
make O=/home/#user#/build/linux-2.6.27-xen/ -j12 #put number of cores X1.5 here#
sudo make O=/home/#user#/build/linux-2.6.27-xen/ modules_install install
sudo depmod 2.6.27.5
sudo mkinitramfs -o /boot/initrd-2.6.27.5.img 2.6.27.5


The most important part here is the menuconfig, by default none of the xen stuff is enabled. I had to disable 10000 Mbps ethernet (ixgbe), the slub allocator, and perfmon (or maybe I selected that and shouldn't have) because they wouldn't compile. You can either build two kernels, one with backend drivers (dom0), and the other with frontend drivers (domUs), or you can put both of them in one build and use the kernel for both dom0 and domUs. See this and this for various xen options. Pushing h in the menuconfig helps a lot

Now you have to edit your /boot/grub/menu.lst to boot to the new kernel under xen. It should look something like:


title Xen 3.3 / Ubuntu 8.10, kernel 2.6.27-xen
uuid 538bd858-c116-45e8-bbb5-93c073464889
kernel /xen-3.3.gz
module /vmlinuz-2.6.27.5 root=/dev/mapper/LVMVG-LV ro console=tty0
module /initrd-2.6.27.5.img


You should now be able to boot in to xen by simply rebooting. If your dom0 doesn't come up then simply reboot and change the grub selection to the non-xen and fix whatever it is you broke ;p.

Oddly the ubuntu-xen-server doesn't include hardy or intrepid configs for it's xen tools configs. This will fix that though:


sudo ln -s /usr/lib/xen-tools/edgy.d /usr/lib/xen-tools/hardy.d
sudo ln -s /usr/lib/xen-tools/edgy.d /usr/lib/xen-tools/intrepid.d


Now you can create an intrepid image using:


sudo xen-create-image --hostname=somehost.example.com --size=10Gb --swap=2Gb --ide --mac=00:00:00:00:00:00 --memory=2Gb --arch=amd64 --lvm=VolumeGroup --kernel=/boot/vmlinuz-2.6.27.5 --initrd=/boot/initrd-2.6.27.5.img --install-method=debootstrap --dist=intrepid --mirror=http://archive.ubuntu.com/ubuntu/ --passwd --accounts --dhcp

---------------------Cut---------------------------
You might be able to fix this somewhere in the default config, but you need to change the console both in the generated .cfg file and in the actual disk image it created:


sudo echo "console='xvc0'" >> /etc/xen/somehost.example.com.cfg
sudo mkdir /media/domudisk
sudo mount /dev/mapper/LVMVGroup--somehost.example.com--disk /media/domudisk
cd /media/domudisk/etc/event.d/
sudo nano tty1
cd ~
sudo umount /media/domudisk


On the last line change 'tty1' to 'xvc0'.
---------------------------------------------------
Update: You can fix this with a config:

sudo su
echo "console='xvc0'" >> /etc/xen-tools/xm.tmpl

(sudo echo doesn't work for me...)
Then add the line "sed -i "s/tty1/${serial_device}/" ${prefix}/etc/event.d/tty1" to /usr/lib/xen-tools/edgy.d/30-disable-gettys, so that last if section reads:

#
# Are we using an alternative serial device?
#
if [ ! -z "${serial_device}" ]; then

serial_device=`basename ${serial_device}`
echo "Setting DomU Serial Device to $serial_device"
# Let the user know.
logMessage "Replacing default serial device (tty1) with ${serial_device}"

# replace existing device.
sed -i "s/tty1/${serial_device}/" ${prefix}/etc/event.d/tty1
mv ${prefix}/etc/event.d/tty1 ${prefix}/etc/event.d/${serial_device}
sed -i -e s/tty1/${serial_device}/ ${prefix}/etc/inittab

# make sure that it is allowed to login.
echo $serial_device >> ${prefix}/etc/securetty
echo "Serial device set"
echo $serial_device
fi


and, finally, uncomment "serial_device = xvc0" and "disk_device = xvda" in /etc/xen-tools/xen-tools.conf

There are other goodies in /etc/xen-tools/xen-tools.conf (such as defaults for image creation) too.


The vanilla kernel that ships with Ubuntu as well as building a new kernel from kernel.org (if you enable the paravirt_ops and frontend drivers), will both run as domUs, you simply have to change the hda/sda references in the cfg file to xvda manually, or you can uncomment the "disk_device = xvda" in /etc/xen-tools/xen-tools.conf. MAKE SURE YOU COPY YOUR /lib/modules/2.6.27.whatever over to the domU drive!!! xen-tools will not do this for you (without more script mods at least -- a simple hack would be to replace what /usr/lib/xen-tools/edgy.d/80-install-modules does with cp -r /usr/lib/modules/* ${prefix}/usr/lib/modules).

To build the latest linux kernel:

mkdir -p /home/user/build/linux-2.6.27/
git clone git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-2.6.27.y.git
cd linux-2.6.27.y
make O=/home/#user#/build/linux-2.6.27-xen/ menuconfig
make O=/home/#user#/build/linux-2.6.27-xen/ -j12 #put number of cores X1.5 here#
sudo make O=/home/xepra/build/linux-2.6.27.y/ modules_install install
depmod 2.6.27.9
mkinitramfs -o /boot/initrd.img-2.6.27-9 2.6.27.9


Notes:

I will update with some more details... hopefully

On another note -- I actually did this with a gentoo install cd initially and booted directly to a xen kernel. It worked surprisingly well, especially since I am a gentoo neophyte, but there weren't any image creation tools. After manually creating a domU and getting it running I had all sorts of networking problems that I never had with Ubuntu... so I went back to Ubuntu.

I also went through this process on Debian lenny, but also had some problems; I think xen 3.3 wasn't in the repos so I had to compile it from scratch... I also was using grub2, which was a bit different, but not hard to figure out and get working.

Update: It looks like Boris Derzhavets used this entry and did a follow up with a lot more detail, screen shots, and output.