Wednesday, December 17, 2008

Howto Ubuntu Intrepid Ibex 8.10 Xen Dom0

Intro

So... much to my dismay Ubuntu quit supporting Xen as a Dom0. The default kernel intrepid ships with is supposed to support being a domU, but that is just because of the addition of paravirt_ops into the standard linux kernel.

Anyway, they aren't supporting xen anymore because they are going with KVM as their server virtualization technology, and virtualbox as their destkop virt tech. Not that I am opposed to KVM, it just seems like this is a bit more complicated than xen-create-image... especially since I am not familiar with it.

As it so happens there are a couple branches of 2.6.27 linux xen kernels, and while they are not official, they seem to be working well. I think that the plan is to integrate dom0 functionality into the paravirt_ops, which basically means creating the backend drivers, but since that is not completed yet, we will need to use one of these unstable branches to get a functioning dom0 with a recent kernel (although they seem very stable to me -- the one I'm using is from Novell Suse).


Howto:

Install a fresh Ubuntu Server 8.10 AMD64. I opted for lvm, but that is certainly not necessary. Luckily Xen 3.3.0 is in the intrepid repos, unfortunately it doesn't have a compatible linux kernel, so we have to download, compile, and install one.

Note: MAKE SURE ITS AMD64, apparently there are build problems with 32bit. Besides there really is no point to have a 32bit dom0 -- it can't see more than like 3gb of ram, and you can always make 32bit domUs.


sudo apt-get install ubuntu-xen-server build-essential libncurses5-dev gawk mercurial
mkdir -p /home/#user#/build/linux-2.6.27-xen
cd /usr/src/
sudo hg clone http://xenbits.xensource.com/ext/linux-2.6.27-xen.hg
cd linux-2.6.27-xen.hg
make O=/home/#user#/build/linux-2.6.27-xen/ menuconfig
make O=/home/#user#/build/linux-2.6.27-xen/ -j12 #put number of cores X1.5 here#
sudo make O=/home/#user#/build/linux-2.6.27-xen/ modules_install install
sudo depmod 2.6.27.5
sudo mkinitramfs -o /boot/initrd-2.6.27.5.img 2.6.27.5


The most important part here is the menuconfig, by default none of the xen stuff is enabled. I had to disable 10000 Mbps ethernet (ixgbe), the slub allocator, and perfmon (or maybe I selected that and shouldn't have) because they wouldn't compile. You can either build two kernels, one with backend drivers (dom0), and the other with frontend drivers (domUs), or you can put both of them in one build and use the kernel for both dom0 and domUs. See this and this for various xen options. Pushing h in the menuconfig helps a lot

Now you have to edit your /boot/grub/menu.lst to boot to the new kernel under xen. It should look something like:


title Xen 3.3 / Ubuntu 8.10, kernel 2.6.27-xen
uuid 538bd858-c116-45e8-bbb5-93c073464889
kernel /xen-3.3.gz
module /vmlinuz-2.6.27.5 root=/dev/mapper/LVMVG-LV ro console=tty0
module /initrd-2.6.27.5.img


You should now be able to boot in to xen by simply rebooting. If your dom0 doesn't come up then simply reboot and change the grub selection to the non-xen and fix whatever it is you broke ;p.

Oddly the ubuntu-xen-server doesn't include hardy or intrepid configs for it's xen tools configs. This will fix that though:


sudo ln -s /usr/lib/xen-tools/edgy.d /usr/lib/xen-tools/hardy.d
sudo ln -s /usr/lib/xen-tools/edgy.d /usr/lib/xen-tools/intrepid.d


Now you can create an intrepid image using:


sudo xen-create-image --hostname=somehost.example.com --size=10Gb --swap=2Gb --ide --mac=00:00:00:00:00:00 --memory=2Gb --arch=amd64 --lvm=VolumeGroup --kernel=/boot/vmlinuz-2.6.27.5 --initrd=/boot/initrd-2.6.27.5.img --install-method=debootstrap --dist=intrepid --mirror=http://archive.ubuntu.com/ubuntu/ --passwd --accounts --dhcp

---------------------Cut---------------------------
You might be able to fix this somewhere in the default config, but you need to change the console both in the generated .cfg file and in the actual disk image it created:


sudo echo "console='xvc0'" >> /etc/xen/somehost.example.com.cfg
sudo mkdir /media/domudisk
sudo mount /dev/mapper/LVMVGroup--somehost.example.com--disk /media/domudisk
cd /media/domudisk/etc/event.d/
sudo nano tty1
cd ~
sudo umount /media/domudisk


On the last line change 'tty1' to 'xvc0'.
---------------------------------------------------
Update: You can fix this with a config:

sudo su
echo "console='xvc0'" >> /etc/xen-tools/xm.tmpl

(sudo echo doesn't work for me...)
Then add the line "sed -i "s/tty1/${serial_device}/" ${prefix}/etc/event.d/tty1" to /usr/lib/xen-tools/edgy.d/30-disable-gettys, so that last if section reads:

#
# Are we using an alternative serial device?
#
if [ ! -z "${serial_device}" ]; then

serial_device=`basename ${serial_device}`
echo "Setting DomU Serial Device to $serial_device"
# Let the user know.
logMessage "Replacing default serial device (tty1) with ${serial_device}"

# replace existing device.
sed -i "s/tty1/${serial_device}/" ${prefix}/etc/event.d/tty1
mv ${prefix}/etc/event.d/tty1 ${prefix}/etc/event.d/${serial_device}
sed -i -e s/tty1/${serial_device}/ ${prefix}/etc/inittab

# make sure that it is allowed to login.
echo $serial_device >> ${prefix}/etc/securetty
echo "Serial device set"
echo $serial_device
fi


and, finally, uncomment "serial_device = xvc0" and "disk_device = xvda" in /etc/xen-tools/xen-tools.conf

There are other goodies in /etc/xen-tools/xen-tools.conf (such as defaults for image creation) too.


The vanilla kernel that ships with Ubuntu as well as building a new kernel from kernel.org (if you enable the paravirt_ops and frontend drivers), will both run as domUs, you simply have to change the hda/sda references in the cfg file to xvda manually, or you can uncomment the "disk_device = xvda" in /etc/xen-tools/xen-tools.conf. MAKE SURE YOU COPY YOUR /lib/modules/2.6.27.whatever over to the domU drive!!! xen-tools will not do this for you (without more script mods at least -- a simple hack would be to replace what /usr/lib/xen-tools/edgy.d/80-install-modules does with cp -r /usr/lib/modules/* ${prefix}/usr/lib/modules).

To build the latest linux kernel:

mkdir -p /home/user/build/linux-2.6.27/
git clone git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-2.6.27.y.git
cd linux-2.6.27.y
make O=/home/#user#/build/linux-2.6.27-xen/ menuconfig
make O=/home/#user#/build/linux-2.6.27-xen/ -j12 #put number of cores X1.5 here#
sudo make O=/home/xepra/build/linux-2.6.27.y/ modules_install install
depmod 2.6.27.9
mkinitramfs -o /boot/initrd.img-2.6.27-9 2.6.27.9


Notes:

I will update with some more details... hopefully

On another note -- I actually did this with a gentoo install cd initially and booted directly to a xen kernel. It worked surprisingly well, especially since I am a gentoo neophyte, but there weren't any image creation tools. After manually creating a domU and getting it running I had all sorts of networking problems that I never had with Ubuntu... so I went back to Ubuntu.

I also went through this process on Debian lenny, but also had some problems; I think xen 3.3 wasn't in the repos so I had to compile it from scratch... I also was using grub2, which was a bit different, but not hard to figure out and get working.

Update: It looks like Boris Derzhavets used this entry and did a follow up with a lot more detail, screen shots, and output.

11 comments:

BDerzhavets said...

# mount /dev/VG01/IntrepidPVG-disk /media/domudisk
# cd /media/domudisk/boot
# ls -l
total 0
Is it expected ?

Clayton Shepard said...

Did that mount succesfully even?

You need to mount the /dev/mapper/VG01-IntrepidPVG--disk

Micro said...

We've been having trouble following your path here..

We've managed to compile the kernel, but have not been able to get past kernel panics when we boot it.

Is there more detail available somewhere that would help?

BDerzhavets said...

Novell's Kernel works under Xen Unstable & Xen 3.3.1 port to Ubuntu 8.10 Server.

root@ServerUbuntu:~# uname -a
Linux ServerUbuntu 2.6.27.5 #1 SMP Thu Jan 29 06:54:25 EST 2009 x86_64 GNU/Linux
root@ServerUbuntu:~# xm info
host : ServerUbuntu
release : 2.6.27.5
version : #1 SMP Thu Jan 29 06:54:25 EST 2009
machine : x86_64
nr_cpus : 2
nr_nodes : 1
cores_per_socket : 2
threads_per_core : 1
cpu_mhz : 3005
hw_caps : bfebfbff:20100800:00000000:00000140:0008e3fd:00000000:00000001:00000000
virt_caps : hvm
total_memory : 8191
free_memory : 127
node_to_cpu : node0:0-1
node_to_memory : node0:127
xen_major : 3
xen_minor : 4
xen_extra : -unstable
xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler : credit
xen_pagesize : 4096
platform_params : virt_start=0xffff800000000000
xen_changeset : Fri Jan 30 11:12:57 2009 +0900 19130:c8962b24fb50
cc_compiler : gcc version 4.3.2 (Ubuntu 4.3.2-1ubuntu11)
cc_compile_by : root
cc_compile_domain :
cc_compile_date : Mon Feb 2 10:28:03 EST 2009
xend_config_format : 4
root@ServerUbuntu:~# brctl show
bridge name bridge id STP enabled interfaces
eth1 8000.001e8c25cca5 no peth1
pan0 8000.000000000000 no

BDerzhavets said...

You can check for details
here

Clayton Shepard said...

Micro:

Hmm, I am assuming you are talking about booting dom0?

It sounds like a config issue.

Here is the config I was using:

http://pastebin.com/f60ca7631


You are on 64 bit right? And you are following this step by step...? What processor are you using?

Priya Bhat said...

Hey!

DomO does not automatically get an internet connection on booting up. I tried changing the settings on /etc/xen/xend-config.sxp but I don't think I have it right...

I am using Ubuntu 8.0.4 and installed xen following the site: http://www.howtoforge.com/ubuntu-8.04-server-install-xen-from-ubuntu-repositories.

After completing the steps, I rebooted the machine to Xen, but it did not get a network; A simple ifconfig is only showing peth0 (not eth0).

BDerzhavets said...

View :-
Setup Xen 3.3.0 Ubuntu Intrepid Server Dom0 via build xen-kernel based on http://xenbits.xensource.com/ext/linux-2.6.27-xen.hg
regarding your complaints and manual
bridge reconfiguration.

Clayton Shepard said...

Priya: While your problem doesn't exactly have anything to do with this blog post, I'll try to help you out a bit. I mean you aren't even talking about 8.10, and my post has nothing to do with networking.

First and foremost some of the xen kernels in ubuntu repositories are bad. I remember specifically a networking bug in the first kernel shipped with 8.04. Since then I believe the bug has been fixed, but if you are doing everything right then I would try a different kernel.

The default settings should have worked out of the box, but a lot of it depends on your network config. Are you doing a bridge, or creating a private network with NAT to the rest of the network? Are you on DHCP or is it static?

Basically peth0 is your physical NIC. Round about xen 3.2 rather than call the bridge "xenbr0" they changed it to "eth0" since that is where your traffic is going, even though it is still a bridge. So it sounds like you modified your xend-config.sxp to not set up a bridge. You can take a look at my latest xen post for more info (at least that one is about xen networking...)

Matt said...

FYI, "sudo echo" doesn't work because you are essentially doing
(sudo echo text) >> file
So it runs echo as root, but tries to append to file as the user. A good way around this is to use the tee command:
echo text | sudo tee -a file
Don't forget the -a or you replace instead of appending.

Clayton Shepard said...

@Matt

Very cool, thanks for the info. Makes sense.