This would have been relatively easy if both NICs used static IPs, this tutorial would have worked more or less out of the box. It turns out that throwing dhcp in the mix really screws with routing and dns. Even worse the dhcp daemon running in the background keeps screwing it up unless you disable it.
-------------------------------------------------
Howto:
Note: This is on Ubuntu 8.04 and Xen 3.2. I want dom0 to access the internet via a static ip on eth0.
First create /etc/xen/scripts/multi-network-bridge then put the following in it:
#!/bin/sh
/etc/xen/scripts/network-bridge $@ vifnum=1 netdev=eth1 bridge=xenbr1
/etc/xen/scripts/network-bridge $@ vifnum=0 netdev=eth0 bridge=xenbr0
ifconfig xenbr1 up
ifconfig xenbr0 up
ip route flush table main
ip route add X.X.X.X/X dev xenbr0
ip route add default via X.X.X.X
cp /etc/resolv.conf.bak /etc/resolv.conf
Where X.X.X.X/X is your static network such as 192.168.1.0/24 and X.X.X.X is your static gateway (192.168.1.1). /etc/resolv.conf.bak is a copy of your static dns servers.
Now make the script executable and disable the dhcp daemon:
sudo chmod +x /etc/xen/scripts/multi-network-bridge
sudo update-rc.d -f dhcdbd remove
Finally, find the line in /etc/xen/xend-config.spx that has "(network-script network-bridge)" and change it to "(network-script multi-network-bridge)".
Now check to make sure that /etc/network/interfaces is configured properly, mine looks like:
auto lo
iface lo inet loopback
auto eth1
iface eth1 inet dhcp
auto eth0
iface eth0 inet static
address x.x.x.x
gateway x.x.x.x
netmask x.x.x.x
For DomUs to use the bridge you need to change the cfg file to look like:
vif = [ 'mac=xx:xx:xx:xx:xx:xx,ip=x.x.x.x,bridge=xenbr0' ]
For the static network, or:
dhcp = 'dhcp'
vif = [ 'mac=xx:xx:xx:xx:xx:xx,bridge=xenbr1' ]
For the dhcp network.
Reboot Dom0 to make sure that networking comes back right.
------------------------------------------------
If you don't specifically name the bridges, then they will automatically be named eth0 and eth1 (and you probably won't have to do the ifconfig xxxx up either). This makes the output of ifconfig look slightly more normal.
A couple other things I found out along the way:
1. The interfaces should already be up and configured properly (with ips) before the bridge script is run. This should happen automatically if /etc/network/interfaces is set up properly.
2. Dom0 connects to the bridge directly, which is why it has an ip assigned.
3. "ip route" and "brctl show" are your friends.
4. The tapX devices are from hardware virtual machines.
Note: Doing this can confuse firefox in to thinking it is offline. If you have issues with firefox make sure File->Work Offline is unchecked (sometimes checking it then unchecking it helps).
Update:
Here is my new multi-network-bridge script:
#!/bin/sh
/etc/xen/scripts/network-bridge $@ vifnum=1 netdev=eth1
/etc/xen/scripts/network-bridge $@ vifnum=0 netdev=eth0
ip route flush table main
ip route add X/X dev eth0
ip route add default via X
cp /etc/resolv.conf.bak /etc/resolv.conf
If you don't specify a bridge then it defaults to eth0.
