High performance XEN setup on CentOS 5.4 (amd64)

The Xen® hypervisor, the powerful open source industry standard for virtualization, offers a powerful, efficient, and secure feature set for virtualization of x86, x86_64 and other CPU architectures.

With Xen virtualization, a small software known as the Xen hypervisor is inserted between the server's hardware and the operating system. This provides an abstraction layer that allows each physical server to run one or more "virtual servers", effectively decoupling the operating system and its applications from the underlying physical server. It needs extremely low overhead and provides near-native performance for guests OS's (virtual servers). Xen is licensed under the GNU General Public License (GPL2) and is available at no charge in both source and object format. Xen is, and always will be, open sourced, uniting the industry and the Xen ecosystem to speed the adoption of virtualization in the enterprise.

I will be using CentOS 5.4 (x86_64) for both the host OS (dom0) and the guest OS (domU's) and LVM based block devices for guest OS's.
Partition planning:
For example we have 80GB disk space and we want to use 3 guest operating systems on it:

Host-OS (Dom0) => 14GB
VM-1 (DomU-1) => 20GB
VM-2 (DomU-2) => 20GB
VM-3 (DomU-3) => 20GB

So while installing Host-OS manage your disk partitions accordingly. This means /dev/sda1 for Dom0 and leave the rest of disk-space (i.e /dev/sda2) for LVM. Post install of Host-OS, we will create LVM based blocks for all three VM's.

Host-OS preparation:
Now proceed with your install on first partition only. Must select "Virtualization" while installing OS. It will install xen hyperviser, kernel-xen, xen-tools and rest of dependents during installation only. Otherwise just install OS as usual and later after boot you can install xen stuff as:

yum update
yum install kernel-xen xen

/boot/grub/grub.conf should be similar to below contents:

title CentOS (2.6.18-128.7.1.el5xen)
root (hd0,0)
kernel /boot/xen.gz-2.6.18-128.7.1.el5 dom0_mem=196608
module /boot/vmlinuz-2.6.18-128.7.1.el5xen ro root=LABEL=/
module /boot/initrd-2.6.18-128.7.1.el5xen.img

dom0_mem=196608 at the end of kernel-line boots the Host-OS (Dom0) with 192MB - which is enough to handle and supervise guest OS's (DomU's). Now reboot the Host-OS...

Post reboot you can confirm this using :

uname -a
xm list

LVM setup for guest OS's:

pvcreate /dev/sda2
vgcreate xvms /dev/sda2
lvcreate -L20G -nvm1 xvms
lvcreate -L20G -nvm2 xvms
lvcreate -L20G -nvm3 xvms

To create minimal domU file-system structure:
Create a file named "/etc/yum-xen.conf" with below contents:

[main]
cachedir=/var/cache/yum
keepcache=1
debuglevel=2
logfile=/var/log/yum.log
pkgpolicy=newest
distroverpkg=redhat-release
tolerant=1
exactarch=1
obsoletes=1
gpgcheck=1
plugins=1
metadata_expire=1800
[base]
name=CentOS-$releasever - Base
mirrorlist=http://mirrorlist.centos.org/?release=5&arch=x86_64&repo=os
gpgcheck=1
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-5
#released updates
[updates]
name=CentOS-$releasever - Updates
mirrorlist=http://mirrorlist.centos.org/?release=5&arch=x86_64&repo=updates
gpgcheck=1
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-5
#packages used/produced in the build but not released
[addons]
name=CentOS-$releasever - Addons
mirrorlist=http://mirrorlist.centos.org/?release=5&arch=x86_64&repo=addons
gpgcheck=1
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-5

Then create the domU filesystem structure as:

mkdir /minimal-base
yum install yum-fastestmirror
yum -c /etc/yum-xen.conf --installroot=/minimal-base -y groupinstall core --disablerepo=extras

Some important changes must needed:

1. /etc/inittab creation and xen-console addition

cp /etc/inittab /minimal-base/etc/inittab
echo "# xen console to provide console access" >> /minimal-base/etc/inittab
echo "co:2345:respawn:/sbin/agetty xvc0 9600 vt100-nav" >> /minimal-base/etc/inittab

2. Copy /etc/resolv.conf and /etc/hosts

cp /etc/resolv.conf /minimal-base/etc/resolv.conf
cp /etc/hosts /minimal-base/etc/hosts

3. Create /etc/sysconfig/network-scripts/ifcfg-eth0 to get IP from Network DHCP Server

echo "DEVICE=eth0" > /minimal-base/etc/sysconfig/network-scripts/ifcfg-eth0
echo "BOOTPROTO=dhcp" >> /minimal-base/etc/sysconfig/network-scripts/ifcfg-eth0
echo "ONBOOT=yes" >> /minimal-base/etc/sysconfig/network-scripts/ifcfg-eth0

4. copy kernel-modules to the minimal-base

mkdir /minimal-base/lib/modules/`uname -r`
cp -R /lib/modules/`uname -r`/* /minimal-base/lib/modules/`uname -r`/

5. Prepare the chroot environment for /minimal-base.

mount --bind /dev /minimal-base/dev
mount proc /minimal-base/proc -t proc
chroot /minimal-base /bin/bash

6. Inside the chroot environment, set the root password and install some minimal softwares

passwd
yum install vim ssh

This should be enough for minimal domU file-system structure creation.

Now create a new initrd image for guest OS's (DomU's) as the default one is not sufficient with all modules:

cd /boot/
mkinitrd --omit-scsi-modules --with=xennet --with=xenblk --preload=xenblk initrd-$(uname -r)-no-scsi.img $(uname -r)

DomU xen config file creation:
Create a xen config file names vm1.cfg in /etc/xen folder with following contents

kernel ="/boot/vmlinuz-2.6.18-128.7.1.el5xen"
ramdisk ="/boot/initrd-2.6.18-128.7.1.el5xen-no-scsi.img"
memory = 256
name = "vm1"
vif = ['mac=00:16:3e:62:da:bb']
disk = ['phy:/dev/xvms/vm1,sda1,w']
root = "/dev/sda1 ro"

Later format /dev/xvms/vm1 with ext3

mkfs.ext3 /dev/xvms/vm1

mount the newly formated lvm block and copy entire /minimal-base to it

mount /dev/xvms/vm1 /mnt
cp -R /minimal-base/* /mnt/
umount /minimal-base/

Start the first VM:

xm create /etc/xen/vm1.cfg -c

Here "-c" option is to provide for console. It should prompt for login, use the root credentials and login to it. To exit from console use Ctrl+].

Similarly do for rest of VM's.
In dom0, to list running OS’s, type:

xm list
xm top

Thats it...