Steve's Blog

This is a quick-n-dirty guide on how to get decently performing DomUs. Disk IO is one of the biggest limitations of Xen setups. Many guides seem to neglect this. This guide helps to make things easier for the Xen community and goes though setting up Xen and CentOS for best performance. It can probably be modified VERY easily for RHEL or Fedora.

I have a 3 x 1Tb drive RAID5 array created as follows:

md2 : active raid5 sdc1[0] sde1[3] sdd1[1]
1953519616 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/3] [UUU]

Having Xen DomUs based on files has massive performance issues. The following figures are taken from a bonnie++ run on a file based DomU.

	Writes	Reads
Per Char	547K/sec, 97% CPU	1310K/sec, 95% CPU
Block	8604K/sec, 0% CPU	97868K/sec, 0% CPU

The raw performance of the RAID5 array in Dom0 is as follows:

	Writes	Reads
Per Char	528K/sec, 97% CPU	1291K/sec, 96% CPU
Block	63418K/sec, 9% CPU	218841K/sec, 1% CPU

As you can see, there is a massive difference in performance! Following this guide, I managed to get the performance of a DomU to:

	Writes	Reads
Per Char	531K/sec, 96% CPU	1276K/sec, 96% CPU
Block	40229K/sec, 3% CPU	195382K/sec, 0% CPU

To make management easy, I used LVM to create a single VG over the entire 2Tb of available storage. This gives me:

# vgdisplay
— Volume group —
VG Name RAID5
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 21
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 6
Open LV 4
Max PV 0
Cur PV 1
Act PV 1
VG Size 1.82 TB
PE Size 4.00 MB
Total PE 476933
Alloc PE / Size 291968 / 1.11 TB
Free PE / Size 184965 / 722.52 GB
VG UUID CnvbWc-r6oM-f8Jg-jvWO-IIfG-JJMN-qtOwgb

From here, I create a partition for the DomU filesystem as such:

— Logical volume —
LV Name /dev/RAID5/template.vm
VG Name RAID5
LV UUID 4rpz9q-hiCf-SN99-wr9Y-Kw5V-PSIE-Fa6ktu
LV Write Access read/write
LV Status available
# open 0
LV Size 1.50 GB
Current LE 384
Segments 1
Allocation inherit
Read ahead sectors auto
– currently set to 512
Block device 253:4

Now here is where we separate from the Xen Docs. These steps have given me 40MB/sec+ write speeds on DomU domains!

1) Create a filesystem on your new LV
# mkfs.ext3 -m1 /dev/RAID5/template.vm

2) Mount the file system on /mnt/template.
# mount /dev/RAID5/template.vm /mnt/template

3) Make some initial directories and files on the new LV:
# mkdir -p /mnt/template/tmp
# mkdir -p /mnt/template/var/lib/rpm
# mkdir -p /mnt/template/var/log
# touch /mnt/template/var/log/yum.log

4) Initialise the RPM database:
# rpm --root=/mnt/template --initdb

5) Download centos-release-5-4.el5.centos.1.x86_64.rpm from your local mirror and install it:
# rpm -ivh --nodeps --root /mnt/template centos-release-5-4.el5.centos.1.x86_64.rpm

6) Install the base packages for CentOS:
# yum --installroot=/mnt/template -y groupinstall base

7) Chroot to your new installation for some quick configuration:
# chroot /mnt/template

Install the xen kernel:
# yum install kernel-xen

9) Recreate your initrd image so that it has the xenblk module included – by default it will not. Substitute kernel versions if your differs:
# cd /boot
# mkinitrd -v -f --with=xenblk initrd-2.6.18-164.15.1.el5xen.img 2.6.18-164.15.1.el5xen

10) Install grub:
# yum install grub

11) Create /boot/grub/grub.conf. It’s contents will be something like:
default=0
timeout=5
splashimage=(hd0)/boot/grub/splash.xpm.gz
hiddenmenu
title CentOS (2.6.18-164.15.1.el5xen)
root (hd0)
kernel /boot/vmlinuz-2.6.18-164.15.1.el5xen ro root=/dev/xvda console=xvc0
initrd /boot/initrd-2.6.18-164.15.1.el5xen.img

12) We also now need to create /etc/fstab as we didn’t have an install process to create it for us! The following should work in most cases:
/dev/xvda / ext3 defaults 1 1
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0


13) At the moment, we don’t have any users configured – nor have we configured the system auth types, so lets do that and then set a root password:
# system-config-auth
# passwd

14) Exit the chroot environment and set up the Xen Vm file. Here is the template I have used, yours will probably differ slightly:
memory = 512
name = "template.vm"
vcpus = 1
disk = [ 'phy:/dev/RAID5/template.vm,xvda,w' ]
on_poweroff = 'destroy'
on_reboot = 'restart'
on_crash = 'restart'
vif = [ "mac=00:16:36:35:35:97,bridge=virbr0,script=vif-bridge" ]
kernel="/usr/lib/xen/boot/pv-grub-x86_64.gz"
extra="(hd0)/boot/grub/grub.conf"

15) Start the DomU and cross your fingers!

# xm create template.vm -c