FreeBSD 9.0, Paravirtualized Drivers, & Xen-Tools On XenServer 6

We are often asked by clients to help them squeeze more performance from an existing infrastructure, to speed up an application or shorten an IT-dependent business process. Early on in the discussion there's a tendency for sysadmins and architects to dive in to technical minutiae, the black magic and chicken-waving of memory page sizes, block alignment, cache segmentation, and thread pinning. This is almost always a mistake at this early stage and can drain days and weeks with few results. The business gets frustrated when deadlines are missed and the techies can't answer simple questions like "when will this be fixed?", "how much sooner will I get my results?", and "what will it cost to process data within that time window?". Black magic has a time and a place, but at 360is we save our voodoo until the latter stages of a performance tuning project.

Before making non-portable, hard-to-maintain, unsupported, obscure, or fragile configuration changes that break when you least expect them (and may not even be understood by those operating the infrastructure) we start with the basics:

• Is what you have, setup properly? Many complex multi-vendor data centres have a setup that is sub-optimal in some way.
• Is the system already performing more-or-less as you would expect "on paper" +/-25%? We whiteboard a block diagram of devices, buses, networks, data volumes, benchmarks.
• If it isn't then there is probably a significant configurational mistake to be found. Forget about jumbo frames if your switch is stuck in half-duplex mode.
• Focus on the biggest wins first, they are often the easiest to achieve. Only then do we go chasing marginal gains with our consultant's Juju. Don't waste days chasing the last 3% unless that gain makes economic sense for your business process.

Where are we going with all this and what does it have to do with FreeBSD?

At 360is we are fans of FreeBSD, and regularly recommend it as a secure, low-maintenance, stable, and performant, server operating system. Unfortunately there is no official support for FreeBSD in the Citrix XenServer product. What this means in practice is that those wanting to run FreeBSD on XenServer deploy it in HVM mode and without Xen-Tools. Pure HVM mode is slow for network and disk access, and no Xen-Tools means no live migration, not good for production workloads.

Shooting for the biggest win first, we have made available a basic, paravirtualized-drivers-with-xentools-installed, FreeBSD 9.0 64-bit Template that delivers approximately twice the performance of the pure HVM install of the Operating System. No tuning, no special settings, absolutely no chicken-waving.

• FreeBSD 9.0
• 64-Bit (amd64)
• Paravirtualized Drivers in the XENHVM kernel
• Open Source Xen-Tools pre-installed
• Small (389MB) XVA file

Bugs/Errata: Hot-adding of additional NICs works, hot-remove not so much. Use at your own risk. Please submit any other problems as comments, feedback is always welcome.

Get the FreeBSD 9.0 XenServer Template XVA, (cookies/valid Email required in order to be sent the password).

If you need to extract more performance or reliability from an existing infrastructure, application, or IT-driven process, 360is consultants know how, get in touch.

=== Update 26-07-12 ===
As we always get asked about these "relative-to" bar charts, the absolute figures were 188MB/sec for the VM derived from our template, and 92MB/sec for the ordinary install using Citrix "Other OS" template. The fast VM consumed 67% of 1 vCPU on an otherwise idle system. The physical hardware used was a VMCo VA12xx Appliance with its local IOPS sink configured as an SR.

=== Update 11-10-12 ===
This XVA was prepared for XenServer 6.0.2, and probably wont work on XenServer 6.1. If you have a commercial imperative for it on a different version of XenServer, with a different version of FreeBSD other than 9.0, or for that matter with i386 versus amd64, then get in touch with our project office.

Paravirtualized OpenSolaris & Solaris on XenServer

NOTE: This entry was written for XenServer v5.0 and 5.5 and may not be applicable to 5.6. If you would like 5.6 compatibility then let us know and we'll consider updating this post.

We are big fans of Sun Solaris at 360is. Particularly for anything mission critical or when infrastructure is needed that requires little or no maintenance. Features like Zones, Dtrace, ZFS, and Live Upgrade, make it a joy to work with especially in large, complex environments. We are also big advocates of virtualization, including the XenServer product which brings enterprise virtualization features to the world, free of charge. We aren't alone, the world's largest VM platform runs on Xen technology.

Although Solaris and OpenSolaris run in XenServer, historically they have done so using something called HVM. HVM is slow for IO operations like disk or network activity. HVM works fairly well for pure CPU/RAM based tasks thanks to features AMD and Intel built into recent CPUs, but forget about doing anything else, and forget about using an HVM-mode VM in a real production environment. HVM is particularly unsuitable if low IO latency is a requirement.

Paravirtualization is a much more efficient method of virtualization than HVM, but requires the co-operation of the guest OS in order to play nicely with the hypervisor. Unfortunately Citrix don't supply a paravirtualized OpenSolaris template for XenServer, and there are some bugs in the PV interface that prevent regular Solaris running at full speed too. What a shame!

For those of you not already familiar with the differences between OpenSolaris and Solaris, check out Jim's blog. If you thought Solaris was just a George Clooney film, you can safely ignore this whole posting and probably the entire blog, you are in the wrong place.

Benchmarking HVM Versus Paravirtualized OpenSolaris
The chart at the top shows HVM versus PV OpenSolaris storage performance on a puny Dell system with 1 local hard disk and no optimisations (besides enabling a PV kernel). A single vCPU was used with 512MB RAM. Although the absolute numbers behind this chart are not exciting (we report just the relative figures), the percentage jump from HVM to PV certainly is. Our production systems (VMCo Virtual Appliances), running PV OpenSolaris/Solaris easily exceed 100MB/sec on a single iSCSI Gigabit Ethernet link, again with 1 vCPU allocated. The Paravirtualized guest has about 2x the performance of HVM guest in our simple filesystem test above.

Considering CPU & RAM Performance
Besides advocating Solaris and XenServer, we are also fans of Geekbench, the independent benchmarking software for... well... geeks. Both HVM and PV virtual machines clocked up almost identical scores around 2400 on Geekbench2 (a CPU/RAM based benchmark), thanks to the Dell's AMD-V processors and XenServer's efficiency. Again, not exciting as an absolute measure of performance, but the near identical Geekbench score for HVM and PV virtual machines confirms the vendors assertions about AMD-V/Intel-VT technology. For computationally intensive (pure RAM/CPU) work, HVM and PV have little to split them.

Getting Solaris & OpenSolaris Performing on XenServer
Getting the best performance from either kind of Solaris guest requires running it in PV mode. This is achieved via a slightly different route depending on whether its Solaris or OpenSolaris you are using. First up we will take the easier of the 2.

• Solaris
  We have provided a trivial template to get around a bug with XenServer/Solaris that would otherwise prevent the guest running at full speed. It runs Solaris on XenServer using Sun's own hybrid drivers described here. "Performance of PV drivers in HVM domain looks similar to that of a fully PV guest domain". Download the template, import it as a custom template, and you are ready to start creating Solaris VMs with better performance than standard.
  
  
  
  
  
  [Download] Solaris 10 Template (32KB)

• OpenSolaris
  Things are not quite so simple for OpenSolaris as they are for Solaris. In order to get OpenSolaris using PV drivers we need to have it boot to a PV-enabled kernel from cold, and set a couple of critical XenServer options on the VM. The first of these tasks involves making a simple change to the Domain0 machine of each physical server you want to run PV-OpenSolaris on.

Health Warning: In the extremely unlikely event that you to break your XenServer, it's your problem. Yes 360is provides famously good XenServer support contracts, but this is not an attempt to break your system and then sell you one. This blog entry was originally written for XenServer v5.0. Don't experiment with production systems.

Step By Step Guide, Running Paravirtualized OpenSolaris On XenServer

1. Locate 2 files from the OpenSolaris install CD, and copy them to the Domain0 of each and every XenServer you want to run a PV OpenSolaris instance on. If you can't find these 2 files on your OpenSolaris CD, you have hold of the wrong media:
   
   
   
   
   
   1. Copy
      
      
      

      /boot/x86.microroot

      from your OpenSolaris media into
      
      
      

      /opt/opensolaris/

      on your XenServer Dom0.
      
      
   2. Copy
      
      
      

      /platform/i86xpv/kernel/unix

      into the same directory.
2. Create a new VM in XenCenter, using the "Other Install Media" template", do not start it up yet.
   
   
3. Allocate enough RAM to the VM, 800MB to 1024MB is plenty.
   
   
4. Identify the uuid of your new soon-to-be-OpenSolaris VM using the
   
   
   

   vm-list

   command.
   
   
5. Execute the command:
   
   
   

   xe vm-param-set uuid=vm_uuid PV-kernel='/opt/opensolaris/unix'

   Our VM now knows to load this kernel from the Dom0 filesystem.
   
   
6. Now tell tell the VM where to read the PV RAM Disk from:
   
   
   

   xe vm-param-set uuid=vm_uuid PV-ramdisk='/opt/opensolaris/x86.microroot'

   
   
   
7. We set up our boot arguments for the VM:
   
   
   

   xe vm-param-set uuid=vm_uuid PV-args='/platform/i86xpv/kernel/unix -B console=ttya'

   
   
   
8. Clear the VM boot policy, forcing it into PV operation:
   
   
   

   xe vm-param-set uuid=vm_uuid HVM-boot-policy=

   
   
   
9. Also clear the bootloader value:
   
   
   

   xe vm-param-set uuid=vm_uuid PV-bootloader=

   
   
   
10. Use the following command to identify the uuid of the vbd for our new VM:
    
    
    

    xe vbd-list

    
    
    
11. Set our VM to boot from the vbd:
    
    
    

    xe vbd-param-set uuid=vbd_uuid bootable=true

    
    
    
12. Finally we are ready to begin the OpenSolaris install proper. Attach the OpenSolaris install CD/ISO to your VM in the normal way.
    
    
13. Boot the VM, and begin your OpenSolaris install. When finished, shut it down.
    
    
14. Once your OpenSolaris install has completed, we will need to change some settings for the VM in XenServer, to get to to boot to the newly created OpenSolaris root filesystem. The trouble with this, is that the path to the root filesystem depends partly on the name you gave your new OpenSolaris system, in the installer. For this example we are going to assume you called it simply "opensolaris".
    
    
    

    xe vm-param-set uuid=vm_uuid PV-args='/platform/i86xpv/kernel/unix -B console=ttya,zfs-bootfs=rpool/ROOT/opensolaris,bootpath="/xpvd/xdf@51712:a”‘

    
    
    
15. Disconnect your OpenSolaris ISO image from the VM and you are ready to go.
    

We still don't have any xen-tools for OpenSolaris so things like live migration will not work. If you need to move the VM then XenServer can't help and I haven't looked at OpenSolaris suspend-to-disk capabilities (suspend-to-RAM is no good to us), Randy is probably the man who knows whether you can suspend OpenSolaris to disk, and then shutdown/restart the VM on a different XenServer in the pool.

Solaris Vs OpenSolaris (PV) Performance on XenServer

Geekbench has the Solaris VM scoring around 2400, so CPU/RAM performance is similar to OpenSolaris, but sequential read and write speed is also very similar to OpenSolaris-PV. So Sun's claim about Solaris with Hybrid drivers being close to the performance of a fully PV (OpenSolaris) domain is true. Over the next few weeks we are going to be investigating further and finding ways of improving performance in vitualized Solaris and OpenSolaris under VMware, XenServer, and Sun's own xVM for our clients. For now, our advice is simple:

• If the VM workload is purely CPU/RAM based then Solaris or OpenSolaris as PV or HVM performs pretty much equally.
  
• If the workload is IO centric then you'd better use either Solaris with hybrid drivers or a Paravirtualized OpenSolaris, built following our instructions.
  

Links

• To find out how we transform IT performance for clients who are capital constrained, read more.
  
• For Solaris & OpenSolaris tips read Mark's blog, one of our founders.