Paravirtualized OpenSolaris & Solaris on XenServer

NOTE: This entry was written for XenServer v5.0 and 5.5 and may not be applicable to 5.6. If you would like 5.6 compatibility then let us know and we'll consider updating this post.

We are big fans of Sun Solaris at 360is. Particularly for anything mission critical or when infrastructure is needed that requires little or no maintenance. Features like Zones, Dtrace, ZFS, and Live Upgrade, make it a joy to work with especially in large, complex environments. We are also big advocates of virtualization, including the XenServer product which brings enterprise virtualization features to the world, free of charge. We aren't alone, the world's largest VM platform runs on Xen technology.

Although Solaris and OpenSolaris run in XenServer, historically they have done so using something called HVM. HVM is slow for IO operations like disk or network activity. HVM works fairly well for pure CPU/RAM based tasks thanks to features AMD and Intel built into recent CPUs, but forget about doing anything else, and forget about using an HVM-mode VM in a real production environment. HVM is particularly unsuitable if low IO latency is a requirement.

Paravirtualization is a much more efficient method of virtualization than HVM, but requires the co-operation of the guest OS in order to play nicely with the hypervisor. Unfortunately Citrix don't supply a paravirtualized OpenSolaris template for XenServer, and there are some bugs in the PV interface that prevent regular Solaris running at full speed too. What a shame!

For those of you not already familiar with the differences between OpenSolaris and Solaris, check out Jim's blog. If you thought Solaris was just a George Clooney film, you can safely ignore this whole posting and probably the entire blog, you are in the wrong place.

Benchmarking HVM Versus Paravirtualized OpenSolaris
The chart at the top shows HVM versus PV OpenSolaris storage performance on a puny Dell system with 1 local hard disk and no optimisations (besides enabling a PV kernel). A single vCPU was used with 512MB RAM. Although the absolute numbers behind this chart are not exciting (we report just the relative figures), the percentage jump from HVM to PV certainly is. Our production systems (VMCo Virtual Appliances), running PV OpenSolaris/Solaris easily exceed 100MB/sec on a single iSCSI Gigabit Ethernet link, again with 1 vCPU allocated. The Paravirtualized guest has about 2x the performance of HVM guest in our simple filesystem test above.

Considering CPU & RAM Performance
Besides advocating Solaris and XenServer, we are also fans of Geekbench, the independent benchmarking software for... well... geeks. Both HVM and PV virtual machines clocked up almost identical scores around 2400 on Geekbench2 (a CPU/RAM based benchmark), thanks to the Dell's AMD-V processors and XenServer's efficiency. Again, not exciting as an absolute measure of performance, but the near identical Geekbench score for HVM and PV virtual machines confirms the vendors assertions about AMD-V/Intel-VT technology. For computationally intensive (pure RAM/CPU) work, HVM and PV have little to split them.

Getting Solaris & OpenSolaris Performing on XenServer
Getting the best performance from either kind of Solaris guest requires running it in PV mode. This is achieved via a slightly different route depending on whether its Solaris or OpenSolaris you are using. First up we will take the easier of the 2.

• Solaris
  We have provided a trivial template to get around a bug with XenServer/Solaris that would otherwise prevent the guest running at full speed. It runs Solaris on XenServer using Sun's own hybrid drivers described here. "Performance of PV drivers in HVM domain looks similar to that of a fully PV guest domain". Download the template, import it as a custom template, and you are ready to start creating Solaris VMs with better performance than standard.
  
  
  
  
  
  [Download] Solaris 10 Template (32KB)

• OpenSolaris
  Things are not quite so simple for OpenSolaris as they are for Solaris. In order to get OpenSolaris using PV drivers we need to have it boot to a PV-enabled kernel from cold, and set a couple of critical XenServer options on the VM. The first of these tasks involves making a simple change to the Domain0 machine of each physical server you want to run PV-OpenSolaris on.

Health Warning: In the extremely unlikely event that you to break your XenServer, it's your problem. Yes 360is provides famously good XenServer support contracts, but this is not an attempt to break your system and then sell you one. This blog entry was originally written for XenServer v5.0. Don't experiment with production systems.

Step By Step Guide, Running Paravirtualized OpenSolaris On XenServer

1. Locate 2 files from the OpenSolaris install CD, and copy them to the Domain0 of each and every XenServer you want to run a PV OpenSolaris instance on. If you can't find these 2 files on your OpenSolaris CD, you have hold of the wrong media:
   
   
   
   
   
   1. Copy
      
      
      

      /boot/x86.microroot

      from your OpenSolaris media into
      
      
      

      /opt/opensolaris/

      on your XenServer Dom0.
      
      
   2. Copy
      
      
      

      /platform/i86xpv/kernel/unix

      into the same directory.
2. Create a new VM in XenCenter, using the "Other Install Media" template", do not start it up yet.
   
   
3. Allocate enough RAM to the VM, 800MB to 1024MB is plenty.
   
   
4. Identify the uuid of your new soon-to-be-OpenSolaris VM using the
   
   
   

   vm-list

   command.
   
   
5. Execute the command:
   
   
   

   xe vm-param-set uuid=vm_uuid PV-kernel='/opt/opensolaris/unix'

   Our VM now knows to load this kernel from the Dom0 filesystem.
   
   
6. Now tell tell the VM where to read the PV RAM Disk from:
   
   
   

   xe vm-param-set uuid=vm_uuid PV-ramdisk='/opt/opensolaris/x86.microroot'

   
   
   
7. We set up our boot arguments for the VM:
   
   
   

   xe vm-param-set uuid=vm_uuid PV-args='/platform/i86xpv/kernel/unix -B console=ttya'

   
   
   
8. Clear the VM boot policy, forcing it into PV operation:
   
   
   

   xe vm-param-set uuid=vm_uuid HVM-boot-policy=

   
   
   
9. Also clear the bootloader value:
   
   
   

   xe vm-param-set uuid=vm_uuid PV-bootloader=

   
   
   
10. Use the following command to identify the uuid of the vbd for our new VM:
    
    
    

    xe vbd-list

    
    
    
11. Set our VM to boot from the vbd:
    
    
    

    xe vbd-param-set uuid=vbd_uuid bootable=true

    
    
    
12. Finally we are ready to begin the OpenSolaris install proper. Attach the OpenSolaris install CD/ISO to your VM in the normal way.
    
    
13. Boot the VM, and begin your OpenSolaris install. When finished, shut it down.
    
    
14. Once your OpenSolaris install has completed, we will need to change some settings for the VM in XenServer, to get to to boot to the newly created OpenSolaris root filesystem. The trouble with this, is that the path to the root filesystem depends partly on the name you gave your new OpenSolaris system, in the installer. For this example we are going to assume you called it simply "opensolaris".
    
    
    

    xe vm-param-set uuid=vm_uuid PV-args='/platform/i86xpv/kernel/unix -B console=ttya,zfs-bootfs=rpool/ROOT/opensolaris,bootpath="/xpvd/xdf@51712:a”‘

    
    
    
15. Disconnect your OpenSolaris ISO image from the VM and you are ready to go.
    

We still don't have any xen-tools for OpenSolaris so things like live migration will not work. If you need to move the VM then XenServer can't help and I haven't looked at OpenSolaris suspend-to-disk capabilities (suspend-to-RAM is no good to us), Randy is probably the man who knows whether you can suspend OpenSolaris to disk, and then shutdown/restart the VM on a different XenServer in the pool.

Solaris Vs OpenSolaris (PV) Performance on XenServer

Geekbench has the Solaris VM scoring around 2400, so CPU/RAM performance is similar to OpenSolaris, but sequential read and write speed is also very similar to OpenSolaris-PV. So Sun's claim about Solaris with Hybrid drivers being close to the performance of a fully PV (OpenSolaris) domain is true. Over the next few weeks we are going to be investigating further and finding ways of improving performance in vitualized Solaris and OpenSolaris under VMware, XenServer, and Sun's own xVM for our clients. For now, our advice is simple:

• If the VM workload is purely CPU/RAM based then Solaris or OpenSolaris as PV or HVM performs pretty much equally.
  
• If the workload is IO centric then you'd better use either Solaris with hybrid drivers or a Paravirtualized OpenSolaris, built following our instructions.
  

Links

• To find out how we transform IT performance for clients who are capital constrained, read more.
  
• For Solaris & OpenSolaris tips read Mark's blog, one of our founders.