Thursday, March 04, 2010

Paravirtualized OpenSolaris & Solaris on XenServer

NOTE: This entry was written for XenServer v5.0 and 5.5 and may not be applicable to 5.6. If you would like 5.6 compatibility then let us know and we'll consider updating this post.

We are big fans of Sun Solaris at 360is. Particularly for anything mission critical or when infrastructure is needed that requires little or no maintenance. Features like Zones, Dtrace, ZFS, and Live Upgrade, make it a joy to work with especially in large, complex environments. We are also big advocates of virtualization, including the XenServer product which brings enterprise virtualization features to the world, free of charge. We aren't alone, the world's largest VM platform runs on Xen technology.

Although Solaris and OpenSolaris run in XenServer, historically they have done so using something called HVM. HVM is slow for IO operations like disk or network activity. HVM works fairly well for pure CPU/RAM based tasks thanks to features AMD and Intel built into recent CPUs, but forget about doing anything else, and forget about using an HVM-mode VM in a real production environment. HVM is particularly unsuitable if low IO latency is a requirement.

Paravirtualization is a much more efficient method of virtualization than HVM, but requires the co-operation of the guest OS in order to play nicely with the hypervisor. Unfortunately Citrix don't supply a paravirtualized OpenSolaris template for XenServer, and there are some bugs in the PV interface that prevent regular Solaris running at full speed too. What a shame!

For those of you not already familiar with the differences between OpenSolaris and Solaris, check out Jim's blog. If you thought Solaris was just a George Clooney film, you can safely ignore this whole posting and probably the entire blog, you are in the wrong place.

Benchmarking HVM Versus Paravirtualized OpenSolaris
The chart at the top shows HVM versus PV OpenSolaris storage performance on a puny Dell system with 1 local hard disk and no optimisations (besides enabling a PV kernel). A single vCPU was used with 512MB RAM. Although the absolute numbers behind this chart are not exciting (we report just the relative figures), the percentage jump from HVM to PV certainly is. Our production systems (VMCo Virtual Appliances), running PV OpenSolaris/Solaris easily exceed 100MB/sec on a single iSCSI Gigabit Ethernet link, again with 1 vCPU allocated. The Paravirtualized guest has about 2x the performance of HVM guest in our simple filesystem test above.

Considering CPU & RAM Performance
Besides advocating Solaris and XenServer, we are also fans of Geekbench, the independent benchmarking software for... well... geeks. Both HVM and PV virtual machines clocked up almost identical scores around 2400 on Geekbench2 (a CPU/RAM based benchmark), thanks to the Dell's AMD-V processors and XenServer's efficiency. Again, not exciting as an absolute measure of performance, but the near identical Geekbench score for HVM and PV virtual machines confirms the vendors assertions about AMD-V/Intel-VT technology. For computationally intensive (pure RAM/CPU) work, HVM and PV have little to split them.

Getting Solaris & OpenSolaris Performing on XenServer
Getting the best performance from either kind of Solaris guest requires running it in PV mode. This is achieved via a slightly different route depending on whether its Solaris or OpenSolaris you are using. First up we will take the easier of the 2.

  • Solaris
    We have provided a trivial template to get around a bug with XenServer/Solaris that would otherwise prevent the guest running at full speed. It runs Solaris on XenServer using Sun's own hybrid drivers described here. "Performance of PV drivers in HVM domain looks similar to that of a fully PV guest domain". Download the template, import it as a custom template, and you are ready to start creating Solaris VMs with better performance than standard.

  • OpenSolaris
    Things are not quite so simple for OpenSolaris as they are for Solaris. In order to get OpenSolaris using PV drivers we need to have it boot to a PV-enabled kernel from cold, and set a couple of critical XenServer options on the VM. The first of these tasks involves making a simple change to the Domain0 machine of each physical server you want to run PV-OpenSolaris on.
    Health Warning: In the extremely unlikely event that you to break your XenServer, it's your problem. Yes 360is provides famously good XenServer support contracts, but this is not an attempt to break your system and then sell you one. This blog entry was originally written for XenServer v5.0. Don't experiment with production systems.
Step By Step Guide, Running Paravirtualized OpenSolaris On XenServer
  1. Locate 2 files from the OpenSolaris install CD, and copy them to the Domain0 of each and every XenServer you want to run a PV OpenSolaris instance on. If you can't find these 2 files on your OpenSolaris CD, you have hold of the wrong media:

    1. Copy

      from your OpenSolaris media into

      on your XenServer Dom0.

    2. Copy

      into the same directory.
  2. Create a new VM in XenCenter, using the "Other Install Media" template", do not start it up yet.

  3. Allocate enough RAM to the VM, 800MB to 1024MB is plenty.

  4. Identify the uuid of your new soon-to-be-OpenSolaris VM using the


  5. Execute the command:

    xe vm-param-set uuid=vm_uuid PV-kernel='/opt/opensolaris/unix'
    Our VM now knows to load this kernel from the Dom0 filesystem.

  6. Now tell tell the VM where to read the PV RAM Disk from:

    xe vm-param-set uuid=vm_uuid PV-ramdisk='/opt/opensolaris/x86.microroot'

  7. We set up our boot arguments for the VM:

    xe vm-param-set uuid=vm_uuid PV-args='/platform/i86xpv/kernel/unix -B console=ttya'

  8. Clear the VM boot policy, forcing it into PV operation:

    xe vm-param-set uuid=vm_uuid HVM-boot-policy=

  9. Also clear the bootloader value:

    xe vm-param-set uuid=vm_uuid PV-bootloader=

  10. Use the following command to identify the uuid of the vbd for our new VM:

    xe vbd-list

  11. Set our VM to boot from the vbd:

    xe vbd-param-set uuid=vbd_uuid bootable=true

  12. Finally we are ready to begin the OpenSolaris install proper. Attach the OpenSolaris install CD/ISO to your VM in the normal way.

  13. Boot the VM, and begin your OpenSolaris install. When finished, shut it down.

  14. Once your OpenSolaris install has completed, we will need to change some settings for the VM in XenServer, to get to to boot to the newly created OpenSolaris root filesystem. The trouble with this, is that the path to the root filesystem depends partly on the name you gave your new OpenSolaris system, in the installer. For this example we are going to assume you called it simply "opensolaris".

    xe vm-param-set uuid=vm_uuid PV-args='/platform/i86xpv/kernel/unix -B console=ttya,zfs-bootfs=rpool/ROOT/opensolaris,bootpath="/xpvd/xdf@51712:a”‘

  15. Disconnect your OpenSolaris ISO image from the VM and you are ready to go.
We still don't have any xen-tools for OpenSolaris so things like live migration will not work. If you need to move the VM then XenServer can't help and I haven't looked at OpenSolaris suspend-to-disk capabilities (suspend-to-RAM is no good to us), Randy is probably the man who knows whether you can suspend OpenSolaris to disk, and then shutdown/restart the VM on a different XenServer in the pool.

Solaris Vs OpenSolaris (PV) Performance on XenServer
Geekbench has the Solaris VM scoring around 2400, so CPU/RAM performance is similar to OpenSolaris, but sequential read and write speed is also very similar to OpenSolaris-PV. So Sun's claim about Solaris with Hybrid drivers being close to the performance of a fully PV (OpenSolaris) domain is true. Over the next few weeks we are going to be investigating further and finding ways of improving performance in vitualized Solaris and OpenSolaris under VMware, XenServer, and Sun's own xVM for our clients. For now, our advice is simple:
  • If the VM workload is purely CPU/RAM based then Solaris or OpenSolaris as PV or HVM performs pretty much equally.
  • If the workload is IO centric then you'd better use either Solaris with hybrid drivers or a Paravirtualized OpenSolaris, built following our instructions.
  • To find out how we transform IT performance for clients who are capital constrained, read more.
  • For Solaris & OpenSolaris tips read Mark's blog, one of our founders.


Anonymous said...


I have question about Solaris 10u8 and Citrix XenServer, are you run this version solaris on the xenserver 5.5 update 2 or 5.6 with PV mode?

Oswaldo said...

Hello, I just tried the solaris template on XenServer 5.6. I can confirm it does not work. The DVD boots, I could choose to boot Solaris from the install media but Solaris 10 wont boot.
Would it be too much asking to upgrade the template to work on 5.6 ?
thanks in advance.

Luiyi said...

Hi, nice work! Any news concerning XenServer 5.6 template?. Tks!

Anonymous said...


I'm try to run soalris 10 10/09 under xenserver 5.6 with HVM support.
What I have to do to run PV mode support.
NIC change from rtls0 to xnf0.
Can you contact with me:



TomTheGeek said...

Thanks for this, I found it really helpful. I'm running Xenserver with OpenSolaris as an iSCSI host for the local storage then using Open HA Cluster to replicate the local storage between physical servers. Boom! Fully redundant storage with live migration using only two servers. That's the idea anyways.

Just wanted to let you know a couple things: First, your web layout is cutting off the commands. For example, step 5, all I can see is "xe vm-param-set uuid=vm_uuid PV-kernel='/opt/opensola" when the full command is "xe vm-param-set uuid=vm_uuid PV-kernel='/opt/opensolaris/unix'"

I'm using Firefox 3.6

Second, could you update this post for XenServer 5.6? I'm using the OpenSolaris VMs to share out the local storage over iSCSI so I/O performance is very important to me.

Cheng Zhang said...

Yes, please make it support XenServer 5.6

Unknown said...

Hello, great post. Shame there is no official support/templates for Solaris. It would be nice to have an updated blog for XenServer 5.6

Tom said...

Unfortunately this template is not working - after creating VM with it in Xen 5.6 I'm getting an error that HVM is required for this installation. Because I have server with Opteron 850 I can only install OS without HVM. Any one knows how to do it with Solaris 10 ?

Greg said...

I don't suppose it is possible to update this post to include XenServer6 and Openindiana 151a?
I'm trying to do the same as Tom.

Maxlen Santos said...

I am interested to know how this would work procedures or non for XenServer 6.02. I'm having problems getting the install NexentaStor 3.1.1 Community Edition as a VM.