Paravirtualized OpenSolaris & Solaris on XenServer

NOTE: This entry was written for XenServer v5.0 and 5.5 and may not be applicable to 5.6. If you would like 5.6 compatibility then let us know and we'll consider updating this post.

We are big fans of Sun Solaris at 360is. Particularly for anything mission critical or when infrastructure is needed that requires little or no maintenance. Features like Zones, Dtrace, ZFS, and Live Upgrade, make it a joy to work with especially in large, complex environments. We are also big advocates of virtualization, including the XenServer product which brings enterprise virtualization features to the world, free of charge. We aren't alone, the world's largest VM platform runs on Xen technology.

Although Solaris and OpenSolaris run in XenServer, historically they have done so using something called HVM. HVM is slow for IO operations like disk or network activity. HVM works fairly well for pure CPU/RAM based tasks thanks to features AMD and Intel built into recent CPUs, but forget about doing anything else, and forget about using an HVM-mode VM in a real production environment. HVM is particularly unsuitable if low IO latency is a requirement.

Paravirtualization is a much more efficient method of virtualization than HVM, but requires the co-operation of the guest OS in order to play nicely with the hypervisor. Unfortunately Citrix don't supply a paravirtualized OpenSolaris template for XenServer, and there are some bugs in the PV interface that prevent regular Solaris running at full speed too. What a shame!

For those of you not already familiar with the differences between OpenSolaris and Solaris, check out Jim's blog. If you thought Solaris was just a George Clooney film, you can safely ignore this whole posting and probably the entire blog, you are in the wrong place.

Benchmarking HVM Versus Paravirtualized OpenSolaris
The chart at the top shows HVM versus PV OpenSolaris storage performance on a puny Dell system with 1 local hard disk and no optimisations (besides enabling a PV kernel). A single vCPU was used with 512MB RAM. Although the absolute numbers behind this chart are not exciting (we report just the relative figures), the percentage jump from HVM to PV certainly is. Our production systems (VMCo Virtual Appliances), running PV OpenSolaris/Solaris easily exceed 100MB/sec on a single iSCSI Gigabit Ethernet link, again with 1 vCPU allocated. The Paravirtualized guest has about 2x the performance of HVM guest in our simple filesystem test above.

Considering CPU & RAM Performance
Besides advocating Solaris and XenServer, we are also fans of Geekbench, the independent benchmarking software for... well... geeks. Both HVM and PV virtual machines clocked up almost identical scores around 2400 on Geekbench2 (a CPU/RAM based benchmark), thanks to the Dell's AMD-V processors and XenServer's efficiency. Again, not exciting as an absolute measure of performance, but the near identical Geekbench score for HVM and PV virtual machines confirms the vendors assertions about AMD-V/Intel-VT technology. For computationally intensive (pure RAM/CPU) work, HVM and PV have little to split them.

Getting Solaris & OpenSolaris Performing on XenServer
Getting the best performance from either kind of Solaris guest requires running it in PV mode. This is achieved via a slightly different route depending on whether its Solaris or OpenSolaris you are using. First up we will take the easier of the 2.

• Solaris
  We have provided a trivial template to get around a bug with XenServer/Solaris that would otherwise prevent the guest running at full speed. It runs Solaris on XenServer using Sun's own hybrid drivers described here. "Performance of PV drivers in HVM domain looks similar to that of a fully PV guest domain". Download the template, import it as a custom template, and you are ready to start creating Solaris VMs with better performance than standard.
  
  
  
  
  
  [Download] Solaris 10 Template (32KB)

• OpenSolaris
  Things are not quite so simple for OpenSolaris as they are for Solaris. In order to get OpenSolaris using PV drivers we need to have it boot to a PV-enabled kernel from cold, and set a couple of critical XenServer options on the VM. The first of these tasks involves making a simple change to the Domain0 machine of each physical server you want to run PV-OpenSolaris on.

Health Warning: In the extremely unlikely event that you to break your XenServer, it's your problem. Yes 360is provides famously good XenServer support contracts, but this is not an attempt to break your system and then sell you one. This blog entry was originally written for XenServer v5.0. Don't experiment with production systems.

Step By Step Guide, Running Paravirtualized OpenSolaris On XenServer

1. Locate 2 files from the OpenSolaris install CD, and copy them to the Domain0 of each and every XenServer you want to run a PV OpenSolaris instance on. If you can't find these 2 files on your OpenSolaris CD, you have hold of the wrong media:
   
   
   
   
   
   1. Copy
      
      
      

      /boot/x86.microroot

      from your OpenSolaris media into
      
      
      

      /opt/opensolaris/

      on your XenServer Dom0.
      
      
   2. Copy
      
      
      

      /platform/i86xpv/kernel/unix

      into the same directory.
2. Create a new VM in XenCenter, using the "Other Install Media" template", do not start it up yet.
   
   
3. Allocate enough RAM to the VM, 800MB to 1024MB is plenty.
   
   
4. Identify the uuid of your new soon-to-be-OpenSolaris VM using the
   
   
   

   vm-list

   command.
   
   
5. Execute the command:
   
   
   

   xe vm-param-set uuid=vm_uuid PV-kernel='/opt/opensolaris/unix'

   Our VM now knows to load this kernel from the Dom0 filesystem.
   
   
6. Now tell tell the VM where to read the PV RAM Disk from:
   
   
   

   xe vm-param-set uuid=vm_uuid PV-ramdisk='/opt/opensolaris/x86.microroot'

   
   
   
7. We set up our boot arguments for the VM:
   
   
   

   xe vm-param-set uuid=vm_uuid PV-args='/platform/i86xpv/kernel/unix -B console=ttya'

   
   
   
8. Clear the VM boot policy, forcing it into PV operation:
   
   
   

   xe vm-param-set uuid=vm_uuid HVM-boot-policy=

   
   
   
9. Also clear the bootloader value:
   
   
   

   xe vm-param-set uuid=vm_uuid PV-bootloader=

   
   
   
10. Use the following command to identify the uuid of the vbd for our new VM:
    
    
    

    xe vbd-list

    
    
    
11. Set our VM to boot from the vbd:
    
    
    

    xe vbd-param-set uuid=vbd_uuid bootable=true

    
    
    
12. Finally we are ready to begin the OpenSolaris install proper. Attach the OpenSolaris install CD/ISO to your VM in the normal way.
    
    
13. Boot the VM, and begin your OpenSolaris install. When finished, shut it down.
    
    
14. Once your OpenSolaris install has completed, we will need to change some settings for the VM in XenServer, to get to to boot to the newly created OpenSolaris root filesystem. The trouble with this, is that the path to the root filesystem depends partly on the name you gave your new OpenSolaris system, in the installer. For this example we are going to assume you called it simply "opensolaris".
    
    
    

    xe vm-param-set uuid=vm_uuid PV-args='/platform/i86xpv/kernel/unix -B console=ttya,zfs-bootfs=rpool/ROOT/opensolaris,bootpath="/xpvd/xdf@51712:a”‘

    
    
    
15. Disconnect your OpenSolaris ISO image from the VM and you are ready to go.
    

We still don't have any xen-tools for OpenSolaris so things like live migration will not work. If you need to move the VM then XenServer can't help and I haven't looked at OpenSolaris suspend-to-disk capabilities (suspend-to-RAM is no good to us), Randy is probably the man who knows whether you can suspend OpenSolaris to disk, and then shutdown/restart the VM on a different XenServer in the pool.

Solaris Vs OpenSolaris (PV) Performance on XenServer

Geekbench has the Solaris VM scoring around 2400, so CPU/RAM performance is similar to OpenSolaris, but sequential read and write speed is also very similar to OpenSolaris-PV. So Sun's claim about Solaris with Hybrid drivers being close to the performance of a fully PV (OpenSolaris) domain is true. Over the next few weeks we are going to be investigating further and finding ways of improving performance in vitualized Solaris and OpenSolaris under VMware, XenServer, and Sun's own xVM for our clients. For now, our advice is simple:

• If the VM workload is purely CPU/RAM based then Solaris or OpenSolaris as PV or HVM performs pretty much equally.
  
• If the workload is IO centric then you'd better use either Solaris with hybrid drivers or a Paravirtualized OpenSolaris, built following our instructions.
  

Links

• To find out how we transform IT performance for clients who are capital constrained, read more.
  
• For Solaris & OpenSolaris tips read Mark's blog, one of our founders.

Performance Expert Services

Over the last 12 months we have seen a sharp increase in clients targeting poor performance of systems and applications, particularly around storage, virtualization, and wide area networking. As a result of which we have formalised our professional services for performance investigation, reporting, and remediation. We call this our Performance Expert Service.

We provide fixed-fee projects where our consultants work either independently of your vendors, or with their assistance, to get the performance you need from mission critical systems. When more performance cannot be liberated from existing assets, we are able to provide a quantified case for additional investment, couched in business terms.

While everyone strives for more performance, it is only since the credit crunch and economic slowdown in the UK that there has been a significant increase in these projects for 360is. We put this down to factors impacting IT departments like reduced staffing, frozen budgets, and lack of visibility into the future. Not since the great Y2K spending freeze has there been such a focus on making do with what you have and ensuring it runs efficiently. Bad news for product vendors, but not necessarily bad for end users. Many found the Y2K hiatus in new IT deployments to be no bad thing, some even said IT had never worked so well.

Benefits

• Get next years hardware performance now. (useful if your capex budget has been frozen)
  
• Free up staff from nursing overloaded systems. (good if your team has recently shrunk)
  
• Reduce license costs through higher utilization of fewer systems. (interesting if you just got your support renewal quotes)
  

Our consultants are there for when performance problems defeat your IT team’s efforts, and are beyond the scope of vendor patches and support contracts. Contact us to find out how we can solve your performance problems.

Web Conference, Virtualization Hardware

We are holding a 15 minute webinar on choosing dedicated virtualization hardware. Some of you will have already received the invite directing you to register.

Click on the date below to register for a place and receive the calendar appointment and PDF agenda via Email. Nearer the date and time you will also receive a brief Email reminder.

Save the date: 10am, Friday 19th March 2010, to be held online.

Agenda

"How Virtualization Demands Are Changing Hardware Choices, The Emergence Of A Virtualization Appliance"

Registered attendees will receive further information on the event and a full agenda. A recording of the material will be available for a period after the initial broadcast.

Updated 19-03-10 10am

A recording of the webinar (40MB) is now available. If you cannot read PowerPoint (.pps) files then please contact us for an alternative format.

Solaris vs RedHat vs Windows

Its been a while since our last Solaris/OpenSolaris post. Now, like buses, suddenly 2 come along at once. Both are related to performance, the first (this one) is about benchmarking Solaris versus other mainstream Operating Systems. The second will follow in a few days, and is about maximising Solaris performance when virtualized on XenServer.

Most people see Solaris as extremely reliable, but one misconception that many hold about Sun's OS is that it is slow. Unfortunately, few take the trouble to assess their choice of OS with an open mind, so Solaris remains a "well kept secret", known only to those who work regularly with all the major server Operating Systems.

One of our consultants has been doing some benchmarks for a client, and came up with results we published here.


Solaris x86 was almost always the best performer above Linux (RedHat), and Microsoft, on HP hardware. From 2 cores to 24 cores with a couple of exceptions, it outperformed the others on Geekbench, a CPU and Memory intensive benchmark.

Not a lot of people know that.

360is Flourishes In 2009

Where has this year gone?

We have acquired many new clients in 2009, some in new sectors such as Biotech and Scientific Research (outside of our historical finance and telecoms verticals). We have seen increasing crossover between our security work and the data centre practice, as clients seek to protect their estates "post-consolidation", and vendors introduce virtual versions of traditional hardware appliances for Email, Web, and Network security.

Positive Numbers:

• New client intake up 25%
  
• Average engagement size up by 30%
  
• 25% of engagements were with existing clients
  
• No bad debts

Looking Forward
2010 is already shaping up to be a busy year.

Security Assessments & Investigations
Much of the large scale fraud and corruption which flourished in the global boom is now being brought to light under greater scrutiny and regulation. This has lead to an increase in inquiries for our post-incident clean-up services. The impact of the downturn on organised crime was the subject of a recent Wilton Park Conference hosted by the Home Office.

Virtualization
Early adopters of Virtualization are now looking to maximise their firepower through increased VM density and dedicated virtual appliances. Microsoft's (Hyper-V) entrance into the market, and continued focus of Citrix (XenServer), is putting this technology within reach of the small to medium enterprise, and small departmental projects in large companies. Both products are now completely free. Of course VMware is not going away, and is still present in about 80% of our clients estates.

Desktop Delivery
Microsoft has announced that from July 13, 2010, it will close down the support for Windows XP SP2. That's only 7 months away. Some security critical patches will be made available for XP up to 2014, but it is Microsoft's intention to upgrade all business users to Windows 7 that did not previously adopt Vista (so...everyone then) as soon as possible. At 360is we suggest clients give very serious consideration to whether or not they bother attempting a traditional roll out of Windows 7 to physical desktops, or whether they adopt a virtual desktop strategy for the new OS. Deploying Windows 7 as a virtual desktop has many benefits:


• No hardware refresh necessary
  
• No loss of end user productivity
  
• Easy swap-over, parallel running, rollback
  
• Reduced risk of application incompatibility

360is Business Model
Our roots in IT security always kept us vendor neutral. It's widely acknowledged that a multi-vendor, multi-technology strategy is best when securing large complex infrastructures. We are still convinced that vendor-neutral is the right way to be, not just for security projects but also for our data centre practice. Whether its VMware, Citrix, Microsoft, or a technology embedded in the next generation of CPUs. Our focus remains on professional services and expertise above software and hardware sales. Clients work with us because of what we know, not because of our position in the hierarchy or because they are forced by distribution and purchasing restrictions.

In 2010 we shall continue to deploy our capabilities in the sole interest of our clients to find the right solution to their challenges.

We wish you all a Merry Christmas & a Happy (and prosperous) New Year.