Wednesday, July 25, 2012

FreeBSD 9.0, Paravirtualized Drivers, & Xen-Tools On XenServer 6


We are often asked by clients to help them squeeze more performance from an existing infrastructure, to speed up an application or shorten an IT-dependent business process. Early on in the discussion there's a tendency for sysadmins and architects to dive in to technical minutiae, the black magic and chicken-waving of memory page sizes, block alignment, cache segmentation, and thread pinning. This is almost always a mistake at this early stage and can drain days and weeks with few results. The business gets frustrated when deadlines are missed and the techies can't answer simple questions like "when will this be fixed?", "how much sooner will I get my results?", and "what will it cost to process data within that time window?". Black magic has a time and a place, but at 360is we save our voodoo until the latter stages of a performance tuning project.

Before making non-portable, hard-to-maintain, unsupported, obscure, or fragile configuration changes that break when you least expect them (and may not even be understood by those operating the infrastructure) we start with the basics:
  • Is what you have, setup properly? Many complex multi-vendor data centres have a setup that is sub-optimal in some way.
  • Is the system already performing more-or-less as you would expect "on paper" +/-25%? We whiteboard a block diagram of devices, buses, networks, data volumes, benchmarks.
  • If it isn't then there is probably a significant configurational mistake to be found. Forget about jumbo frames if your switch is stuck in half-duplex mode.
  • Focus on the biggest wins first, they are often the easiest to achieve. Only then do we go chasing marginal gains with our consultant's Juju. Don't waste days chasing the last 3% unless that gain makes economic sense for your business process.
Where are we going with all this and what does it have to do with FreeBSD?

At 360is we are fans of FreeBSD, and regularly recommend it as a secure, low-maintenance, stable, and performant, server operating system. Unfortunately there is no official support for FreeBSD in the Citrix XenServer product. What this means in practice is that those wanting to run FreeBSD on XenServer deploy it in HVM mode and without Xen-Tools. Pure HVM mode is slow for network and disk access, and no Xen-Tools means no live migration, not good for production workloads.

Shooting for the biggest win first, we have made available a basic, paravirtualized-drivers-with-xentools-installed, FreeBSD 9.0 64-bit Template that delivers approximately twice the performance of the pure HVM install of the Operating System. No tuning, no special settings, absolutely no chicken-waving.
  • FreeBSD 9.0
  • 64-Bit (amd64)
  • Paravirtualized Drivers in the XENHVM kernel
  • Open Source Xen-Tools pre-installed
  • Small (389MB) XVA file
Bugs/Errata: Hot-adding of additional NICs works, hot-remove not so much. Use at your own risk. Please submit any other problems as comments, feedback is always welcome.

Get the FreeBSD 9.0 XenServer Template XVA, (cookies/valid Email required in order to be sent the password).

If you need to extract more performance or reliability from an existing infrastructure, application, or IT-driven process, 360is consultants know how, get in touch.

=== Update 26-07-12 ===
As we always get asked about these "relative-to" bar charts, the absolute figures were 188MB/sec for the VM derived from our template, and 92MB/sec for the ordinary install using Citrix "Other OS" template. The fast VM consumed 67% of 1 vCPU on an otherwise idle system. The physical hardware used was a VMCo VA12xx Appliance with its local IOPS sink configured as an SR.

=== Update 11-10-12 ===
This XVA was prepared for XenServer 6.0.2, and probably wont work on XenServer 6.1. If you have a commercial imperative for it on a different version of XenServer, with a different version of FreeBSD other than 9.0, or for that matter with i386 versus amd64, then get in touch with our project office.

19 comments:

Anonymous said...

This is a wonderful thing. I entered my email, where do I get the password?

Anonymous said...

Hi,

Our product works on freebsd 7 and we could bring it up in HVM. Was not sure how to provide PV support for it. Is there a doc going over how to make the freebsd7 paravirtualized

Nick said...

Hi Anon,

The root password for the VM is fbsd360is. Remember to change it to something unique. SSH is disabled for root by default.

Nick said...

Anonymous FreeBSD 7 reader...

Unfortunately prior to FreeBSD 8.1 the xen code in the kernel had quite a few significant bugs, so the results are not going to be very good. Version 8.1, 8.2, and 8.3 are what we would recommend.

Anonymous said...

if i update freebsd, does it break the PV mode?

Anonymous said...
This comment has been removed by a blog administrator.
Nick said...

Hi Anon FBSD User,

You can update the OS without breaking the PV functionality provided you dont update the kernel. If you change the kernel then you may lose the PV drivers unless you explicitly include them.

Anonymous said...

Nick,

How does your kernel differ from the XENHVM kernel whose spec is included in FreeBSD?

Can you provide your kernelconf so a kernel rebuild is possible in case of vulnerability reports?

Anonymous said...

Hi Anon,

The standard XENHVM kernel was used as it now contains all the required Xen drivers and devices, to that we added a handful of useful modules (chiefly nfs ones, plus packet filtering since our clients use case was a FBSD Firewall).

The main addition is xen-tools from ports and the required shell scripts to enable live migration and communication of performance statistics to the Citrix XenServer Hypervisor.

Anonymous said...

I've noticed that there are no kernel modules available in the template. Since I cannot recompile the kernel without breaking things, how would I go about adding kernel modules to this?

Nick said...

Hi Anon,

If you would like to add kernel modules, then the best option is for you to install the source tree (takes up quite a bit of disk so was not included in this XVA file), and recompile the kernel (XENHVM) for your needs. We included packet filtering and nfs modules as well as the default modules.

If you have a project that requires our help then we'd be happy to talk about an engagement, just get in touch with info@360is.com.

Unknown said...

How have you managed to Install the Xen-Server tools ??

i have copied your kernel over to "freenas vm" it boots your kernel nice.. but the xen-tools are missing or not loaded.. so it would be nice if you can say, how you installed the xen-server tools.

Unknown said...

How have you managed to Install the Xen-Server tools on the VM?

plz help

Nick said...

Hi William Harrell,

Xen-Tools is userspace software that collects various pieces of information from the system and reads/writes then to/from some special devices created by a Xen-capable Kernel. Information like how much RAM is free and details about what the guest OS is.

The 360is FreeBSD 9 XVA has this software running, on a Xen-capable kernel, with Paravirtual drivers for speed.

If you have a project that requires our help then we'd be happy to talk about an engagement, just get in touch with sales on info@360is.com.

mike said...

Something has broke this :(

Looks like it's hitting this error:
http://lists.freebsd.org/pipermail/freebsd-xen/2011-May/000918.html

and I can't figure out how to boot into the OS at all.

Using the 360is FBSD 9.0_amd64_HVMPV

I believe we upgraded XenServer a bit ago, it is currently XenServer 6.1, and it will not boot. Originally it worked just fine.

Some people say "remove 'ioemu' from the vif config" but there isn't a way I can tell to do that from the boot console or the VM config...

Nick said...

@mike,

This template was only tested on XenServer version 6.0 (the current version at the time). It looks like it doesn't work in XenServer 6.1, so yes something must have changed :-(

If there is enough commercial demand for FBSD 9.0 on XenServer 6.1 then we can put someone on the job.

Currently it seems there is some demand for paravirtualised pfsense and freenas on XenServer.

Jamie said...

xenserver tools is a necessity if you want to attach more than 2 more vhd's.... so like if you want to use freebsd's zfs to pool 3 x 2tb vhds.... you are pretty much SOL.

2tb is the vhd limit, and 2 vhds is the limit for hvm. suxors!

Unknown said...

Doesn't boot on XS6.1 (or XCP1.6). *Sadface*

Anonymous said...

You can get it to boot up on XCP 6.1 by removing the DVD drive from the machine. Follow the steps here: http://support.citrix.com/article/CTX132411