High Performance, Low Latency, Hyper-Converged Computing

Recently 360is implemented several systems for clients who needed very high performance, within a stated budget, and had limited physical space and power to work with. For these clients we designed hyper-converged compute/storage units built from non-proprietary, commercial off the shelf components, supportable by their in-house IT team. Thanks to recent advances in storage technology it is now possible to obtain very high performance for a fraction of the cost of a traditional Server + SAN approach. Better still, these systems aren't subject to the vendor’s ideas of life-span (often artificially foreshortened), and can remain operational for 5, 10, or more years if required. You the customer, remains in control.

• 70GB/sec streaming transfers, 4M IOPS, 4U of space, 5TB to 250TB raw capacity, 2.5PB per rack
• 2GB/sec streaming transfers, 480TB raw capacity, 4U of space. 4.8PB per rack
• 75% less power for a given performance level
• 3X to 6X the performance when compared to similarly priced Server + SAN
• On-site spares for instant access to replacement parts, forever
• Scale-out capability with clustered filesystems like Lustre, GlusterFS, and Ceph
• No chance the vendor can make the systems obsolete

If you are challenged to provide performance, either on-premise or in the cloud, then a hyper-converged system may be for you, and will certainly have a longer lifetime without vendor or service lock-in. For a fixed cost, a properly designed hyper-converged system will always deliver significantly more performance than Server/SAN systems. Let us know your constraints and we can give you an immediate indication of whether hyper-converged is for you.

About 360is

Our scientific approach to performance analysis and engineering has been proven in previous engagements. We work with top 5 Investment Banks, Telcos, and technology vendors. If you have an IT performance problem that is impacting your business, contact us to arrange a no-obligation meeting with one of our consultants.

XenServer Creedence Alpha 3, Disk I/O testing (part 2)

We did some more testing of XenServer Creedence Alpha (XSCA3) disk performance, and plotted large streaming reads for a variety of record sizes against both a physical and Brand-X Hypervisor.

Recap:

• System is an AMD6176SE, 2 CPU, 192GB RAM 
• Local storage, 3x 10Krpm SATA, LSI 9261-8i, RAID0, thick provisioned 
• No special settings, tuning, or configuration 
• Testing is with dd and iozone, with and without Direct I/O (dd iflag=direct, iozone -I)
• CentOS 6 2 vCPU, 2GB vRAM (updated 3-07-2014) VM and physical 
• The system was idle 

Physical achieves ~600MB/s transfer speed. 
Brand-X achieves a similar figure.
XSCA3 achieves less than 50% of that, unless Direct I/O is used.
Neither physical nor Brand-X are significantly affected by use (or not) of Direct I/O.

Results for physical without Direct I/O are excluded as with 192GB RAM and only 8GB of test data, transfer rates are in the 2500-1700 MB/s range due to the abundance of RAM for cache. We took no steps to limit the physical CentOS to 2 cores either.
  
We know the disappointing XenServer performance is only for asynchronous (not Direct I/O) disk access, and that the system behaves as expected when running physical or Brand-X hypervisor. The mystery deepens!

XenServer Creedence Alpha 2, Disk Performance

360is gets paid to make information technology go faster.

Sometimes its hardware which doesn't hit the stated performance, or software which cant fully utilise the capability of modern hardware. Sometimes it's a lifetime extension for an old platform, squeezing in another 18 months growth before a replacement arrives. If we are really lucky we get to re-design an entire end-to-end process and make it more efficient. More layers and more abstraction means more scope for performance problems, so virtualisation has been a rich seam for us. With Citrix release of XenServer Creedence Alpha 2 (XSCA2) should we be worried? Is it time to throw in the towel on IT performance-tuning and setup that high-end bicycle-shop-come-espresso-bar we've always talked about?

We've been following XenServer performance from the start, and have a tome of magic spells to instrument and improve network, storage, and CPU performance. Without resorting to black-magic we were interested in seeing how XSCA2 performed straight out of the box.

Firstly let me say that all we have time for here is the most superficial of testing. Large sequential reads and writes are the 0-60 time of the storage world. That is to say, while they have some value, unless your use-case is an out-and-out drag race this test probably isn't a good approximation of the kind of performance you will see in your applications. Single VM large sequential read/writes are even more of a corner-case. If you only had a single VM to run you should probably run physical, just a suggestion...

Secondly, XSCA2 is alpha, and so it is slightly unfair subjecting it to a performance benchmark.

Finally, we used the equipment we had spare in the lab at the time. The storage back-end is puny. We had a handful of 10Krpm spindles and SSDs laying about. Out in the real world, 360is regularly deliver 1.5GB/sec to 2GB/sec of storage bandwidth (at high IOPS) to Hypervisors and physicals of one kind or another either over local or network storage.


The Goal
We were interested to see how XSCA2 performed against XenServer 6.2, against physical, and against "Brand-X" Hypervisor, all of which were "out of the box".

The Test
The test couldn't have been simpler. For a 2 vCPU VM, for each of 9 record sizes (64KB-16MB), we write (or read) 8GB of data and measure the performance in MB/sec for each record size. Why 2 vCPUs? Adding more doesn't change the results. Why 8GB? We can be sure 8GB blows through any caching that may be happening on disks, RAID controller, VM, or Hypervisor. Even at a 16MB record size, 8GB takes a lot of writes. For the physical test case we force direct IO to get around the fact that the physical system has much more RAM than 8GB. We use the same guest Operating System, installed in the same way for each of the VM tests. Everything is thick provisioned. This isn't a test of how fast each configuration can be made to go, it is a test of how fast each actually goes, straight out of the box on the lab system that was available at the time.

Tuning
None. No changes to the default install of XenServer, Brand-X Hypervisor, the CentOS VM or physical instance, with the exception of taking XSCA2 out of debug mode. No CPU pinning, no IO scheduler changes, no disk/virtual disk alignment, no IRQ balancing, no interrupt coalescing, no filesystem tweaking, no queue size alteration, no waving of dead chickens or reciting of incantations.

Results  
Enough talk, on with the results:

8GB Streaming Write At A Variety Of Record Sizes

8GB Streaming Read At A Variety Of Record Sizes

On this system, for this test, XSCA2 is an improvement over XS61-SP1, but is still significantly behind the physical, and more disappointingly behind the other well known brand of Hypervisor. Besides the obvious, there are a few points from the chart which warrant further investigation for starters:

• High jitter in all XS results.
• Odd dip at the 512KB record size test on both XSCA2 and "Brand-X" hypervisor.
• Slow start to the physical test at 64KB record size.

  The tests shown here were on a RAID0 of 3x 10Krpm spindles (maximum sustained transfer rate ~200MB/s each). Conducting the same test on a RAID0 of SSDs made little difference to the XenServer results, adding 20MB/s to the average write result and 40MB/s to the average read value.

Conclusions

1. We aren't out of the performance tuning business just yet it seems!
2. There is a significant difference in performance between the physical and "Brand-X" and XenServer.
3. Read performance is particularly disappointing for XenServer in this test.

"It is easier to repair a bucket with a big hole, than an inner tube with a slow puncture." - Ancient 360is Engineer's Proverb.

For this system, for this test, the hole in the bucket is large, with a bit of further investigation it shouldn't be too hard to find. XenServer Dom0 (which strictly speaking we don't care about) comfortably achieves ~600MB/sec in read performance tested using "dd" with direct IO (no cache effect), so we know the problem is with the guest disk virtualisation IO path. First port of call will be instrumenting CPU consumption in the guest and Dom0, paying particular attention to XSCA2 susceptibility to numa-effects on the CPUs. We love a mystery. The game is afoot!


Further Information

Test VM Spec.
CentOS6 x86_64 Linux, default install from distribution, updated with "yum update" 3-07-2014, with the following additional packages: wget, openssh-clients, iozone (3.424-2 x86_64). 2 vCPUs, 1GB RAM, 20GB virtual hard disk.

Test Hardware Spec.
AMD 6176 CPUs (x2), 192GB 1066MHz RAM, LSI 9260-4i RAID, 3x WD1000DHTZ, 2x SSDSC2BW12.

Test Hardware OS.
CentOS6 x86_64 (same as VMs).

Brand-X Hypervisor.
Latest version, chose the PV SCSI device.

Benchmark.
We used the continuous benchmarking feature of VMCo Virtual Estate Manager (VEM). VEM's benchmarking alerts administrators to performance regressions in your XenServer or VMware estate, whether they be caused by bugs, patches, hardware problems, subtle interactions between network elements or administrator misconfiguration. VEM's continuous benchmarking shows you where the performance regression is, when it started, and it's impact is.

 

Higher Hard Drive Prices, Still The New Normal

About 7 months ago, back at the end of Q1 2012 we researched hard drive prices and compared them to pre-flood levels. We concluded that higher prices would be here to stay for the next few years for a couple of reasons:

• Nobody is investing in next-generation facilities with only 2 serious players in the spinning rust market, there is no need reduce manufacturing cost in the medium term with next generation factories or processes.

• A lot of the specialist tools and precision equipment used for testing and building hard drives are no-longer available. The firms that made them have long since gone bust as the market consolidated in the last 15 years. Lack of such equipment is a barrier to increasing production and (re)building plants.

• While Solid State Disks continue to march into mobile, desktop, and enterprise storage, the bulk of new added capacity will continue to be provided by spinning disks. In the enterprise data centre they form either a thin tier or are deployed tactically to take care of IOPS/latency hot-spots. Hard disks in the enterprise are not about to be wiped out by SSD storage.

The bad news is that the majority of drives are still more expensive now than pre-flood. Some are around 60-70% more expensive than pre-flood levels.

If your organisation or project is hamstrung by storage costs or performance, talk to us. We have helped companies in Finance, Life Science, Academic, and Software sectors accomplish and exceed their goals, on-time and on-budget, in spite of rising storage costs. Get in touch to find out how.

The New Normal For Hard Drive Prices

Hard Drive Market Share (by units) & Consolidation CY2011

In our 2011 end of year message we touched on problems in the hard drive supply chain due to the serious flooding in Thailand starting late July 2011 and running through the rest of that year. At the time, the expectation would be that we would see prices return to normal in Q2 or 2012. It now looks like hard drive prices will not return to pre-flood levels. The evidence is that higher hard drive prices are "the new normal".

Consolidation in the hard drive industry, coupled with a prolonged period of low margins most likely means that prices will not return to their previous levels. So if you have been holding off on a storage upgrade, there is little incentive to keep waiting. The hard drive market has seen at least 2 rounds of mass extinction or consolidation over the last 20 years with tens of companies exiting the market, leaving just 3 manufacturers standing. Together Western Digital and Seagate have 87% of the market leaving Toshiba a distant 3rd. Nobody is investing $500M to build a new factory in order to lower their manufacturing costs and put one over on the other 2 guys.

While the take-up of SSDs continues in mobile devices (laptop, tablet), and they make an appearance in the enterprise as specialist devices or as a thin tier above enterprise SATA/SAS drives, "spinning rust" will continue form the bulk of all storage shipped in terms of Terabytes for as far out as anyone is brave enough to predict. Seagate has publicly stated it will be able to produce a 30-60TB 3.5-in. hard drive by 2020.

As an aside, anyone who has waited for a RAID5/6 rebuild on an array of full 1TB drives will know, increasing data density without increasing interface speed brings its own challenges, as does silent data corruption and the need to counter it . If you are worried about either of these problems then we can help you avoid them!

If you need to get more performance from your existing storage, or are struggling with data volumes, or if you simply require impartial advice ahead of making a purchase from one of the big storage vendors, we can probably help you. 360is has experience with environments of all sizes and have completed successful projects with most of the major vendors including  NetApp, EMC, Hitachi, HP, Sun/Oracle, Dell, 3PAR, Datacore, Westek, and Nexenta. Get in touch to get ahead of your storage problems.

Performance Expert Services

Over the last 12 months we have seen a sharp increase in clients targeting poor performance of systems and applications, particularly around storage, virtualization, and wide area networking. As a result of which we have formalised our professional services for performance investigation, reporting, and remediation. We call this our Performance Expert Service.

We provide fixed-fee projects where our consultants work either independently of your vendors, or with their assistance, to get the performance you need from mission critical systems. When more performance cannot be liberated from existing assets, we are able to provide a quantified case for additional investment, couched in business terms.

While everyone strives for more performance, it is only since the credit crunch and economic slowdown in the UK that there has been a significant increase in these projects for 360is. We put this down to factors impacting IT departments like reduced staffing, frozen budgets, and lack of visibility into the future. Not since the great Y2K spending freeze has there been such a focus on making do with what you have and ensuring it runs efficiently. Bad news for product vendors, but not necessarily bad for end users. Many found the Y2K hiatus in new IT deployments to be no bad thing, some even said IT had never worked so well.

Benefits

• Get next years hardware performance now. (useful if your capex budget has been frozen)
  
• Free up staff from nursing overloaded systems. (good if your team has recently shrunk)
  
• Reduce license costs through higher utilization of fewer systems. (interesting if you just got your support renewal quotes)
  

Our consultants are there for when performance problems defeat your IT team’s efforts, and are beyond the scope of vendor patches and support contracts. Contact us to find out how we can solve your performance problems.