Tuesday, June 18, 2013

A Quick Enumeration Of The PRISM Program

Enumeration of PRISM
As IT security consultants, former representatives at the GSMA, contributors to IETF's "Raven", and one-time employes of the worlds largest ISP, the inevitable questions started coming up in meetings and conversations shortly after the 8th June.




"...two reporters from the Guardian newspaper announced to the world the source of one of the most significant classified-document leaks in history. Edward Snowden, a 29-year-old national-security contractor from Hawaii, revealed that he was compelled by conscience to inform the world about a massive abuse of authority perpetrated by the US National Security Agency. According to the documents Snowden provided, which have been authenticated, the US government has been systematically collecting the phone records and online communications of millions of American citizens for years."

Clients, commentators, and friends all want answers to the same questions:
  1. What are the NSA doing, and how are they doing it?
  2. What does this mean for IT security?
  3. Where does this go next?

While we can expect this story to develop over many months as more information is dripped out,  we can already go some way towards answering question (1). See the 360is enumeration of PRISM possibilities. PDF.

Where will the story go next? That this story will run and run is something of which we can be sure. It has elements of Manning/Wikileaks (an ethically conflicted individual with access to state secrets), of Leeson/Barings (a young man, on the run from authorities in a foreign land), and just enough direct relevance to UK readers through glimpses into the actions our own GCHQ, who are normally more publicity shy than their US counterparts. The UK intelligence services have been relatively fortunate in recent years, suffering  few leaks, disgruntled former employees, or clumsiness. This episode illustrates both the frequency and extent to which intelligence sharing occurs between the UK and US and the fact that occurrences such as the Snowden event can have an impact for both services with unintended consequences that cannot be anticipated easily.

Timing is everything,  the publication of these revelations may have some bearing on future of The Draft Communications Data Bill or "Snoopers Charter" as it is commonly known. With well publicised recent convictions of a number of terrorist individuals and groups, significant good-will had been earned among the UK public. Any poor handling of the existing Snowden disclosures, and further leaks that are yet to come, may diminish this good-will and make introduction of such a bill more difficult in the future. Futhermore, the disclosure of GCHQ's activities around the 2009 G20 summit in London is bound to have an impact at the G8 event currently in-progress in Northern Ireland.

Looking further out, companies and individuals in Europe may seek to engage with services and providers that do not come directly under US law. Apple, Microsoft, and FaceBook (3 out of many providers) have stated that they handed over data from 10000, 32000, and 19000 accounts respectively in 7 months. US cloud providers are already having trouble persuading European enterprise customers that their sensitive (but completely legal) data is safe from US government spying. This leak will only make matters worse. As one journalist put it "Not subject to American law' - the next desirable IT feature?"

All that we need now is a catchy name for this episode. Dotcomgate? igate? How about "Cloudgate"?

360is will be presenting further analysis of "Cloudgate"and advice for UK organisations in the next edition of Executive Intelligence, our quarterly for UK CSOs and Information Security Managers.

Related Postings:
WikiLeak's Lessons For UK Information Security Professionals.

Update Saturday 8 June 2013 18.56 BST: Leaked NSA slides confirm, PRISM includes direct monitoring of fiber cables and collection directly from the servers of MS, Yahoo, Google, and Co.

Monday, November 12, 2012

Higher Hard Drive Prices, Still The New Normal

About 7 months ago, back at the end of Q1 2012 we researched hard drive prices and compared them to pre-flood levels. We concluded that higher prices would be here to stay for the next few years for a couple of reasons:
  • Nobody is investing in next-generation facilities with only 2 serious players in the spinning rust market, there is no need reduce manufacturing cost in the medium term with next generation factories or processes.
  • A lot of the specialist tools and precision equipment used for testing and building hard drives are no-longer available. The firms that made them have long since gone bust as the market consolidated in the last 15 years. Lack of such equipment is a barrier to increasing production and (re)building plants.
  • While Solid State Disks continue to march into mobile, desktop, and enterprise storage, the bulk of new added capacity will continue to be provided by spinning disks. In the enterprise data centre they form either a thin tier or are deployed tactically to take care of IOPS/latency hot-spots. Hard disks in the enterprise are not about to be wiped out by SSD storage.
The bad news is that the majority of drives are still more expensive now than pre-flood. Some are around 60-70% more expensive than pre-flood levels.

If your organisation or project is hamstrung by storage costs or performance, talk to us. We have helped companies in Finance, Life Science, Academic, and Software sectors accomplish and exceed their goals, on-time and on-budget, in spite of rising storage costs. Get in touch to find out how.

Monday, September 03, 2012

Why VMware (still) Wont Convince The Cloud Providers


We don’t normally comment on commentary at the 360is blog. Most of our postings are more substantial, whitepapers, guides, seminars, or software we’ve produced or packaged. However there is a good posting (Cloud Hosting & Service Provider Forum) by Richard Talaber (ex-CTO’s office at VMware) on LinkedIn and thought I’d reflect on it here.

For those of you who aren’t on LinkedIn, or don’t want to join the group in question, Richard was responding to Beth Pariseau’s article at TechTarget on VMware’s recent abolition of the vRAM tax.

Richard makes the case that VMware is probably not going to be the right virtualisation stack for a commodity “$75” per-month per-VM hosting/cloud service, vRAM tax or not. He also believes that VMware still makes sense for the smaller market for high-value fully-managed VM hosting, with load balancing, monitoring, fault tolerance, and high performance throughput bells and whistles.

Mostly I agree with Richard, he makes sensible arguments supported by good assumptions but in-spite of this I still see more than a few problems for VMware in the short and medium term. Let me explain...

All service providers, including Cloud & Hosting providers, need to own their own infrastructure if they are to have a sustainable business. Whether it be a fibre optic network (in the case of a carriers) or their provisioning, monitoring, and management stack in the case of Cloud providers. Unless a service provider owns his own infrastructure, he cannot exercise full control over his costs. Cost control is vital in a service provider business, as profits depends upon volume, and any tax on profits is unwelcome, especially one that grows in-line with that volume.

Hardware must be paid for (hopefully using inexpensive long-term debt secured against assets you also own), but if there is even a chance of finding a cheap or inexpensive hypervisor-and-management-stack then Cloud providers have to take it. This is why you see so many of them with Xen, XenServer or KVM. This is why the list of users of OpenStack reads like a who’s who of Telecoms and Hosting. For those that argue I'm not considering TCO, think about this. I've hired good technical guys, pay them well, they work hard for me, I'm supposed to be a player in Cloud. If my guys can’t engineer something solid, maintainable, and cost effective then what am I doing in this business? These guys and the platform they build from KVM, Xen, or whatever, are my long-term competitive advantage. At least until I'm big enough to be building my own data centers from scratch.

If I'm a Cloud provider and my maintenance renewal (or any other per-unit-customer cost) jumps even 1%, its a big deal, I've got a bazillion systems after all.

For parts of the infrastructure that can’t be wholly owned, or cannot be had for free (with effort from my hardworking DevOps guys), the Cloud provider's only weapon is over-subscription. Pay for the product then figure out how to dilute that cost by maximising over-subscription, balance providing a great service with running the hypervisor and server hot. It is a very difficult balance. This is one of the reasons the vRAM tax was so wildly unpopular with Cloud providers, it took away one of their profit levers, or at the very least shortened it.

Hosting/Cloud providers are currently at a very immature stage in life, they are talking about RAM, and IOPS, and vCPUs or is it CPUs or is it Cores or Threads? What do any of those things mean? What about transfer rates, or latency, what kind of cores? What is a thread? These things don't mean much to business people so good luck trying to explain them. When business sees 3 prices for what they perceive as essentially the same service, they are going to go for the cheapest and find out later if it was appropriate. By Richard’s own admission, 80% of workloads are relatively modest in demands. VMware will not manage to get across an argument based upon performance; a performance lead of the size VMware achieves (or even aspires to achieve) is not a sustainable advantage for the Cloud provider market. 

Richard's last sentence is key: "Perhaps VMware should consider a public cloud price that is significantly less expensive than a private cloud price.". Logical. If the public cloud VMware price were somehow so small as to barely matter, then service providers would not spend time fiddling around with the competition or knocking up free alternative platforms....but how to segment the product? I think it is too late now, there are alternatives in-use, there are engineers out there with experience of building these infrastructures for service providers, and even if you can't afford to buy such people you can probably rent them. Talk to us.


  • Better performance is not a sustainable advantage for VMware.
  • A richer feature set probably isn’t either.
  • Cloud providers dislike anything that handicaps their ability to over-subscribe.
  • Cost that grows in-line with customer volume is a no-no, unless it is absolutely impossible to avoid.

VMware could make their product more appealing to commodity Cloud providers, but in order to do so they’ll have to start thinking more like them. Or talk to someone who does. 360is has helped companies like CheckPoint, HP, and Microsoft understand the Cloud service provider market. You know where to find us.

Wednesday, July 25, 2012

FreeBSD 9.0, Paravirtualized Drivers, & Xen-Tools On XenServer 6


We are often asked by clients to help them squeeze more performance from an existing infrastructure, to speed up an application or shorten an IT-dependent business process. Early on in the discussion there's a tendency for sysadmins and architects to dive in to technical minutiae, the black magic and chicken-waving of memory page sizes, block alignment, cache segmentation, and thread pinning. This is almost always a mistake at this early stage and can drain days and weeks with few results. The business gets frustrated when deadlines are missed and the techies can't answer simple questions like "when will this be fixed?", "how much sooner will I get my results?", and "what will it cost to process data within that time window?". Black magic has a time and a place, but at 360is we save our voodoo until the latter stages of a performance tuning project.

Before making non-portable, hard-to-maintain, unsupported, obscure, or fragile configuration changes that break when you least expect them (and may not even be understood by those operating the infrastructure) we start with the basics:
  • Is what you have, setup properly? Many complex multi-vendor data centres have a setup that is sub-optimal in some way.
  • Is the system already performing more-or-less as you would expect "on paper" +/-25%? We whiteboard a block diagram of devices, buses, networks, data volumes, benchmarks.
  • If it isn't then there is probably a significant configurational mistake to be found. Forget about jumbo frames if your switch is stuck in half-duplex mode.
  • Focus on the biggest wins first, they are often the easiest to achieve. Only then do we go chasing marginal gains with our consultant's Juju. Don't waste days chasing the last 3% unless that gain makes economic sense for your business process.
Where are we going with all this and what does it have to do with FreeBSD?

At 360is we are fans of FreeBSD, and regularly recommend it as a secure, low-maintenance, stable, and performant, server operating system. Unfortunately there is no official support for FreeBSD in the Citrix XenServer product. What this means in practice is that those wanting to run FreeBSD on XenServer deploy it in HVM mode and without Xen-Tools. Pure HVM mode is slow for network and disk access, and no Xen-Tools means no live migration, not good for production workloads.

Shooting for the biggest win first, we have made available a basic, paravirtualized-drivers-with-xentools-installed, FreeBSD 9.0 64-bit Template that delivers approximately twice the performance of the pure HVM install of the Operating System. No tuning, no special settings, absolutely no chicken-waving.
  • FreeBSD 9.0
  • 64-Bit (amd64)
  • Paravirtualized Drivers in the XENHVM kernel
  • Open Source Xen-Tools pre-installed
  • Small (389MB) XVA file
Bugs/Errata: Hot-adding of additional NICs works, hot-remove not so much. Use at your own risk. Please submit any other problems as comments, feedback is always welcome.

Get the FreeBSD 9.0 XenServer Template XVA, (cookies/valid Email required in order to be sent the password).

If you need to extract more performance or reliability from an existing infrastructure, application, or IT-driven process, 360is consultants know how, get in touch.

=== Update 26-07-12 ===
As we always get asked about these "relative-to" bar charts, the absolute figures were 188MB/sec for the VM derived from our template, and 92MB/sec for the ordinary install using Citrix "Other OS" template. The fast VM consumed 67% of 1 vCPU on an otherwise idle system. The physical hardware used was a VMCo VA12xx Appliance with its local IOPS sink configured as an SR.

=== Update 11-10-12 ===
This XVA was prepared for XenServer 6.0.2, and probably wont work on XenServer 6.1. If you have a commercial imperative for it on a different version of XenServer, with a different version of FreeBSD other than 9.0, or for that matter with i386 versus amd64, then get in touch with our project office.

Monday, May 28, 2012

The New Normal For Hard Drive Prices

Hard Drive Market Share (by units) & Consolidation CY2011
In our 2011 end of year message we touched on problems in the hard drive supply chain due to the serious flooding in Thailand starting late July 2011 and running through the rest of that year. At the time, the expectation would be that we would see prices return to normal in Q2 or 2012. It now looks like hard drive prices will not return to pre-flood levels. The evidence is that higher hard drive prices are "the new normal".

Consolidation in the hard drive industry, coupled with a prolonged period of low margins most likely means that prices will not return to their previous levels. So if you have been holding off on a storage upgrade, there is little incentive to keep waiting. The hard drive market has seen at least 2 rounds of mass extinction or consolidation over the last 20 years with tens of companies exiting the market, leaving just 3 manufacturers standing. Together Western Digital and Seagate have 87% of the market leaving Toshiba a distant 3rd. Nobody is investing $500M to build a new factory in order to lower their manufacturing costs and put one over on the other 2 guys.

While the take-up of SSDs continues in mobile devices (laptop, tablet), and they make an appearance in the enterprise as specialist devices or as a thin tier above enterprise SATA/SAS drives, "spinning rust" will continue form the bulk of all storage shipped in terms of Terabytes for as far out as anyone is brave enough to predict. Seagate has publicly stated it will be able to produce a 30-60TB 3.5-in. hard drive by 2020.

As an aside, anyone who has waited for a RAID5/6 rebuild on an array of full 1TB drives will know, increasing data density without increasing interface speed brings its own challenges, as does silent data corruption and the need to counter it . If you are worried about either of these problems then we can help you avoid them!

If you need to get more performance from your existing storage, or are struggling with data volumes, or if you simply require impartial advice ahead of making a purchase from one of the big storage vendors, we can probably help you. 360is has experience with environments of all sizes and have completed successful projects with most of the major vendors including  NetApp, EMC, Hitachi, HP, Sun/Oracle, Dell, 3PAR, Datacore, Westek, and Nexenta. Get in touch to get ahead of your storage problems.