Storagebod's Blog

Flexible Thinking

I think that there is some interesting discussion to be had from Hu's latest blog in response to my own comments and thoughts about whether storage virtualisation as demonstrated by the external storage virtualisation devices has a long-term future.

The response is not really to do with virtualisation at all; it is all about aligning IT and Business. He talks about an all too familiar case where storage decisions are made locally by the Business Units and the procurement strategy does not take account of the long-term health of the group; ongoing OpEx costs are not born by individual business units and become the problem of the IT department. The concept and value of shared infrastructure was not really understood by the Business.

But I suspect that these very same Business Units will be happy to use Cloud-based services, Infrastructure as a Service etc, concepts which are built on multi-tenanted shared external infrastructure. Why? Because they can deploy rapidly and flexibly; they don't need to engage the slow and cumbersome IT department and they feel that they are masters of their own destiny.

Storage virtualisation won't bring these Business Units back but storage virtualisation may enable IT to put together a flexible and responsive service catalogue. CIOs need to engage with their customers and understand what they want but they need the support of vendors to clearly demonstrate cost to their customers and to their boards.

The service must be something that the customer wants, customers do not understand why it takes weeks to provision servers, storage and networks. They can go down to PC World and the likes and pick things off the shelf *NOW*.

The more savvy customer knows that they can enter a credit card number into a cloud provider and can provision dozens of servers much quicker than the IT department can and what's more, they can turn them off again equally quickly and not be stuck with kit that they do not need and do not want.

Customers want dynamic, flexible infrastructures which can rapidly respond to their needs; Corporate IT departments along with their vendors have been pretty poor at providing these. Virtualisation and abstracted infrastructures are enabling technologies but Hu's right in that people and processes are key...

As for external storage virtualisation devices, I still wonder if they are the future? They are not necessary to provide the service and if I was building a green-field data centre with no legacy to deal with, I am not sure I would deploy them.

Infrastructure provision is at an interesting inflection point; if you don't understand this and your customers do, you have got a problem.

First, the worst....

Who copied who? Does it really matter? Michael quotes a Sun blog where Sun appear to claim that they beat the SP2 with the Sun Enterprise 10K. It's actually quite an amusing piece both the article quoted and the article itself.

Firstly, I must out myself; I worked for IBM resellers in the late 90s working on RS/6000s and SP2s and then I was the Data Centre architecture for a company which made extensive use of SP2 technology in a commercial environment; so I have a bit of soft spot for them.

Indeed, I was a certified SP2 specialist for my sins but I have worked on most type of Unix box. So please take my comments with a pinch of salt.

From Ken Ow-wing's blog

The reminds when when when IBM clustered a bunch of RS 6000s togeather around a moderately performant switch/network, called it an RS 6000 SP2, and  placed Oracle Parallel Database on it for the backend database for applicatioins such as SAP R3.Sun created a competitive program to compete with the  IBM SP2 with the earliest  ancestor of the Sun Enterprise 9000 server, The Sun Enterprise 10000. which went GA 1997. This was Sun first Supercomputer based on the designs acquired when Sun purchased the Business Systems Divsion of Cray Research (the first and foremost SuperComputer company of that era).

You may notice that the Sun Enterprise 9000 is still around and you do not hear much about IBM SP2s. Sun won. I will spare you more details.

The SP2 does still exist but it's not called the SP2 any more; the SP2 is now the Cluster 1350 and Cluster 1600; that's why you don't hear much about it, it's changed its name! It is an MPP Cluster and indeed a completely different beast to the Sun Enterprise 10K.

Although it did at times get deployed in a commercial environment; that's not what it's sweet spot was and OPS was a real pig to get working! It was massive in the realms of Seismic and Scientific processing; I had colleagues whose job was to install SP2s on survey ships.

And before the days of Gigabit ethernet; if you wanted to chuck large quantities of data around between your nodes; the SP2 switch was an excellent way of doing so! If an SP/2 was deployed in commercial, it was more often to utilise the switch and excellent (but rather unique) management tools to enable a large number of servers to be managed from a single point.

Much of the technology from the SP2 has found it's way into the BlueGene devices. The SP2 lives on!

The Sun Enterprise 10K was an excellent large SMP box and IBM did try to compete using the SP2 but it was always tough because managing a single box was always easier than managing a whole bunch and coding for a single box is often easier than trying to write MPP code. But if you really want to do massively complex deep computing which needs thousands of cores, MPP is often the way to go.

However IBM did manage to get their SMP story together and Sun never really got their MPP story together; IBM can now offer excellent SMP performance and couple it with their MPP know-how to deliver clusters of large SMP devices.

Sun have a 1% share of the TOP500 Supercomputers, IBM have more than a 37% share of the TOP500 Supercomputers using technology derived from the SP2 technology. That doesn't look a defeat to me!

So perhaps IBM did copy Sun and decide that they needed large SMP boxes but they took the idea and ran with it; coupling it to their own strengths and produced something which was greater than the sum of the parts. And hell, it wasn't Sun trying to buy IBM at the beginning of the year was it?

Perhaps EMC are copying HDS but perhaps they too will run with it and produce something better? Innovation in IT is rarely revolution; it is always building on what has gone before.

As a customer, I really don't care who is doing the copying; it's not art and I don't need the original, I just need the one which works best! Tell me why yours is better not that you were the first!

As kids, we used to have a rhyme...

 'First, the worst!
 Second, the best!
 Third, the one with the hairy chest!'

Sometimes it's not best to be the first!

Set the Wide Stripes Free

There have been a couple of articles recently on the HDS blogs about HDP; Hitachi Dynamic Provisioning is HDS' thin provisioning offering and like all of the thin-provisioning, it also offers wide-striping. I am not going to get into whether HDS' offering is chubby, skinny or just slightly overweight but what I am going to ask is...
if wide-striping is so foundational and so important to the storage industry and especially to improving my TCO as a end--user, why do I have to pay extra for it?

HDS and EMC are both extremely guilty in this regard, both Virtual Provisioning and Dynamic Provisioning cost me extra as an end-user to license. But this is the technology upon which all future block-based storage arrays will be built. If you guys want to improve the TCO and show that you are serious about reducing the complexity to manage your arrays, you will license for free. You will encourage the end-user to break free from the shackles of complexity and you will improve the image of Tier-1 storage in the enterprise.

I understand that you feel that you have to maintain the legacy architectures and designs that you have inflicted upon us in the past but it is time that you stopped enabling our mashochistic tendencies and it is time that you encouraged us to move away from the pain of the past. You can do this by removing the licensing costs for wide-striping; keeping the cost for thin-provisioning is just about acceptable but charging for this key simplifying technology is not!

And if one of you does it first; it's okay to copy! Really it is!! I can feel that we're going to be friends.

More Virtual Discussion

Barry Whyte writes on one of my favourite topics 'Storage Virtualisation'.

Now, I do actually think that both SVC and the USP-V have a place in the data centre; they shouldn't but they do! They (and the v-Series working in block from NetApp) provide a common layer of abstraction to the various storage devices which exist in a data centre. If you have more than one type of storage or indeed more than one array and especially if you are attempting to tier data across arrays of many types; you should probably be considering these.

If you are going to attempt a mass data migration, the SVC is something that should certainly be considered due to the relative ease of getting into SVC and getting back out of it. IBM have thought long and hard about the migration use-case for SVC and added the necessary functionality to get out of SVC. A sensible move because one of the major objections to most major virtualisation projects is; what do I do if it doesn't work for me? How the hell do I get out?

Also, virtualisation devices allow me to drive up utilisation by enabling pockets of storage dotted across arrays to be pooled up and used. In fact, often this is the major selling point of the virtualisation devices; it makes the previous unuseable, useable. However, I would argue that these little pockets of storage are an aberation caused by the legacy architectures caused by the traditional arrays and how they are carved up into logical (some would say virtual) disks.

The virtualisation devices can take these little pools of disk and like a baker with pastry offcuts; roll them back into a ball to be re-used. However, you do have to be careful when collecting your pastry offcuts that you don't end with a ball which doesn't stick together especially well and has a habit of flaking and breaking when baked.

This could well be your experience when rolling up your little pockets of storage; disks from different arrays and vendors will perform in different ways. Spreading virtual devices across multiple arrays from different vendors may cause some interesting performance challenges. If the array with the pocket of storage already is fully utilised from an I/O point of view; attempting to use the disk might well have a negative impact on application performance.

Actually spreading a single application across multiple arrays needs careful planning and consideration. It will complicate your change management, problem determination, configuration management and many other necessary processes. You still need to maintain the underlying arrays and understanding the impact of an outage to one of these arrays when you have fully virtualised might be interesting; an outage to a single array could take out your whole data-centre environment.

You might find you get more benefit in the long term by considering an array architecture which inherently enables you to use disk in a more flexible and efficient manner. You may also find that using automation and allowing the array to make decisions about where it puts data drives up efficiency.

So as we move to more scalable, efficient and automated environments; I wonder if we will look back at things like USP-V, SVC etc as a cul-de-sac driven by today's necessity! Or perhaps they truly are the future?

Storage Wars

Is storage the last IT hardware competitive battle-field?

Is it the place where vendors can actually really differentiate between products?

If you look at the server market, pretty much everything comes in flavour of Intel (or AMD) but there's not much between any of them. It's not the hugest hassle to swap an HP environment out for an IBM environment (although listening to some people, you would think the world would come to the end if you were to switch server vendor) and with a hypervisor in the mix; life becomes even easier. When was the last time a start-up became a major player in the server world?

But with storage, there is still everything to play for; start-ups can still get a toe-hold and can still innovate.  The established vendors can knock chunks out of each other without damaging their own product line and I wonder if this is why we see some really quite combative blogging between the vendors. And recently I've noticed the trash-talk stepping up a notch.

Hey, this isn't a plea for everyone to get along but more a plea for more reasoned blogging. Less name-calling and more this is our product and this what it can do for you. More of the 'it works like this' and less of 'company X is rubbish'.

And oh yes; it's great to say what a marvellous place your company is to work but it probably won't make me:

1. Buy your product!

2. Buy your stock!

3. Sell you my stock!

Thoughts from the Twitteratti

@ianhf wondered "Why buy a clariion when baby vmax at same price? (ignoring vmax hype'd 'not ready yet' features)"; various people @davegraham, @storagezilla, @smith_Cameron, @basraayman and myself chipped in.

Ian poses a good question and one which gets asked alot but there's actually a fairly simple answer; the Clariion is a better array for most people! No, architecturally it's not as good; certainly not as reliable and it's replication capabilities are not as good but if I was going to parachute a new EMC array into a non-EMC customer and tell them to get on and manage it; I reckon that most IT shops would have a good chance of getting a Clariion up and working without going through a big training exercise and it'd still be okay in six months. Depending on what software options etc, I reckon there's a good chance that the V-MAX would be a huge mess and unmanageable within six months.

So until EMC can make a V-MAX as simple as a CX to manage; CX has a place but once that happens, then I think Ian has an interesting question. But I would ask the question, if FLARE had replication as good as Enginuity; why would I want a V-MAX? And as they both run on x86 now; why not run FLARE on top of the V-MAX hardware?

With Apologies....

"I'm just a little Blue Tin Cloud
Hovering under the money tree
I'm only a little Blue Tin Cloud
Pay mo' attention to little me
Everyone knows that a Tin cloud
Never costs money, no, not a bit
I'm just floating around over the ground
Wondering where I will fit?"

With IBM's announcement of a Cloud in Can! (And I've been a bit unfair, it's not just tin, it comes with a raft of management software as well) I now have an adequate definition of what Cloud is!!

It's a little bear hanging from a blue balloon, having rolled himself in mud and trying to fool the bees whilst he steals all their money, I mean honey!

Stuff Happens!

Somewhere in the world, there are a bunch of sys-admins, DBAs, application specialists, storage specialists, incident managers, service managers, network specialists and probably a whole bunch of other people running round trying to recover service after a data-centre outage.

How much running around, panic, chaos, shouting and headless chicken mode depends on how much planning, practice and preparedness they have for the event. You might not even notice even if you are using the service because if they have done their work properly, you shouldn't.

Outages happen; big horrible nasty outages happen. In a career which now spans over twenty years, I've been involved with probably half a dozen; from PDUs catching fire due to overload to failed air-conditioning to wrong application of the EPO*. I have been involved in numerous tests; failing over services and whole data-centres on a regular basis and for most of these tests, the end-user would not have been aware anything was happening.

So when Amazon loose a data-centre in their cloud, this should not be news! It will happen, it may be a whole data centre, it may be a partial loss. This not a failure of the Cloud as a concept; it is not even a failure of the public Cloud; there are thousands of companies who host their IT at hosting companies and it's not that different.

What it is a failure of is those companies who are using the Cloud without considering all the normal disciplines. Yes, deploying to the Cloud is quick, easy and often cheap but if you do it without thought, without planning, it will end up as expensive as any traditional IT deployment. Deploying in the Cloud removes much of the grunt-work but it doesn't remove the need for thought!

Shit happens, deal with it and plan for it!

* Emergency Power Off switches should always be protected by a shield and should never be able to be mistaken for a door opening button! But the momentary silence is bliss!

Flying South...

No, I'm not going anywhere; well, not yet! I was hoping for a billion dollar take-over bid from someone in light of some of the goings on in the market. Hey EMC or NetApp, whichever one of you looses out; just throw a billion dollars my way!

Actually, what I'd like to talk about is data migration, with HDS' HAM announcement and the promise of seamless migrations forever; easy and smoothly, it seems a killer feature. Indeed EMC are talking about the same capabilities for the V-MAX. As long as you going from V-MAX to V-MAX, USP-V to USP-V and even IBM SVC to IBM SVC; migrations should be outage free and relatively easy; that's as long as you meet all the pre-requisites with firmwares, driver-levels, multi-pathing software, probably operating system levels; migration will be relatively simple, outage free and automagic!

Of course, as soon as you want to go out of family let's say USP-V -> V-MAX; you've got a problem but as long as you want to keep your disk-controllers the same; you are fine. Yes, I know you can use the USP-V or SVC to bring in external arrays but I am talking about fundamental changes to the storage architecture.

Block-based migration can also be achieved at the host level with minimal to no outage by using host-based tools such as volume managers. It is this technique which is probably most commonly used; it is laborious but it is a well-travelled path. So when it comes to block-level migrations, you have options.

I am assuming that you are not taking the opportunity to re-tier; re-layout for performance, stack LUNs, remove dead data and generally do a tidy-up of your storage environment. You are simply going to move one LUN to another LUN.

However, as we are aware; we don't just have block-storage these days; NAS is becoming the default option for many companies. Management of NAS is generally easier, it can certainly be quicker to provision and it's TCO is often lower. It is an attractive option but....it's a pain to migrate seamlessly and without outage!

In the past, we had data on NAS which probaby did not have the availability requirements of the data sitting on our Tier-1 arrays; it was not mission critical but this is no longer the case; mission critical data sitting on NAS is becoming more common and availability requirements for this data are in the five and six nines levels. Taking outages for migration is will not be acceptable to businesses and we need to come up with strategies for seamless migration of NAS data.

There are tools such Acopia from F5 and Rainfinity from EMC which virtualise at the file-level. Isilon promise no more fork-lift upgrades and you simply incrementally upgrade and migrate; as do others. Or clustered file-systems might be the answer? Perhaps using the facilities in the hypervisor, almost akin to what we do on a host for block?

But this is not yet a mature and well understood discipline for most people. And as NAS becomes de-facto, it will need to be. Also, as NAS has been sold on simplicity, reduced management costs etc; it is going to have to be easy.

Who do I want to win?

So now EMC have entered the bidding for Data Domain; who do I want to win? Actually neither of them, I think I'd prefer someone else to get them!

So why do I want neither of them? Firstly, I must declare that I'm a NetApp and an EMC customer which colours my opinions somewhat.

When it was just NetApp bidding, I was quite content for them to get Data Domain. It strengthened their portfolio and would allow them to compete just a little more with EMC. Now EMC have entered the fray and with a much improved offer; I'm not so keen and NetApp actually getting Data Domain could hamper their ability to compete, it would eat into their cash reserves entirely too much. It would weaken their position and I suspect EMC could force NetApp to significantly overpay if they let hubris get to them.

Also NetApp do not have a great track record of integrating their acquisitions and driving value out of them; so I'd really rather them see to concentrate on building their core and perhaps look different acquisitions. But I think they should have another bid, just to ensure that EMC end up paying a bit more!

Sure, it'd give NetApp an issue competing with a product which they wanted but better that than running out of cash!

However, I really would prefer not to see EMC get Data Domain; EMC do compete with Data Domain in the dedupe space and I'd prefer to see it kept that way. I don't think it would be especially good for the competitive landscape for EMC to get them.

I think I'd prefer HP to pick up Data Domain and although HP really don't need another storage product, I think Data Domain might well fit quite nicely into their line-up. And hey, at least if HP entered the bidding, it might ensure that EMC end up overpaying instead of NetApp.

Oh well, sure is fun sitting on the sidelines. Who's going to be next? I like the idea of Farley become EMC's new star blogger! But that's just for comedy-value!