Storagebod's Blog

Sign in Triplicate please....

And so it begins; looks like SunOracle might be the first vendor to start shipping triple parity in their storage devices; scarey isn't it? I wondered who would be first because as we start down the route of 2Tb spindles and beyond, dual parity wasn't going to be enough; rebuild times etc were just going to get more frightening.

Now, everyone could have just decided to go down the XIV route and mirror everything; RAID-1 with chocolate chunks I like to think of it as. But I had a sneaky suspicion that we would start to see triple parity RAID systems. It'll be interesting to see what the performance is like and what the trade-off is. Will we see quad parity, enormous caches to hide the parity penalty, variable parity schemes which adjust according to load?

And if anyone is already shipping triple parity; apologies....

 

Living on a prayer

Much of what we do in Storage Management can be considered living on a prayer; this is not just a result of the parlous state of the storage management tools that we use but also due to the complex interactions which can happen within shared infrastructure.

And frighteningly enough, we are in the process of making the whole thing worse! The two darling storage technologies of the moment; thin provisioning and de-dupe scare me witless. Both of these technologies in the wrong hands have the capability to bring a server estate to its knees. By wrong hands, I mean just about anybody.

Both of them allow you to virtualise capacity and allocate storage which isn't actually there and hope that you never need it.

Thin provisioning is predicated on the inefficient practises that we have come to know and love; we all know that when a user asks for storage, they nearly always ask for too much! Thin provisioning allows us to logically allocate the disk and only when it gets written to, does it actually get consumed.

The problem is, what happens in the event of a perfect storm and every application wants its capacity at the same time? How much do you over commit your physical capacity? Or maybe not a perfect storm, you just realise that you're going to add physical capacity above and beyond that which is supported by the array simply to cater for rate the thinnly provisioned storage is being consumed.  A rapid application migration ensues.

And then there is the scarey proposition of de-duped primary storage. You could be many times over-subscribed with de-duped storage; certainly in a virtualised server environment or a development environment where you have many copies of the same data. And then someone does something; a user decides to turn on encryption and what was many deduped copies of the same data actually becomes many full copies of the same data and you have run out of storage space in a spectacular fashion.

Also migrating deduped primary storage between arrays is going to be a lot of fun as well and is going to need a lot of planning. Deduping primary storage may well be one of the ultimate vendor lock-ins if we are not careful.

Both thin-provisioning and primary storage dedupe take a lot of control away from the storage team; this is not necessarily a bad thing but the storage team now need to understand a lot more about what their users are doing.

It will no longer be enough to just to think about supplying spinning rust which can deliver a certain amount of capacity and performance. We are going to have to understand what the users are doing day-to-day and we are going to have to communicate with each other.

And yes, we'll need better tools which allow us to see what is going on in the environment but also to model the complex interactions and impacts of various events. We are going to need to know if a user is intending to do something like enabling encryption, a big data-refresh, operating system patching; events which in the past were not hugely significant could now have serious ramifications to our storage capacity.

I still think that thin-provisioning and de-dupe are good ideas but like all good ideas; they come with challenges and a certain amount of risk...

Living in the Cloud

I'm a big fan of Dropbox in general and use it extensively to share files between my various machines; it's especially useful when I'm at work with my MacBook and I need documents from the desktop at home.

But there is one issue which is irritating; that is no selective sync and I really didn't want to sync my whole personal Dropbox with my work laptop; actually, my work laptop's hard drive is so small that my Dropbox account would fill it.

I recently P2Ved my work-laptop so that I could run a virtual instance of it at home on my big desktop and my MacBook; makes working at home much more comfortable and I don't actually have to lug my work-laptop around any more but this gave me an issue. I keep all my work-documents on my laptop and I didn't want to have to rely on flash-drives to carry my documents around on.

After a bit of googling and a plea on Twitter; I got my solution. Set-up a second free Dropbox account and share a folder between that and my personal account; it works great. So I now have all my documents in sync without having to lug my clunky work-laptop around.

And then the Twitter document leak story broke; how to cope with security in the cloud? Oh, easy; I just used TrueCrypt to create an encrypted file-container to store my documents in and moved that to my Dropbox folder. TrueCrypt is a great piece of free software, if you haven't got a copy; you should have!

I still wish that Dropbox had selective sync but with a tiny bit of fiddling about, I have a secure and robust way of sharing my stuff. This cloud stuff is great and with a little bit of thought; you can ensure that your files are safe.

Of course, you should also be ensuring you use strong passwords to try and prevent people gaining access to your accounts but people like Google could make life tougher for the hackers and certainly make password reclaim a lot more secure.

Caveat Emptor

EMC's announcement that they are going to start including the license for Virtual Provisioning for DMX/3/4 and V-MAX for free is very welcome. Not only because it includes useful and necessary technology for the simplification but it also simplifies the license model by bundling something else into the base.

Feature/functionality licensing is the bane of many a storage manager's life; it can almost be a full time role to sit and work out what you are paying for and what you should be for. The license models are arcane at times and make the job of comparing like for like when doing competitive analysis an absolute minefield. Not just a pure technical comparison but also a commercial comparison becomes a tedious exercise of endless spreadsheets!

The vendors themselves know this and play on it; you get vendors going in at the highest level claiming that your incumbent is massively over-charging you and you could be paying a lot less. The note comes down from on high and you as the poor storage manager sit there trying to work out how the price was derived; you then discover that the cost that this new vendor is quoting actually is missing essentials like replication, cloning and indeed any management software.

Suddenly the cost quoted rises substantially and you state the actual cost and the whole thing descends in recrimination and blame. And all due to arcane and incoherent pricing models. On occasion I've had to explain the pricing model to the sales-guy in front of me to save him from making a serious error and understating a cost. So what would make things easier?

Firstly, a simple per terabyte cost for useage of features. I'd love to see straight line pricing without tiering but that appears to be a model adopted widely by the software industry. You guys could be the first to step away from the tiered model but I can see reasons why you don't; at least not on your published price lists.

Secondly, a clear definition of the functionality of each package. It's not always clear from the name what it provides and what it does. Teach your product managers to name products in clear, concise terms and not to come up with weird and wonderful acronyms.

Thirdly, the base cost per terabyte should include everything needed to provision and manage a LUN simply and effectively. So management software (both CLI and GUI), wide-striping (if you support it), thin provisioning and arguably all the automated optimisation software.  I'm sure there are other things; there are some real weird and wonderful things out there which vendors don't include as their base.

Imagine my surprise to find that CIFS on a NetApp filer was included but NFS was a chargeable option? It can be justifiable but it was a real WTF moment. SMC to manage your Symms is included if you buy Control Center but it's not included with the base software at the moment.

Barry, perhaps as part of my campaign to simplify storage management and provision, you can encourage EMC to include SMC as part of the base?

UPDATE: Barry mailed and pointed out that SMC is part of Sym Package which does include CC and other bits but it just goes to show how unclear the licensing is! Also, it gave me a reason to check a recent quote; the cost of the software when compared to the cost of the tin is frightening! Frighteningly high that is. Actually EMC do a very poor job of encouraging the use of SMC.

And fourthly, be a lot more open and honest about what you are including when you are talking to people who might know less than the readers of this blog. At times, I think that if you were selling double glazing; you'd forget to mention that the second pane of glass was an additional cost!

'Caveat Emptor' should not be your motto!

How Wide is Wide?

No, not a cricket question!! But one which is is nearly as important! We are now all coming to the happy agreement that wide-striping is generally a good thing! But just as there are some implementations of thin-provisioning which are a little chubbier than the others; there are implementations of wide-striping which are a little narrower than the others.

So at what point does striping become wide? Is 14 disks wide? Is it 16 disks? 26 disks? 50 disks? 100 disks? At what point, does the implementation become wide?

Any thoughts? Go on, I know there are!!

Flexible Thinking

I think that there is some interesting discussion to be had from Hu's latest blog in response to my own comments and thoughts about whether storage virtualisation as demonstrated by the external storage virtualisation devices has a long-term future.

The response is not really to do with virtualisation at all; it is all about aligning IT and Business. He talks about an all too familiar case where storage decisions are made locally by the Business Units and the procurement strategy does not take account of the long-term health of the group; ongoing OpEx costs are not born by individual business units and become the problem of the IT department. The concept and value of shared infrastructure was not really understood by the Business.

But I suspect that these very same Business Units will be happy to use Cloud-based services, Infrastructure as a Service etc, concepts which are built on multi-tenanted shared external infrastructure. Why? Because they can deploy rapidly and flexibly; they don't need to engage the slow and cumbersome IT department and they feel that they are masters of their own destiny.

Storage virtualisation won't bring these Business Units back but storage virtualisation may enable IT to put together a flexible and responsive service catalogue. CIOs need to engage with their customers and understand what they want but they need the support of vendors to clearly demonstrate cost to their customers and to their boards.

The service must be something that the customer wants, customers do not understand why it takes weeks to provision servers, storage and networks. They can go down to PC World and the likes and pick things off the shelf *NOW*.

The more savvy customer knows that they can enter a credit card number into a cloud provider and can provision dozens of servers much quicker than the IT department can and what's more, they can turn them off again equally quickly and not be stuck with kit that they do not need and do not want.

Customers want dynamic, flexible infrastructures which can rapidly respond to their needs; Corporate IT departments along with their vendors have been pretty poor at providing these. Virtualisation and abstracted infrastructures are enabling technologies but Hu's right in that people and processes are key...

As for external storage virtualisation devices, I still wonder if they are the future? They are not necessary to provide the service and if I was building a green-field data centre with no legacy to deal with, I am not sure I would deploy them.

Infrastructure provision is at an interesting inflection point; if you don't understand this and your customers do, you have got a problem.

First, the worst....

Who copied who? Does it really matter? Michael quotes a Sun blog where Sun appear to claim that they beat the SP2 with the Sun Enterprise 10K. It's actually quite an amusing piece both the article quoted and the article itself.

Firstly, I must out myself; I worked for IBM resellers in the late 90s working on RS/6000s and SP2s and then I was the Data Centre architecture for a company which made extensive use of SP2 technology in a commercial environment; so I have a bit of soft spot for them.

Indeed, I was a certified SP2 specialist for my sins but I have worked on most type of Unix box. So please take my comments with a pinch of salt.

From Ken Ow-wing's blog

The reminds when when when IBM clustered a bunch of RS 6000s togeather around a moderately performant switch/network, called it an RS 6000 SP2, and  placed Oracle Parallel Database on it for the backend database for applicatioins such as SAP R3.Sun created a competitive program to compete with the  IBM SP2 with the earliest  ancestor of the Sun Enterprise 9000 server, The Sun Enterprise 10000. which went GA 1997. This was Sun first Supercomputer based on the designs acquired when Sun purchased the Business Systems Divsion of Cray Research (the first and foremost SuperComputer company of that era).

You may notice that the Sun Enterprise 9000 is still around and you do not hear much about IBM SP2s. Sun won. I will spare you more details.

The SP2 does still exist but it's not called the SP2 any more; the SP2 is now the Cluster 1350 and Cluster 1600; that's why you don't hear much about it, it's changed its name! It is an MPP Cluster and indeed a completely different beast to the Sun Enterprise 10K.

Although it did at times get deployed in a commercial environment; that's not what it's sweet spot was and OPS was a real pig to get working! It was massive in the realms of Seismic and Scientific processing; I had colleagues whose job was to install SP2s on survey ships.

And before the days of Gigabit ethernet; if you wanted to chuck large quantities of data around between your nodes; the SP2 switch was an excellent way of doing so! If an SP/2 was deployed in commercial, it was more often to utilise the switch and excellent (but rather unique) management tools to enable a large number of servers to be managed from a single point.

Much of the technology from the SP2 has found it's way into the BlueGene devices. The SP2 lives on!

The Sun Enterprise 10K was an excellent large SMP box and IBM did try to compete using the SP2 but it was always tough because managing a single box was always easier than managing a whole bunch and coding for a single box is often easier than trying to write MPP code. But if you really want to do massively complex deep computing which needs thousands of cores, MPP is often the way to go.

However IBM did manage to get their SMP story together and Sun never really got their MPP story together; IBM can now offer excellent SMP performance and couple it with their MPP know-how to deliver clusters of large SMP devices.

Sun have a 1% share of the TOP500 Supercomputers, IBM have more than a 37% share of the TOP500 Supercomputers using technology derived from the SP2 technology. That doesn't look a defeat to me!

So perhaps IBM did copy Sun and decide that they needed large SMP boxes but they took the idea and ran with it; coupling it to their own strengths and produced something which was greater than the sum of the parts. And hell, it wasn't Sun trying to buy IBM at the beginning of the year was it?

Perhaps EMC are copying HDS but perhaps they too will run with it and produce something better? Innovation in IT is rarely revolution; it is always building on what has gone before.

As a customer, I really don't care who is doing the copying; it's not art and I don't need the original, I just need the one which works best! Tell me why yours is better not that you were the first!

As kids, we used to have a rhyme...

 'First, the worst!
 Second, the best!
 Third, the one with the hairy chest!'

Sometimes it's not best to be the first!

Set the Wide Stripes Free

There have been a couple of articles recently on the HDS blogs about HDP; Hitachi Dynamic Provisioning is HDS' thin provisioning offering and like all of the thin-provisioning, it also offers wide-striping. I am not going to get into whether HDS' offering is chubby, skinny or just slightly overweight but what I am going to ask is...
if wide-striping is so foundational and so important to the storage industry and especially to improving my TCO as a end--user, why do I have to pay extra for it?

HDS and EMC are both extremely guilty in this regard, both Virtual Provisioning and Dynamic Provisioning cost me extra as an end-user to license. But this is the technology upon which all future block-based storage arrays will be built. If you guys want to improve the TCO and show that you are serious about reducing the complexity to manage your arrays, you will license for free. You will encourage the end-user to break free from the shackles of complexity and you will improve the image of Tier-1 storage in the enterprise.

I understand that you feel that you have to maintain the legacy architectures and designs that you have inflicted upon us in the past but it is time that you stopped enabling our mashochistic tendencies and it is time that you encouraged us to move away from the pain of the past. You can do this by removing the licensing costs for wide-striping; keeping the cost for thin-provisioning is just about acceptable but charging for this key simplifying technology is not!

And if one of you does it first; it's okay to copy! Really it is!! I can feel that we're going to be friends.

More Virtual Discussion

Barry Whyte writes on one of my favourite topics 'Storage Virtualisation'.

Now, I do actually think that both SVC and the USP-V have a place in the data centre; they shouldn't but they do! They (and the v-Series working in block from NetApp) provide a common layer of abstraction to the various storage devices which exist in a data centre. If you have more than one type of storage or indeed more than one array and especially if you are attempting to tier data across arrays of many types; you should probably be considering these.

If you are going to attempt a mass data migration, the SVC is something that should certainly be considered due to the relative ease of getting into SVC and getting back out of it. IBM have thought long and hard about the migration use-case for SVC and added the necessary functionality to get out of SVC. A sensible move because one of the major objections to most major virtualisation projects is; what do I do if it doesn't work for me? How the hell do I get out?

Also, virtualisation devices allow me to drive up utilisation by enabling pockets of storage dotted across arrays to be pooled up and used. In fact, often this is the major selling point of the virtualisation devices; it makes the previous unuseable, useable. However, I would argue that these little pockets of storage are an aberation caused by the legacy architectures caused by the traditional arrays and how they are carved up into logical (some would say virtual) disks.

The virtualisation devices can take these little pools of disk and like a baker with pastry offcuts; roll them back into a ball to be re-used. However, you do have to be careful when collecting your pastry offcuts that you don't end with a ball which doesn't stick together especially well and has a habit of flaking and breaking when baked.

This could well be your experience when rolling up your little pockets of storage; disks from different arrays and vendors will perform in different ways. Spreading virtual devices across multiple arrays from different vendors may cause some interesting performance challenges. If the array with the pocket of storage already is fully utilised from an I/O point of view; attempting to use the disk might well have a negative impact on application performance.

Actually spreading a single application across multiple arrays needs careful planning and consideration. It will complicate your change management, problem determination, configuration management and many other necessary processes. You still need to maintain the underlying arrays and understanding the impact of an outage to one of these arrays when you have fully virtualised might be interesting; an outage to a single array could take out your whole data-centre environment.

You might find you get more benefit in the long term by considering an array architecture which inherently enables you to use disk in a more flexible and efficient manner. You may also find that using automation and allowing the array to make decisions about where it puts data drives up efficiency.

So as we move to more scalable, efficient and automated environments; I wonder if we will look back at things like USP-V, SVC etc as a cul-de-sac driven by today's necessity! Or perhaps they truly are the future?