Storagebod's Blog

Different strokes for different folks

I am seeing a lot of bizarre talk from EMC's competitors about V-MAX. The gist of it is that V-MAX is too big and too expensive for the market. The market just wants mid-range etc, etc. Why the hell do EMC's competitors care; if EMC have got it horribly wrong, no-one will buy V-MAX and it'll be the death of EMC! They'll go the way of Sun, SGI, DEC etc and become a memory!

But we know this is simply not true; there is still a market for high-end; large enterprises are still generating bits at a crazy rate! There are various ways of solving the problem; monolith, clusters,modular, m-monolith; external virtualisation but it is simply the case that data grows.

Not everyone needs a bus, some people need a people-carrier, some people an estate car, some people a sports car. No one size fits all. Mainframes still sell. If you decide that you want to leave the rarified slopes of high-end Enterprise storage to EMC; that's cool.

Now, you might want to attack EMC about the sheer number of storage products that they have and the sheer number of interfaces you need to learn (V-MAX/DMX, CX, Celerra, Centera, Atmos, Iomega, Hokey-Cokey 3000 etc) but to attack them because V-MAX is too big; that's just odd!

*m-monolith is a modular monolith like Stone Henge...I am waiting for the first V-MAX installation to be arranged in a Henge-like layout!

Do Better!!

Storage workloads can be measured in a number of metrics but have historically been measured on one rating alone; capacity! How much disk! Now storage workloads can be measured in a combination of the below

1) Space
2) IOPs
3) Throughput
4) Latency/Response times.

The problem is that the industry has really struggled in the trade-off between the top one and the metrics below. Disks have got bigger and bigger with whilst the others on a per-spindle basis have stayed pretty much constant; this has resulted in the cost/GB falling dramatically whereas the decline in the others has been less dramatic.

As a result of this, we have seen the effective utilisable capacity of the disks in many cases fall off the cliff. Storage has become horribly inefficient apparently; in fact, some techniques such as primary storage dedupe could actually exasperate the whole situation; deduping the data on 100% saturated disks would just make things even less efficient.

So just over a year ago, EMC introduced SSDs into their DMX array; followed by the CX range later on in the year. Big fanfares from EMC and lots of raspberries by the rest of the industry followed by a number of 'me too' announcements.

Now EMC argue that for some workloads, the deployment of SSDs will save you money; you need a lot less of them to fulfill particular types of workload. However, EMC's implementation in the DMX/CX was arguably too early, too expensive and a pain the rear to get the implementation right.

Yes, we had a feasibility study done and although it would have had substantial improvements in performance; the cost savings might well have been outweighed by the amount of work to relay the array out and more importantly to re-engineer some of the application file-system layouts. It was decided that it did not make economic sense at this time to do the work.

I had also had a number of discussions with various storage vendors including EMC about the most efficient use of SSDs; almost everyone agreed that sub-LUN storage tiering would bring about massive improvements in the effectiveness of SSDs and the use of SATA drives. It seems that a large amount of research was going on in various vendors to enable this. EMC even had a name for theirs; Fully Automated Storage Tiering but they are certainly not the only people I had this discussion with.

I was a tad disappointed because I thought I was a genius and had come up with the concept; it seems that I was a little late to the party because so many of the buggers were already working on it.

And I'll tell you another thing; no storage vendor I have spoken to believes that SSDs are a bad thing! Some of them are waiting for the costs to come down and they are coming down extremely rapidly; some are still puzzling how to make the most effective use of them (I'll give you a clue, automatic migration of hot data blocks is a really good idea!!) and some are probably having a bit of a sulk because EMC got there first.

The latter is not really surprising because I have seen EMC do the same thing; their RAID-5 implementation was lousy, so they rubbished everyone else! Now, it is fairly unusual to hear an EMCer seriously suggest that RAID-5 is not appropriate for most Tier-1 workloads. The horror stories I heard in the past are going away.

Then 3Par started the thin-provisioning meme in a big way and yet again we were warned that the seas would boil and our data disappear into the ether! Now that EMC have wide-striping and virtual provisioning; it's all goodness!

External storage virtualisation is the work of the Devil Incarnate and a gateway to the underworld would be opened. EMC still believe this and I suspect that secretly they believe that VMWare is actually playing with unnatural powers with some of their storage virtualisation stuff! 

Guess what, none of this is true!

And guess what, at some point all storage vendors will probably have SSDs in their systems; EMC just had the misfortune to be first! They've also gone and launched their biggest revision to the Symmetrix line since DMX in an economic down-turn. That takes balls! I'm intrigued to see if the other vendors who last year were planning big announcements this year actually release product into a fairly hostile market.

But if you don't; EMC are going to have a huge headstart, they might not ship shed-loads of V-MAX this year and they may not ship shed-loads of SSD, however their sales-guys, their flag-wavers etc can all talk about this stuff openly and they can steal a march.

This might not be time to take large orders but it's time to be bold and to talk to customers; get your new products out in the open; share roadmaps. Sure the end-users still have to keep the lights on and grow our estates but we are in 'make and mend' mode at the moment; we are probably buying less but we are planning to come out of this downturn and we might have time at the moment to make some more considered decisions. Don't rubbish EMC's roadmap and vision...do better!

Maintenance Madness

Despite working for a vendor, David Merrill has a habit of posting some very good entries full of common sense; I find myself nodding in agreement with much of what he posts. His latest couple of entries here and here had me nodding in agreement; it's not just the vendors who are guilty of some dubious voodoo economics, I'm sure that most of us have put together business cases which if were really scrutinised, don't really stack up.

We often talk about trying to make capital acquistions cost neutral in less than eighteen months; a reduction in Opex to offset the capital cost. Vendors are often complicit in this, as I mentioned in my previous entry, inflated maintenance costs mean that is often cheaper to refresh and take the bundled maintenance offered with a new system than to continue to pay maintenance on the legacy kit.

However, if I examine the failures that we tend to have; it is generally the moving parts which fail; you know those things which spin at speed? Yes, the spinning rust. And if there is one thing which has fallen in cost; it is spinning rust.

Okay; with the very much older disks, vendors simply can't get new drives that small but I assume that most of you are aware that a large number of maintenance replacements are not actually new components? They can be previously failed and reconditioned components or perhaps pulled from arrays which have been migrated to the latest and greatest technology.

Maintenance in the IT industry is a fantastic example of Voodoo Economics...but hey it's green, well they are recycling and re-using! But remember, there is a third part to that; REDUCE!

Vendors don't have any real incentive to reduce maintenance costs; it firstly enables a constant upgrade treadmill because if you really had to evaluate the value of the new features, life would be a lot more complex but if you don't upgrade, maintenance is a very nice and high margin activity.

Actually EMC should be thanking companies like HDS and IBM; it enables people to keep their legacy arrays around for a lot longer and hence keep paying EMC high maintenance! And no I'm not saying that EMC's maintenance charges are especially high, there are much worse offenders out there!

Do Something with Nothing....

A useful discipline every now and then is to sit down and imagine what would happen if your storage budget was cut to zero. I'll allow you to keep your staff and ongoing maintenance costs but you can spend no more money on 'stuff'! The business is going to continue and you need to support business growth.

So what do you do? Where do you start? Doing nothing is not really an option so make a list of the things that you can do.

First thing you need to know is what capacity you've got where. You need to look at what is actually used as opposed to what is allocated. You also need to identify where storage is 'reserved' for projects which never came to fruition.

That array which only ever got partially migrated because it was too hard to finish. Or circumstances changed and the plan changed.

The DR disk for that application which doesn't exist any more.

How many switch ports you've got reserved to support expansion which is never going to happen.

Identifying multiple copies of the same data-set; I've come across places where there have been more than a dozen copies of production databases; full-size copies. There's the production copy, a reporting database copy, a DR copy, a back-up copy, a production support copy and the multitude of development copies. Perhaps it's time to speak to the developers about their requirements; do they really need full-sized copies?

Sure it is easier to take a full-sized copy but you should be data-cleansing your production databases before they go anywhere near development and perhaps as part of the data-cleanse, an extract as opposed to a full-sized copy can be taken. 

Look at your RAID levels; are they really appropriate? How easy is it to change them?

And then ask yourself, how long could I really go without spending any money on storage? The answer may surprise you...if it's a matter of weeks, you can pat yourself on your back...it's more than six months, maybe a year; you really ought to consider doing something with nothing.

Even if you only spend a couple of hours on this thought exercise, I promise you it'll be worthwhile.

Investment Strategies and Virtualisation

I sat in a meeting today where the subject of how often you refresh your storage infrastructure came up. I know that many companies are working on a three-five year model but we were discussing whether this should be increased to seven and what needs to happen to make this so.

There are a few reasons why we were coming to this conclusion; firstly spinning rust in the Enterprise is probably at it's peak and actually anything over the current maximum size of a spindle has potentially limited use i.e anything over a 1-2 terabyte drive is not especially useful for shared storage infrastructure. Please note, I say shared storage infrastructure!

Larger drives may still have a place to play in your archive tier but even that is debatable. And if you look at most Enterprise end-user desktops; they often have rather small local drives. It is the home-user and their insatiable demand for storage which really drives the size of spindles now.

We also know that the performance of the spinning rust is probably not going to improve dramatically. So what does change? Well, yes we have the introduction of SSDs and a couple of things mean that a four-five refresh cycle for that technology is probably sensible. And then there are the storage controllers themselves; these don't especially wear out but technology does move on. 

But the current designs of arrays mean that when we refresh; we are forced to refresh the lot. We are also forced to refresh by overly inflated maintenance costs. Let's be honest; most refreshes are justified by cost savings on the OpEX i.e maintenance. Even if I go to a virtualised infrastructure as espoused by HDS or IBM; these maintenance costs still mean it is often more attractive to refresh rather than sweat the asset.

However the current economic climate means that we are now more closely beginning to examine the model of keeping things for longer and examining our maintenance budgets very carefully. Dropping maintenance for software which is now stable and at terminal releases; potentially talking to third-party maintenance organisations who are much more willing to support legacy kit at a reasonable cost.

And we are considering strategies which enable us to continue to make use of kit for longer. VMWare's announcements today allowing replication and thin-provisioning at the hypervisor layer for example.  So funnily enough, EMC have come round to external storage virtualisation; you just buy it from VMWare as a software product.

It'll be interesting to see what other traditional storage related functionality makes its way into the hypervisor. And at what point EMC realise that they are actually selling 'traditional' storage virtualisation but as a software product and at which point that they do become a software company.

Funny old world, as EMC slowly catalyzes into a software butterfly selling storage virtualisation, Oracle becomes a hardware grub. And in the space of a week; EMC 'kill' DMX with V-MAX, then they kill V-MAX with vSphere. Now that's what I call progress!

Oracular Spectacular

I'm just not sure about this one; I could have seen the IBM acquistion of Sun being very destructive and ripping the guts out of Sun. But I'm not sure that this is really much better; it is as has been pointed out an extremely defensive move by Oracle. However, a defensive move to buy a company which Oracle probably does not really understand. But does anyone really understand Sun in it's present guise because it is obvious that the current management don't!?

A software company buying a software company pretending to be a hardware company would make sense if it was to kill the hardware side of things and just leverage the software. But at the moment, Oracle seem to want to continue with the hardware side of things and even continue with the Sparc franchise. And it was bad enough dealing with Sun to get support for our Storagetek silos; dealing with Oracle fills me with horror!

As for what it means for storage? Exadata is probably dead in the water in it's current form. The ZFS lawsuit becomes NetApp vs Oracle; it feels almost an unfair fight. And of course the NetApp sales-guys are going to probably have to change the 'Oracle uses NetApp for all it's development storage' line.

As for Pillar, well Pillar are always at pains to both stress the Larry relationship but also that they are nothing to do with Oracle; it's one of the more schizoid sales pitches but I don't see anything changing with Pillar.

I am also quite excited by the carnage that maybe wrought in Sun...why? Am I some kind of voyeuristic sadist? No,  I think that we might see a mass exodus (as we would of with IBM) of some very talented engineers who are determined to show the world how good they are and we might get some exciting new start-ups. If I was a VC/company with cash to burn; I'd be looking at what flotsam and jetsam is washed up from wreck and wondering what I could make from it all.

There is also something about the whole deal which leaves me feeling uneasy, last time I had that feeling, Compaq bought DEC. One hopes that Larry's eyes are not too big for Oracle's belly; I mean HP don't have an enterprise database yet....but give it a few years!

More musings

With the V-MAX architecture; EMC are now Intel across all of their hardware; well, they will be once the DMX is finally sunset. This is a necessary step for them to start unifying their storage ranges or at very least, it brings them much closer.

If you look at the way a V-MAX is managed; it is a lot closer conceptually to Clariion than a DMX; well, that's what I feel from my cursory look over. Yes, you can stick to the old ways if you so wish but if you really want to sweat the full potential of V-MAX, you are going to want to change.

I suspect that this first iteration is a transitional product; yes, you can continue to manage it in the same way as a DMX and continue to use those carefully crafted scripts but I really wonder for how long? And at some point those legacy features will become obsolete.

I wonder if there was real debate in EMC whether to break with the past or ensure that customers have a smooth transition.

Going down the Intel route and with all the horse-power which EMC now have available to them leads me to wonder if we are going to see a virtual Celerra which runs within the V-MAX? Perhaps a whole raft of virtualised storage-related appliances could run within the V-MAX itself? IBM had the ability to do this with the DS8K range and never did it but perhaps EMC will.

Also I think the adoption of Intel at EMC's high-end actually throws the long term future of Clariion into question as well. EMC are probably well on their way to unifying their block storage range; EMC have already said that we will see FAST in both DMX and Clariion, this will allow existing customers to transition to a new way of doing things.

But eventually EMC are probably aiming that both sets of customers will eventually transition to either V-MAX-Pro or V-MAX-Mini. I can't see EMC wanting to continue with two code-bases for longer than they have to.

So that's EMC big announcement out the way for the moment. I wonder what the rest of the year will bring? IBM are long overdue a refresh and HDS are probably due an announcement at the high-end. And NetApp will finally bring OnTap 8 to the market and I suspect some new high-end heads; NetApp, like EMC, do have some advantages on refreshing hardware, their boxes are built mostly out of commodity. And IBM's own storage is either a PC or a pSeries box; servers as storage controllers...it's the future I tell you! Or perhaps in EMC's case, storage controllers as servers!

FAST and Furious

Whilst HDS and EMC throw rocks at each other with regards to whether it is better to build custom parts or take things off the shelf and just use custom when you require (I'm expect the other Barry to sit on his hands but there are good reasons why the SVC team decided to build out of commodity parts and I suspect that they are very similar reasons to EMCs). I think we should look beyond the hardware and look at what is coming down the line to us.

The most important thing roadmapped is FAST, Fully Automated Storage Tiering. FAST changes things; it takes a whole bunch of ideas from a whole bunch of places and runs with them. If you are another vendor and you feel aggrieved that EMC have stolen your idea; just take heart, it won't be the first time in history that this has happened and it won't be the last.

The foundation is Wide-Striping* using a model which splits your data into chunk(let)s and spreads it across spindles. Once these chunks are distributed, you can monitor the characteristics of the I/O at an individual chunk level; this allows us to do tiering at a sub-LUN level. A hot chunk of data can be moved to a higher tier and a cooler chunk of data down into a lower tier.

In the past we have been limited to moving a whole LUN (with the exception of Compellant); this has always been a time consuming job, identifying what needs to move and then moving it. Yes, technologies have come along to make this easier but to sweat the asset and especially to make best use of SSDs; we needed to move individual 'blocks' as in a given file-system , it is possible that only some blocks are hot and frequently accessed. Traditionally if you could, you would hold these in cache but if SSDs are expensive, cache is yet more so. This approach will allow some cache to be replaced by SSDs and for some cache unfriendly workloads, to all intents and purposes, you have massively increased the amount of cache available. You might not want to hold a terabyte or so of real cache for that evil 100% random read app but with SSDs; this becomes viable and not at a huge utilisation hit.

But there are going to be issues with the FAST approach; firstly, where do you put a new workload? If you simply assign it some disk and let the array decide, what the hell is it going to do with the workload? It could put it on the slowest tier possible and then migrate up; it could stick it on the fastest tier and migrate down. Both of these approaches have significant risk, so I suspect we are going to have to give the array some clues and we are going to have to understand more about the whole system we are putting in. The difference in performance between the top tier and the bottom tier is going to be large.

No longer will the Storage Admin be a Lun Monkey; they are going to need to really understand their workloads and the applications. They are going to need to learn to talk the application developers and understand workloads, they are also going to have understand business cycles.

For example applications which spend 11 months of the year pretty idle may suddenly at year end need a lot of performance. What happens if all your applications demand stellar performance once a year? Perhaps you need a way of warning the array that it needs to prefetch a load of data. A badly written end-of-year reporting extracting which generates thousands of random read IOPs. A badly written user-generated SQL; in the past, this just crippled the application; with FAST, this could cripple the whole array as it tries to react.

The FAST approach is potentially the thin-provisioning of IOPs. This going to need a lot of thinking about. Potentially you will have to domain storage to protect applications from the impact of one another. We are going to need to know more about the whole system than we have before if we are to truly benefit from FAST,

Building rules which suit your applications; sure, V-MAX will come with its own canned rules for things like VMware and known applications. Indeed EMC will probably be leveraging all the performance data that they have been gathering over the years to help us write the rules. Storage Templates as described by Steve Todd here are just the start.

So although at one level, the Storage Admin's job could get alot easier; the Storage Manager's job has got a whole lot harder. Yes Barry, I asked for FAST and now you've given it to us; now we'll have work out what this all means!

I have some really 'interesting' ideas as to where EMC could take V-MAX but they'll have to wait for another time as I'm still supposed to be on leave from Enterprise IT.

* It's Wide Striping not Wide StripPing as I keep seeing written; Wide Stripping is what happens on a Rugby Tour after a good night out!

So No DMX5

I'm on leave this week, so I'm not going to post a whole deal on this but I actually thought that this announcement was important enough to sit through the video presentation just to see what EMC were going to publically  announce.

So there it is revealed, there is no new DMX! There you go! I told you that there wouldn't be!

I've known a little about about the V-MAX (that's a cool name, pity it sounds like a console) for nearly a year now; I responded to a post on Chuck's blog with a list of things I wanted from an array; Chuck didn't put the post up and just said that they were working on making my desires a reality.

I really wanted Automated Storage Tiering to simplify storage and it looks like EMC have delivered it. The devil will be in the details though but this is the start of something which will have industry wide implications. Automation on the scale that I believe will eventually happen will change the role of the storage management team for ever.

I've spoken before about the ability to federate arrays; I actually look at what much that has been done both by IBM and HDS as moving towards this. EMC have delivered the ability to federate their arrays; pity it can't do third party but hey...this is EMC we're talking about! Technology refreshes and data-migrations will become a breeze!

Building on top of industry standard hardware, this is just plain common sense. Going down the Intel route, just makes sense as part of cost reduction and common components.

Management tools will need to change as part of all this, I'm expecting a refresh of the management suite at some point. Who knows, we might get the long promised re-write of Control Center, I'm not holding my breath though! I might go as blue as the flashy lights on the front of the new V-MAX.

I don't know a whole lot about the details but I know that there have been arrays out in the field for a bit. I don't have one before you ask! And of course, I can't tell you who else I have had very similar conversations with about what they might be doing. Life is going to get interesting.

I am frankly very impressed that EMC taken such a leap. Congratulations one and all!!

Oh yes, I forgot to ask....how much is this all going to cost us?

And yes, I don't for one moment believe that the V-MAX will not replace the existing DMX4 unless we are going to see a DMX5 in the future? Now EMC have surprised us once but a DMX5...not a chance! Actually, I think questions have to be asked about the future of all of the other EMC disk-lines; just think about it!

Damn, this is a longer post than I intended! Oh well, back to the DIY!

Guaranteed to make you laugh!

I really thought we'd got away from guarantees which were caveated to hell and predicated on using the vendor's services but EMC I mean VMWare are at it this now. Chuck and Barry must be fuming; well if they weren't concentrating on their rare cat cross-breeding program allegedly!

It just goes to show really, that you can't keep a good/bad* ^{(delete as appropriate)} idea down. What has kind of amazed me is that VMWare have appeared to put even more caveats in place than NetApp. Can they actually fail? In two years time, will all of the savings be realised? It'd be interesting to see a long term review of these programs to see if the results are actually real. But I suspect in two years time, we won't care and it won't really matter that much.

And it also shows, don't make too much fun out of your competitors' marketing gimmicks, you just might find yourself doing the same thing. Still, it made me chuckle more than a little bit.

Don't get me wrong, virtualisation and consolidation is a good thing and I am sure these guarantees are a good way to get customers thinking about what they are doing. But at the end of the day, they are still about getting you to spend money and you might be able to realise significant savings without spending money.

Have a look at the guarantees and the caveats; the caveats do a nice job at helping you identifying where you should be concentrating your effort and identifying the low hanging fruit.

p.s I know why Barry has been so quiet but what about the other Barry? What is the Madster Inventor up to?