Storagebod's Blog

On the Bench

As my regular readers know (and there’s more of them than I thought), I’m a fan of VSAs and have been playing with them at home. I’ve just built myself a test-rig for playing with such things; a pair of machines running AMD5000+, 4 gigs of memory, GBe and booting to ESXi off USB flash-drives. I’m planning for a third box which will be the main storage server for them but I need to find a VSA which I am happy with to run as the main storage.

So at the moment, I’m just installing VSAs under VMware workstation (on Windows 7) and experimenting. Seeing what works well and what doesn’t. I’m already noticing some differences and I’m wondering if it would be worthwhile doing a more in-depth review/test. I am thinking about reviewing on the following criteria:

1) Ease of Installation: How much fiddling do I have to do to get the VSA up and running? Do I need a manual? Is it intuitive?
2) Ease of Use: How easy is it to configure once up and running?
3) Features: What is the feature set like? Also is it capacity constrained?
4) Footprint: How much resource does it require/consume?
5) Performance: this is the most contentious, how do you bench a bunch of completely different products in a fair manner? Also do the ELAs allow the simulators to be benched?

Anyway, the approach I’m going to take is to take the bog-standard download and install it; no tuning. For those simulators/appliances which require a host operating system such as the NetApp simulator; I’ll probably use Ubuntu.

If I can get ESXi to recognise the SATA controller on the real test rigs; I’ll use them, I have spare disks which I can hang off them. Otherwise, I’ll build a new clean Windows image and run them all under Workstation.

For benchmarking, I’ll probably use iozone but if people have other suggestions; please shout. I draw the line at running any of the Spec benchmarks mind you.

I’m not sure how long this process will take, so you’ll have to be patient; it’s a spare time project. And if you have any appliances that you want me to test, a link to them in the comments would be useful. Also, if anyone has any objections to me benchmarking performance of their appliances, please contact me.

Dedupe or Duped?

I must say that I am extremely underwhelmed by EMC's Celerra 'announcements'; very 'meh'! What I found especially interesting is the fact that EMC are not recommending the use of the inbuilt dedupe for VMWare installs. We all know that NetApp have set great stall by their use of dedupe and thin provisioning for virtualised servers, their infamous and unambitious guarantee is predicated that you will use it for VMWare (not for high performance workloads such as Oracle tho').

So I am wondering why EMC don't recommend using their dedupe for VMWare? Is it worse than NetApps? Does it eat more performance on the head? Is it because their dedupe works across the whole data-store unlike NetApps which is currently limited to an aggregrate ISTR? I guess if that's the case you could end up with alot more VMWare images linked to a single set of disk blocks.

Hopefully there will be a friendly EMCer along to explain in a bit?

And what's this Recoverpoint compression? I thought Recoverpoint compression was only for the journal storage and was related to the CDP capabilities? We aren't talking about using compression a la Storwize are we?

Twitterings of a Storage Community

The storage community on Twitter is an active and vocal bunch; well some of us are anyway. Tony Asaro posted a link on Twitter pointing to his blog entry on Compellent and Automated Storage Tiering.

This spiralled into a bit of a conversation on whether Compellent was merely another ILM solution and not that special; whether Object storage with metadata describing was really the answer and what the future might be.

What Compellent do at the moment is pretty damn cool; they keep metadata on blocks and allow you to set-up rules which allow the data to be moved on depending on last access. Barry Whyte suggested that this was only done a weekly basis, this led the question whether this was actually enough?

 I would suggest that it is a massive improvement on where we are sitting today. Compellent will allow you to have a LUN which will span tiers with 'hot' blocks sitting on Tier 1 and colder blocks sitting on some other tier. But is weekly enough? I think we need to be fairly rapidly look at real-time automated storage tiering based on rules derived from last access, rate of change of data etc. 

I'd love to hear some real world opinions/experiences from end-users who are actually using Compellent in anger; it'd be interesting to see a visualisation of flows of data around the array. For example, would it ensure that the fie-system meta-data ends up on the fastest disk? Automated storage tiering may have some interesting impacts on next-generation file-system design.

File-system design, this leads us onto Object Storage; in a world where Object storage becomes more common, what impact does this have on file-systems and data-access. Dave Graham is blogging lots of good stuff on this and it is no-where as near vendor-focussed as I feared considering his employer is EMC. But I don't see it as AST vs Object Storage, I see it as Objects on top of AST.

I think we have two strands of development at different levels of abstraction; both are complementary and both independently valuable. However when applied together, they become hugely more powerful. Objects which move around storage tiers automatically; just taking the space and I/O footprint that they require, replicating and protecting themselves according to the SLAs defined. All this done by an appliance which sits above a modularised storage estate, there's a thought?

Not sure what work there will left for the poor storage admin in this world but I'm sure we can obfuscate this in acronyms etc (Modularised Automated Storage Tiering Utilising Realtime Based Advanced Technology Engine) and keep ourselves in gainful employment. And when it goes wrong, boy will it go wrong!

Very moving....

Chris has posted an interesting piece about migrations and length of time that it takes. We've recently done a number of migrations to remove some legacy DMX/Clariion from our estate and our experience suggests that this is mostly a server team piece of work. I guess it depends on the split of the responsibilities, but in our world, the storage team is responsible for the environment up to the HBA.

He does a great job of listing the processes and procedures; the things you need to do but I think there are some additional things to consider.

1) When migrating; consider your current LUN sizes. I suspect that if you have a lot of legacy arrays, you will find that you have extremely small LUN sizes. Take the chance now to review and revisit. This will potentially create extra work at migration time but it will be worth doing. As Chris says, this will make it more time consuming and complex but if you don't do it at this time; when will you do it? Once the data is in-place and migrated; you almost certainly won't get the chance to revisit. Try and do it now!

2) Use of thin-provisioning technologies; if you want to leverage thin provisioning technologies, you are going to struggle with the current migration methodologies; see if you can rightsize file-systems but this might require an outage. Your choice of technology if you want to go Thick-Thin is rather limited at the moment. cue: Farley's obligatory 3Par plug.

3) Tiering; you can use this as an opportunity to right tier your data. Moving from Tier-1 to Tier-2 may be possible. But make sure you understand your transaction profiles and I/O rates but odds are you can probably migrate all that RAID-1 to RAID-5.

4) Be aggressive if you want to realise savings; you could be paying maintenance on the array until it is turned off, so that piddly little system which doesn't get moved could cost you thousands.

5) Negotiate maintenance holidays; if you are migrating and not changing vendor, you might find that you can negotiate a maintenance holiday on the arrays you are migrating from. But see 4), these offers are almost always time-bounded.

6) Find out what unsupported really means!! If you are like most big shops, you will find that you've got a load of hosts which are down-rev and no-one wants to touch! So you have a couple of options, keep an old legacy array about and consolidate the down-rev onto that. Or talk to the vendor and find out what unsupported means, does it mean it won't work? Try and force an RPQ but be aware there's a lot of stuff out there which is running quite happily even though theoretically it's not supported.

How quickly can you do it? Well after a couple of months of planning; we consolidated 10 arrays down to two in a couple of months. The bulk of the work was the Veritas scripts and very few outages were needed.

I would like to think that you should be planning your exit from any kit from the moment that you deploy! I am a realist and you probably won't!

Storage Virtualisation 2.Oh...put some meat on the bones!

Okay, so I like picking on HDS's Hu; I don't know why but his blogs drive me to distraction (I should probably stop reading them). And I'm a sarcastic, cynical and miserable bugger at the best of times! I like challenging vendors and I want them to come up with answers. At the moment, I'm not seeing a lot of answers from HDS and undeniably they have some great products, so do their competitors. I suspect that they may have a very interesting roadmap, their competitors certainly do.

Tony Asaro addresses some of my points here and I'd like to respond:

1) Would I deploy SVC into an Enterprise environment; I know of Enterprise environments including financial institutions which have. Would I do it? I'd want to test it in my environment; I have no fundemental objection to it virtualising a mix of tier-1 and tier-2 if the commercial model works.

2) Okay, USP-V means you could claim to have bought an array and then get virtualisation in by stealth. To this day, I find it hard to believe that IBM have not built an array based on the SVC; technically there is no reason that they shouldn't. I suspect internal politics may have prevented them. But because USP-V can get Virtualisation in by stealth is not a feature in itself.

3) See 2

4) Now this is true and the whole focus of HDS' organisation is on USP-V and USP; that's great. They've bet the farm on it and committed to it. That doesn't make it any better than anyone else's product in itself. And HDS' smartest might not be as smart as a competitor's average; I have no way of telling.

But Tony is right, HDS need to turn up the volume but in a useful way! Talk about how you might use virtualisation, talk about it as a practical thing; not a religious thing. Enough about how wonderful it is!

I suspect that a lot of the vendors have some very similar ideas about the next generation of disk arrays. Many of them are in line in what I have been asking for/positting for the last couple of years. Some of them, I am frankly amazed have taken this long to happen.

1) Looser coupling between the spinning rust (going to need a new phrase to cope with SSDs) and the controllers.

2) Virtualisation in the controllers allowing potentially things like                  

• Domainable arrays

• Virtual Appliances such as back-up appliances, dedupe appliance, database accelerators running in the array

3) Commodity hardware usage

4) Greater connectivity options

5) Sub-LUN, block level optimisation

6) Highly automated management

Now some of these already exist, some of these should exist already. None of them exist in a single product...yet!

I'll stop picking on Hu and being horrible to him! Just ask him to put some humour and personality into his blog. Let's have a real discussion on the value of virtualisation, let's have a real discussion on the future of storage.

DAS washes whiter....

'Zilla talks about various things here but what caught my eye is his commentary on the Exchange DAS vs Storage Array conversation.

It's an interesting one; he feels that Microsoft are trying to maintain their margins putting more and more traditional storage functionality into their software and then suggesting that the end-user saves money by putting it on DAS. DAS being cheaper, this allows the TCO for Exchange to fall.

As end-user; as long as my service is maintained and perhaps even improved, I don't care. This is not a religious debate of SAN versus NAS versus DAS, this is more fundamental than that. This is about saving money. This approach could save me both capex and opex; it needs careful modelling but more and more traditional storage functionality is going to end up in applications.

We are going to see an almighty tussle between the storage vendors and the application vendors; EMC find themselves in an interesting position in this tussle, how much traditional storage functionality could find it's way into the hypervisor?

High-speed interconnects such as 10 GbE and Infiniband don't just speed up server-storage connectivity; they speed up server-server  connectivity as well enabling fast block-level replication at the hypervisor layer in a way which is transparent to the operating-systems and the storage (could be DAS or SAN).

So who'd be an array-vendor? Margins are going to be squeezed and differentiators are going to need to be found.

Holey Utilisation Batman

So disk utilisation rears it’s ugly head again and again and again; some reports suggest that things are getting worse not better. And you know what, I think that they may get even worse in a lot of shops. There a few trends which lead me to this conclusion.

1) Focus on the cost per byte stored; disks are getting larger and so everyone expects the costs to fall.
2) Greater choice in disk speeds, sizes and types.
3) Continued data growth
4) Poor management tools

If we continue to merely focus on the cost per byte without consideration to the cost of accessing the data in a manner which is acceptable to the business; we will find that procurement teams and CIOs focus on purchasing capacity at the cheapest price.

This may actually be acceptable and it may be that you can buy lots of slower disks which enable you to get the performance you require but you will be sacrificing 50% of that capacity because the drives are saturated. So rather focusing on capacity utilisation figures, a more complex model needs to be built which takes account of all factors be it physical space, energy consumption, management overheard etc, etc.

Greater choice leads to more chance of the wrong disks being procured but also it introduces more complexity to the environment. Tiering may actually result in poorer utilisation figures if it is poorly implemented and planned. My experience actually suggests you need a lot less Tier 1 than you think and depending on your procurement models, you may find you’ve purchased more Tier 1 than you require. This leads to an oversupply in Tier 1, there will be initial reluctance to use this as Tier 2 but I suggest that pragmatism is the order of the day. You should be able to move your data around later!

Data Growth means that hard pushed teams are being asked to manage more and more storage. There simply isn’t the time to manually move the data around and ensure that is in the right place.

Management tools do not make it easy to report and get a view of the storage estate.

So what’s the answer?

Firstly understand that the utilisation model is more complex than some of the vendors would have you believe. Use your own figures and models to work out what poor utilisation costs you, run some comparison figures; don’t believe everything you are told.

Secondly, investigate thin-provisioning and wide-striping; it’s going to help! There is no doubt in my mind about this.

Vendors will tell you that virtualisation will help as well; it might do but it might simply give you more headaches. Look carefully at it and think how you might use it but it is not a magic bullet.

But the one thing which is going to help you most is automagic, rules-based management tools. Face it, multiple disk tiers, multiple applications with poorly defined NFRs, incessant data-growth, doing more with the same number of people (if you are lucky); you are not going to cope.

We are building an environment which is too complex and large to manage without automation; you will no longer be able to tune it by ear and the discordance you will introduce by trying will make a cat's choir sound like music.

 And when you find these automagic, rule-based management tools which are self-healing, self-tuning, self-managing; drop me an email!! But don’t tell everyone about them or we’ll all be out of jobs!!

Virtualisation 2.Oh Give It A Rest

I feel bad about this but everytime Hu posts on storage virtualisation; I think he does harm to HDS. Redefining Storage Virtualisation, declaring Virtualisation 2.0; these are the acts of a desperate man.

Firstly, although I've never used the HDS products in anger, I've kicked the tyres of them a few times and I believe that USP is a damn fine array. If you've them installed and they work for you, you've technically got absolutely no reason to change vendors. But then again I would probably say that of all of the 'Tier 1' vendors out there; they're all good enough. So I've got nothing against HDS as a technology vendor.

But give the virtualisation thing a rest! It's not the solution to all the storage issues in the world! If it were, we'd all be rushing to put it in (or are we just too stupid to see that it is the solution to all the storage problems in the world?).

And this insistance that USP is somehow different to SVC is rubbish! SVC and USP are extremely similar conceptually just implemented in slightly different ways! Hu and HDS boast of thousands of processors, ports etc; there's must be better because it's bigger and more shiney; IBM have taken the approach that you can build something from commodity hardware, it doesn't have to be huge to be powerful; it sort of reminds of the difference in approach to building sports-cars. If HDS were an American company, I would suggest that they were building muscle-cars whereas IBM are trying to take the more European route.

Hu is right on one thing tho', 80% of storage in future data-centres will probably be commodity-based modular arrays. Unfortunately, I don't think that these are going to be controlled by Enterprise Virtualisation controllers. Certainly not the current range of the USP anyhow; USP is incredibly limited as to what it can support out of the controller as is SVC.

We are going to see more commodity based storage appliances managing/federating/virtualising the storage access layer; these appliances will provide all kinds of interesting functionality but increasingly they will run on top of x86-based Linux hardware. It won't be about the number of processors, ports; it'll be about building efficient and reliable architectures using standard components. Some of these appliances will ship as virtual machines; bring your own hardware.

Aha...perhaps that's Virtualisation 3.0; I'm going to wait for Virtualisation 3.11 tho'!

Elephants in Your Data Centre

Marc 'StorageRap' Farley talks about the Elephant in the data centre and it being storage utilisation! And he's right, it's a big Bull elephant sitting in the middle of a lot data centres; trumpeting loudly. But I'm not sure it's the only one.

Defaulting everything onto Networked Storage?  Nothing wrong with Networked Storage but does everything have to go on it? It's an expensive option, don't discount locally attached/infernal disk.

Storage Software Licensing? Only need 2 Terabytes of Snap? Got a large array; well unfortunately you need to license the lot.

You can only use half of the disk you paid for, not because of over-allocation but because you've topped out the performance of the spindles. And re-organising them is a right royal pain.

8 Gig FC, you probably don't need it. Look at your SAN utilisation figures, you could probably stick with your old Brocade/Cisco/McData switches for a long time before you top them out.

It comes down to a couple of things

• Only pay for what you use
• Use what you pay for

EDIT: Thanks for pointing out the typo in the first Marc but I meant the second wording. It kind of plays into your hands; over allocation etc means that you pay for capacity you can't use.

Thin provisioning is only half the story and yes Marc, I know that 3Par do a pretty good job of telling the rest of the story. This is going to be the year where we look at what we do, why we do and how much we pay for it.

Appliance of Science

As those of you who follow my inane twitterings may have noticed, I've been playing with Virtual Storage Appliances.

It started with the OnTap simulator from NetApp as  I wanted a play with OnTap; try a few things out and generally familiarise myself with it before we installed any. Also gave me the chance to steal a march on the techies who work for me. It's a fully featured simulator with the exception of FC connectivity for obvious reasons; it's an excellent playground and if you want a chance to learn NetApp, debug scripts, mess about with clustering etc; it comes highly recommended but....it is just a simulator and does not support anything like enough disk to make it useful. It could be useful but it needs to provide capacity. However I am not sure that NetApp are really interested!

My appetite whetted, I moved onto having a play with the Celerra simulator. Firstly EMC need to be given massive credit in that anyone can get hold of it; you don't need to be an EMC customer! EMC giving something away, I hope the dark lord has wooly mittens! Secondly, you can actually put useable capacity behind it! Well certainly enough capacity for it to act as a home NAS box, it was fun having an Enterprise-class NAS as my home NAS. And hey, I can put non-EMC disk behind a virtual Celerra ;-) Much kudos to the EMC guys!

Just recently, I've decided to give a few more a play; I've just built a Fishworks environment. Easy-to-use, easy-to-configure, a tad resource heavy but a really nice GUI. I think I could grow to really like this environment. I think I could get a decent amount of disk for me to use for my home environment, I need to reconfigure the virtual environment but definitely useable. Hey, Sun are a good software company, I wish they'd stop messing about with hardware! Actually Fishworks could give both EMC and NetApp a run for their money if it was shipped as a fully supported VSA. It could be very disruptive at the low-end to start with.

There are lots more to try; OpenFiler, FreeNAS, Lefthand and Falconstor; some on evaluation licenses, some fully useable!

I think that we are about to move into some really interesting times; installing VSAs onto whitebox hardware and building our own storage arrays. Using hypervisor technologies to provide clustering and resilience. The world of NAS is going to change I think; more and more storage will be provisioned from VSAs; test and development environments to start with but moving into low-end production.

Products like Atmos are going to ship in appliance form at some point; Atmos is a software product masquerading as a hardware solution. I can see EMC shipping a supported VSA for Atmos, Celerra and Avamar; there has been a rumoured Centera appliance for years now. If Sun have any sense, they'll ship a supported version of Fishworks aimed at virtualised environments. And NetApp...we'll see!

I predict that NAS will become a software product; there may be some pretty stringent hardware qualification requirements if you want support but it's going to come. Whilst we are it, might as well throw iSCSI into the mix; easily done in software!

And FCoE, now life becomes interesting at that point, think about it.

Oh yes, another company needs some credit for making this all possible; our friends at VMware, you can get all the mentioned environments up and running on freely downloadable versions of VMware products; ESXi is my current favourite.