Storagebod's Blog

I Want One of Those

I've spoken to these guys before and I wasn't that impressed; why, it all sounds like an appliance-based Stacker (remember the software-based disk compression for your PC). But they didn't tell me about their killer feature, a little meter on the side of the appliance showing the current compression ratio. Now if they could tie that to how much I pay per terabyte, I could have a real-time ticker showing how much their device has saved!!

And that would give me a real metric to give to the boss! You see simple things please simple folk!!!

Ole!

Okay, I'm at EMC Customer Council in Madrid, obviously there's a lot I can't blog about; NDA stuff and all that. But TSA is here so I'll finally get to meet the Anarchist in person. There's lots of interesting sessions and I'm going to have a hard job choosing the electives tomorrow.

And there's always interesting conversations in the bar! So I'm not sure how many entries I'll do over the next couple of days!!

I got a nice room upgrade due to the hotel keeping me hanging around whilst they got my room ready, after the third time going back, they just put me into a room large enough to play cricket in!! No free complementary bottle of champagne tho'!!

Get Focused....

I was interested to see Chris Mellor's story that NetApp are pulling all their development resource off of existing workloads and focusing on getting GX into OnTap. I used to work for a guy whose maxim was 'get focused or get f****d' and in NetApp's case it looks like get focused or get cluster f****d. I see lots of vendors touting clustered NAS solutions; HP, Exanet, iBrix, onStor, IBM, Isilon too just name just a few; all of them bring up NetApp and GX, all of them are very disparaging about GX and probably with good reason but I think NetApp have finally been poked with the stick enough times and are really waking up at last. They can't keep telling us that GX is great but the next release will be the one which really works; it needs to be the one which really works, offers all of Ontap's functionality and it needs to be with us in the next six months.

If NetApp get GX properly integrated into Ontap, they are going to have a big job countering all the FUD out there but at least they will have a working solution; I'm waiting to see EMC's play in the Clustered NAS Space. I can see an acquistion but then they're going to have to be careful to not to do an NetApp and take 5 years+ to integrate it; if they do, they can forget it or just capitulate the high-end.

And then there is Microsoft's move into Cloud computing and what that means for the market; Chris is predicting doom and gloom with mass consolidation in the storage market. I think even without Microsoft's move into the cloud, we were going to see consolidation in the industry; there are just too many array vendors at the moment. I think he's actually massively under-estimating the number (and that includes that fact that LSI make arrays for a number of people and Hitachi's storage is rebranded by HDS, HP and Sun).

The cloud could also spell trouble for Microsoft, its a move into a business that I don't think they understand. Can you imagine an outage which takes out tens of thousands of businesses? Amazon's outage shows what can and will happen. Perhaps a couple of outages of this sort of scale will scare enough people and will save the storage vendors.

Whatever happens large-scale clustered storage solutions are going to be very important; how these scale-out storage solutions are built and delivered and on whose platform will be interesting. Maybe storage virtualisation to enable anybody's spinning rust to be utilised will be very transient because there just won't be that much choice? Maybe MAUI will be EMC's silver-lining to the cloud? And maybe NetApp will get GX working just in time? And maybe someone will develop some management tools as opposed to administration tools.

Flashman Farley

So 3PAR come to the party and talk about Flash drives in the InServe range. I've extracted the following from Chris Mellor's article on The Reg because I think it sounds particularly interesting.

"The InServe software can recognise a tier zero of SSD storage and place data there, as well as move the data to a lower tier when activity rates and/or time and policies dictate. The InServe will avoid interleaving SSD and hard disk drive I/O so as as not to bottleneck the SSD data stream."

Does this imply that 3PAR are going to be doing some kind of automated storage tiering? If it does, I think that this is more important than the SSD announcement? It also implies that the SSD will have a seperate back-end, perhaps be a seperate 'array'. Sounds a bit like IBM's potential approach with Quicksilver and SVC.

Amateur Punditry!!

Now every array appears to have thin provisioning either today or at least roadmapped for delivery within the next year, I am wondering what the next ubiquitous feature or trend in storage will be. Dedupe appears to be obvious one, certainly within NAS storage; Dedupe is going to work well in alot of NAS environments and if you look at most NAS heads, they probably have spare CPU cycles to enable this. I'm sure we will see offerings from all of the major players and NetApp is already there. Are we going to see it across the piece and for SAN attached storage, not sure at the moment?

The next thing I am expecting to see (with one exception) is the move across the board to a chunk/extent based architecture; it enables wide-striping and thin-provisioning in most of the more traditionally architected arrays. I can see 3Par's 'micro-raid' type architecture becoming popular as well. The exception will be NetApp; they will stick with the WAFL architecture and will continue to be the odd-bods. As the chunk/extent based architecture becomes more prevalent and more understood, I can see it leading to

1) Automated Storage Tiering; with chunks being moved between different storage medium automagically. This will allow SSDs to be utilised economically, you only need to look at the Compellent architecture to see what might come. Compellent might become the next 3Par in that they help to define a technology and become the flag-wavers; they'll have to be carefull that as flag-wavers they don't get squished by everyone charging to the flag.

2) SAN DeDupe; DeDupe will be done at the chunk level, this means getting the chunk size correct is going to be crucial. But this will take a bit longer to appear I suspect.

I'm also expecting to see more RAIN architectures built round commodity hardware; more than just storage-related workloads will be built into the nodes e.g. transcoding in the digital media space.

Lastly I think Xiotech and Atrato are going to be interesting companies to watch; similar concepts and just made for chunk-based architectures. Someone will pick them up in the next year or so.

Anyway, just some thoughts from user-land!

Who'd be a Storage Manager?

A lot of storage management is really rather easy; we like to veil it in mysterious terms and generally pretend that it is a bit of a black art. There's probably only one group of IT people who are worse and that's the Unix guys. I've been both and I should know but the interfaces are getting easier and it probably wouldn't take that long to turn a decent sysadmin into a storage admin. Okay, the command lines take a little longer to learn (and that's generally due to the screwy and arcane syntaxes that we all love) but the GUIs are pretty intuitive.

But life is getting interesting; not the day-to-day but actually planning and proper management of the environment. Why? Because we are beginning to use techniques which mean that our arrays are going to lie to us on a regular basis!! And this is going to bring with it a number of real headaches; how do I mean lie?

Well Steve Foskett's blog on thin provisioning and Symantec's making VxFS thin provisioning aware kicked off some thoughts, it's been something which has been niggling for a while.

Firstly, lets take thin-provisioning. I assume everyone reading this blog understands the premise of thin-provisioning? We can thin-provision storage but it only gets 'consumed' when data is written to it for the first time; this hopefully allows us to cope with some of the profligacy of our customers who always ask for more storage than they are actually going to use.

Some really nasty traps do lay ahead; for example we will over-commit our storage i.e we will have more storage logically allocated than we have physically available; it's the only way of making thin provisioning really useful and save us money. However, we have to be very careful to monitor what is actually being used and the rate of growth; otherwise, one morning, we might come in and find that all our storage is gone. I have suggested ways that we can help to prevent this i.e convincing sys-admins not to allocate the whole LUN for example but at some point we are going to run out of space.

What happens when we've committed more space than the array can actually be expanded to? I can see some really nasty and rapid migrations having to happen if this is not carefully monitored. So if you are going down the thin provisioning route; be careful, you have made your life easier generally in the day-to-day but now you've got to keep a tight handle on capacity/demand management. And you'd better make sure that you do have migration strategies to cope with the over-commit scenarios; plan to migrate and plan to do it probably more often than you do already.

Now, there is another really sneaky trap laying in wait; de-dupe of primary storage. De-dupe of primary storage is coming main-stream slowly but surely; it will happen! A-SIS already has the capability, expect EMC to do things with Avamar running within a head and IBM do something with Diligent running in an XIV (lots of CPU power in the RAIN based arrays) or in a partition on a DS8K.

So there you are happily deduping storage; ensuring there's lots of commonality and lots of space savings. And then someone does something stupid; patches an operating system, let's say half a dozen blade servers with 30 VMs on? Suddenly you see some commonality go; perhaps updates some Oracle binaries, re-encodes some files; any manner of things. And next thing you know and pretty much without warning, you are out of space.

Okay, same thing can potentially happen with Snaps and all it takes is some-one to run a big update job and you find that your reserved pool for Snap space is gone (I've seen it happen).

None of the above is a reason not to use Thin Provisioning, DeDupe or even Snaps. But we need to get better at managing what is going to be an increasingly volatile environment where at the moment a lot of the tools lie to us. Our day-to-day admin has got easier and our users know this, they are demanding storage quicker and demanding that we use the facilities which our vendors are telling our CIOs etc are going to save them money but just lets take care that we have the tools and we have techniques/disciplines in place to enable us to manage our storage well.

Our colleages in the Server and Network disciplines have it easier than us; if they run out of capacity, pretty much most of the time, things just run a little bit slowly...things stop if we run out of capacity.

 

WAFL is a Platypus!

I've decided that after reading Kostadis' interesting and well-written series on why WAFL is not a file-system; that it is a platypus! In much that same way that a Platypus is mostly mammalian but does some-things in a distinctly non-mammalian way; WAFL is mostly a file-system!

So from now on; I shall be refering to WAFL as the Platypus Layer!

It will make such papers as this much more fun; replacing the words 'file system' when 'file system' refers to WAFL, with platypus!

Kostadis' problem is that NetApp's own literature refers to WAFL as a file-system all over the place. In the original patents, in Dave Hitz's blog and in their own marketing!

And at the end of the day, as I pointed out on Kostadis' blog; it doesn't really matter, as long as it works sufficiently well to meet my needs and provides the features I need.

PS I like the Platypus; its existence proves that the divine might have a sense of humour and as such it should be cherished! I'm not saying WAFL proves a sense of humour but it does show originality and the ability to go against the herd, another important quality, especially when the herd are laughing and saying that you are mad!! You might be mad but you might also be inspired.

Storage Management Tools

I was at the UK Storage Management Forum yesterday, in many ways a misnomer as it is EMC and Control Center focussed. Now ECC is a good tool although everyone seems to have issues keeping it stable and responsive, DCPs seem to be the bane of every ECC implementation but it gets better every release.

However, it seems that the nirvana of a universal provisioning tool is a long way off; at one point EMC did appear to be starting down the road of putting in the capability to manage third party arrays. By manage, I mean configure and provision; it seems that EMC are taking a step back placing all the focus for 3rd party arrays on simply monitoring them. I don't really blame them but it's a pity as it means that Storage Teams are still going to have to live with multiple tools to manage the environment.

So is there any light on the horizon? At one point, I had hopes that Onaro might produce something but not sure where NetApp are going to take that. They certainly seem to gone a bit into stealth mode and Onaro really relied on a good ECC implementation to work; I still remember conversations with them where they claimed to be agentless, not really true, they just were parasitic and sucked their information from everyone else's agents. But it had potential, unfortunately NetApp seem to be 'doing a Sun' and killing a product.

So who else? I've not looked at IBM's offering for a long while, it must have got better (it couldn't have got worse!). Hi-Command, Storage Essentials? What else is out there? Anyone got any suggestions?

A Stacked Feature Request

I had Brocade in earlier in the week and we were chewing the fat over SAN futures, FC vs iSCSI vs FCoE etc and of course the normal discussions about Cisco's dominance of the IP space and whether Brocade can do anything to halt the march of Cisco into the SAN space? Can they leverage the Foundry purchase and move into Cisco's space and at least become a credible alternative in the Enterprise Data Centre space? It's going to be hard for them to compete but we all like to cheer the under-dog in the UK and I hope that they do; the IP network space needs someone keeping Cisco honest, dominant players really aren't that healthy.

And the conversation moved round to Virtualisation, meandered round VMware as these things normally do; then onto Storage Virtualisation and onto Brocade's DMM product. I'm always interested in things which could make migration easier and at the moment we tend to do it mostly using host-based mirroring. Everyone has a migration tool but no-one has tool which will do Lun-stacking automagically; by Lun-stacking, I mean the ability to take multiple luns, migrate and concatenate them at the same time.

For example, we've got a situation where we are beginning to hit against the DMP limits for the number of Luns; what would be really useful is to be able to migrate these smaller Luns onto larger Luns but merging them at the same time. We could migrate to larger Luns, then consolidate and then drop some of the Luns but that is a big, time consuming job. So guys, if you want your virtualisation tools to be really useful to me; make them do that!

Thin Provisioning - Saviour of the Universe

Obviously, it should be 'Flash, a ha, Saviour of the Universe' but there's enough going around about flash at the moment. There's also a lot going around on various blogs about Thin Provisioning and especially space reclamation i.e how you get space back from file-systems after files have been deleted etc. It seems that the approach is to make the file-system aware of the storage and get them to communicate allowing deleted blocks to be reclaimed.

All fine and funky but to do this, the file-systems are going to need to be changed; at the moment, Veritas are on-board but we really need Microsoft on-board, perhaps engage the guys who are working on BTRFS for Linux and perhaps Sun for ZFS. If the industry does not get these guys onboard, this will be another good idea which peters out unfortunately. Oh yes, almost forgot; you've got to get VMWare onboard as well because I could see some really bad interactions with things like linked-clones in the future.

Whatever happens needs to be standards based at some point; it must be robust, must not impact non-thinly provisioned storage. Why standards based? I do have cases where a server sees storage from different vendors; I don't want to have treat file-systems differently depending on which storage array it is provisioned from. It'll just add another maintenance headache to my life.