Storagebod's Blog

Arrays Now Good Enough?

I loved this blog entry about Web 2.0 because the last paragraph is beginning to have relevance in our own world of Storage.

"The lesson to take away is that, in this cycle anyway, the hard architectural problems have been solved to a "good enough" degree, now its about making them easier to use etc. Its like post war cars - once it was sorted out how they worked, where to put the pedals etc, it became largely a styling game for quite a while."

Despite a lot of the huffing and puffing by the various vendors, I think that we are about to enter a cycle where this is true for array-based storage. Most vendors, if you have a serious and open conversation admit that their competitors arrays are pretty much good enough for what-ever you want them to do. Of course, there are a few features which differentiate but they are all catching up with each other in this space.

It will get harder and harder to choose between vendors based on features; it is going to come down to useability, manageability and reliability.

The array is heading towards functional maturity and after this year is done, I'm expecting the battle-ground to move to the skies.

Talk to me....

In what might be a foolhardy experiment, I am going to put a contact email on the blog. Since I've started using gmail as my access method to my email, I've found that the spam-filters are so good that I rarely see any spam. So I thought I'd set up an email which I publish and if you want to contact me about anything on the blog you can do so, storagebod@spellen.org will reach me. Now if you've already got my email address, keep using that but to be honest, I'd prefer that storagebod related emails go to my personal domain.

Why? Well, I try and avoid reading my work email outside of work and my personal domain follows me; I've had it for a number of years and it's followed me through a number of jobs.  This doesn't mean that I'm about to change companies, the market isn't that good! But at some point it is possible I might go and do something else and I like to think that this blog which is nearly 6 months old will continue!

On another note, you've probably noticed that I'm now contributing to Gestalt IT and all my Enterprise IT related articles will also appear there. This means that I'm probably going to go off-topic here a bit more often. Feel free to follow me there or here, it's your choice.

The Value of Curiosity

One of the questions which often comes up in interviews is what is your greatest strength? It’s a lazy question and comes from the standard text. And everyone has their answer off pat, I ask it hoping to get an original answer and only rarely do I get answer which surprises me.

Anyway I have my answer down pat as well…I’m intensely curious,  I’ve never grown out of the why stage of my life but instead of bugging my parents, I bug myself and go and find out myself. 

Now if the interviewer gets the answer, I probably want to work for them and if the interviewer doesn’t; it’s probably never going to work out. I think curiosity is a much under-valued trait, wanting to know what’s on the other side of the hill is what drives us forward.  Hey it might kill cats but I'm not a cat

And that’s why I love things like Twitter and Blogs, it lets me find out lots of stuff; a lot of it useless but some of it profoundly useful. I get to talk to people who I wouldn’t normally have a chance to talk with. That’s why sometimes the various vendor blogs that seem to enjoy throwing mud as opposed to telling me about their product and what it can do for me really irritate me at times as they add very little to my learning.

We’re heading into a big year of product announcements, can I respectfully ask that the vendor bloggers do their best to talk about their products and what they can do? And not what the opposition can’t do? If I see a cool new feature, I can find out for myself if someone else also has it.

p.s the funniest answer to the question ‘What’s your greatest strength?’ I’ve heard of is a guy who worked for me who during a HR interview answered ‘Okay, I’ll give you a strength and then you’ll ask me what my greatest weakness is? I’ll give you a weakness which I cleverly twist into a strength. So my greatest strength is I don’t answer stupid questions!’. He’d already got the job at this point but he started off with ‘TROUBLE’ written on his file.

V is for value??

I have a problem with Kostadis’ latest posts on virtualisation and especially the ability of using vSeries as a virtualisation controller to reclaim unused disk and improve efficiency etc. The problem with the vSeries going into an existing environment, is that of the triumvirate of SVC, USP-V and vSeries virtualisation appliances/controllers is that it is just about the most disruptive thing that you can do.

SVC and USP-V will go in with no data migration and you can simply use them to virtualise your existing physical LUNs; that is the beauty of keeping things relatively simple and not relying on an additional abstraction layer such as WAFL. You can then migrate into the native SVC and USP-V formats at your leisure if you wish but you are not forced into making changes which potentially lock you in to the virtualisation vendor.

This is extremely important in today’s environment; firstly it minimises the amount of swing space that you require to do the conversion into a virtualised environment, secondly it minimises the amount of potential outage and thirdly it’s not a one way trip.

If all I want to do is reclaim unused disk and I am prepared to take the level of disruption that putting a v-Series into my environment; I suspect I might be better spending the money on analysing my existing environment and coming up with a better, more efficient lay-out and working at doing things better in my current environment.

For example, I would consider looking at Virtual Provisioning in my EMC environment; sure the licenses cost but it’s not going to be as expensive as going to a vSeries. I could convert from BCVs to Clones or Snaps. There’s a multitude of things which could be done before going virtualised. And once you've got your environment fixed, then take up NetApp and their space guarantee; don't make things easy for them!

Don’t fix with new technology that which could be fixed by being a little smarter with your existing technology. You should always ask the question when being sold something new, can I do that already? It is amazing how often the answer is yes!

WAFL is a great technology with some great features but putting it in as a virtualisation technology would be an expensive mistake at the moment.  Put NetApp in because you want to use NetApp but don’t put it in to virtualise your existing environment unless you are prepared for a whole lot of work. Take it from me, I’ve looked at it.

If you simply want to virtualise and build a consolidated pool of disk, you might well be better looking at SVC or USP-V. If you are looking at re-engineering your environment, NetApp is one of many companies you should look at but you know that anyway.

Storagebod's Performance Check

Every now and then I see a press release which really irritates me; all vendors have done it and often it is when they have a dig at a competitor! The only person who is allowed to have digs at vendors is me (okay and other users). This time, Pillar got on my nerves with an email which pointed me to this; you see EMC have decided not to take part in SPC for whatever reason and NetApp decided that they were going to go ahead and benchmark the Clariion anyway. I thought it was wrong at the time and if I'd been blogging, I'd have said so. And it's even more wrong now for Pillar to be using those figures now; especially since the CX3 is 'old technology'.

Obviously, the solution is for EMC to simply take part in SPC...wrong! The solution is for the other vendors to criticise and even suggest that EMC are hiding something; that's fair game! But to benchmark EMC kit and somehow suggest those figures are fair is cheap!

So, I have a suggestion; on an annual basis, a council of users chaired by myself will sit in a pub and fortified with several pints of Old Peculiar, we shall come up with a performance benchmark. It will change every year and will have the beauty that it reflects today's infrastructure and application design process. We will then calculate a budget; this will be decided by throwing three darts in the general vicinity of a dart boad; trying to multiply them together in our addled brains and probably sticking a few zeroes on the end for good measure.

The vendors will then have the task of specifying an array for the budget stated to perform the benchmark. After the council has sobered up, we will follow the best practise guide-lines to configure the arrays. The benchmark will then be run and the winner will be declared 'King of the Hill' for the next twelve months. This will give them the right to be mean to all other array vendors!

Extreme Cash Cow - Redux

I’m still getting comments on the ‘Extreme Cash Cow’ entry I wrote last year as a diatribe against the current state of the SRM market. I feel it’s probably about time that I updated it.

Firstly, since then I have moved jobs and no longer have responsibility for day-to-day storage management, I do obviously keep an eye on things but unfortunately it means that I cannot really influence how SRM develops in my organisation. This is a bit sad as actually I had a very positive response from EMC who took the brunt of my diatribe and I have been unable to take them up on their offer to work with me on understanding what I would like to see from a SRM product. I do hope that someone in the organisation I work for does take them up and help them understand what a large storage end-user requires and where the issues are.

Anyway, I thought I would put some thoughts together about the challenges that SRM tools or at least pose some questions.

Is the problem one of scale and complexity? If you look at what we expect the SRM tool to do, we expect currently expect it to understand our storage environment end-to-end. So look at what an SRM tool needs to do.

1) It needs to understand the array and how that is configured – easy

2) It needs to understand the switch fabric – fairly easy

3) It needs to understand the IP fabric – moderately hard

4) It needs to understand the hosts/servers, including virtual – moderately hard

5) It needs to understand the applications – hard

6) It needs to be able to correlate all the above information into useable and consistent model – potentially very hard

So to be fair to the SRM vendors, what they are trying to do is non-trivial and we the end-users don’t always make their jobs easy. We have a duty to ensure that organisational standards are set, adhered to and maintained otherwise the data consistency checking becomes horrible. We have to give them a chance.

Do we want an end-to-end management tool which allows us to understand our whole IT infrastructure because the relationship between storage and data is intrinsically linked?

What do we actually want from a SRM tool which will make it useful to us so that we do not carry on cursing the vendors and writing our own scripts? Perhaps we should hand over the contents of our individual script/tools directories and say, we want a tool which does all this and does it reliably. Perhaps the SRM vendors should send out an investigatory team wearing red shirts to discover what the storage civilisations are up to?

We can probably say that we don’t want ECC and its ilk; perhaps SanScreen is closer to what we want. I suspect that is very much the case; we do not want an all-singing, all-dancing provisioning/configuration tool but we do want something which gives us an immediate view of our storage environment and allows us to drill down the layers into the individual components getting performance, capacity and configuration details? It would be incredibly useful if it understood the reality which is a heterogenous storage environment with SAN, NAS and in future Object/Cloud.

And vendors if you continue to expand the number of different storage families in your product range and do not standardise on your management APIs, interfaces etc, you are making your job harder. And even in a product family, 37 varieties of LUN are not making your job any easier. As part of the development track of any new feature; the question needs to be asked, how will this be managed and the question needs to be asked early in the development cycle.

So what do you want your SRM tool to do?

Deal or No Deal?

Stephen has a post here about pricing, about getting close to your vendor and developing a relationship with just a few trusted partners. Nice idea but in any relationship there needs to be some tension to keep it fresh and alive; otherwise you find yourselves doing something because you have always done so, it becomes comfortable.

Now comfort is all very well at home and in your personal life but when you are spending lots of money (your lots will vary) with a vendor, comfort is not good. I have walked into situations where the position has been far too comfortable and ultimately it becomes dangerous for both parties. You need to shake things up once in a while.

1) Single vendor relationships are not good to drive value. Competition is key, this does not mean that every bid should be competitive, there aren't enough hours in the day and everyone gets tired. But every eighteen months, pick a technology area and review it. Do both a technology review and a commercial review. Your review cycles may be shorter, it depends on your work load.

2) Review street prices regularly; there are a variety of sources for this, some formal, some informal. Vendors do not like it but they know it happens. But it is important to understand why prices differ; it could be size of organisation or it could be a prestige thing. Let your sales-man know that you are paying above street price and that you are reviewing things.

3) New requirements need to go competitive. A sign that things have got too comfortable is that you simply default giving new business to the incumbent.

4) Be aware of the market, talk to the incumbent about competitive products; let them know that you are aware of the competition. Attend trade-shows; talk to other vendors, I know sales-guys are irritating but when the incumbent phones up and finds you are in a meeting with their rivals, there will be a moment of doubt.

And when you are doing the deal, here's a few tips and thoughts. I am not a procurement expert but I have spent a few million here and there on storage.

1) List price is meaningless; discount levels are meaningless. Vendors should produce a cost of manufacturer and then try to negotiate a premium on this. Vendors also hate breaking things down by line-item as it reveals that you are paying massive amounts for commodity items. Do not let a vendor flannel you with a cost for a solution, get it broken down and understand what you are paying for.

2) Maintenance; you should always be able to negotiate improved maintenance terms. Hardware maintenance is pretty easy to get extended gratis, software is often harder. Review maintenance regularly; if a piece of software is at or close to it's terminal release, consider dropping the maintenance. If you really need it at a latter point, you can often re-instate, you'll have to pay the back maintenance but you'll likely not need it anyway. Make sure this is contractually agreed.

3) Technical refresh/take-out; if you are refreshing with the current vendor, only pay maintenance on one lot of kit whilst the refresh is happening. If you are refreshing with a new vendor, agree that the new maintenance/warranty period only starts when the migration is complete. Always try to negotiate a trade-in.

4) Software licensing; try to negotiate a pay for the amount you use as opposed to the pay for the whole frame!  And always try to agree that software licenses are transferable between frames.

5) A vendor TCO model is worthless unless they are willing to guarantee it without caveats. If they think their kit will save you money, skin in the game is key!

6) Training; I have had teams which have had more training than any other team in a department because I ensure that any deal is sweetened by the provision of training for 'free'. Big deals should come with free training and I am amazed at the number of people who do not leverage this.

7) One-off-deals; one-off-deals, you know the end of quarter/year specials? We all do them, we all regret them at times. Plan your one-off-deals, you know they're coming but treat them like a normal deal. You know when the vendor quarters/year-ends are; so try and align your procurement cycles if you can. And I've never had the pricing on a one-off-deal pulled because I have missed the cut-off.

8) Guaranteed price decline, the cost of kit goes down all the time; ensure that you've got a guaranteed price deflator on a quarter-by-quarter basis to reflect this.

You will not get all of the above but at least vendors will know that you are serious, that you are thinking about things. And if your sales-man agrees any kind of special, non-standard terms; get it in writing and keep the evidence. Sales-men move around a lot and the next guy may not honour a verbal, gentleman's agreement; get the evidence.

I am sure there's more tricks that I have forgotten or not even be aware of; please share!

How Many...

The VSA testing is taking longer than I intended. At the moment, I've got six installed but under VMWare workstation, not ESXi as originally intended. Early next month, I'm going to get an ESXi supported SATA card so that I can start testing properly. The Virtual Storage Appliances are as follows

Sun Unified Storage Simulator: Supports both iSCSI and NAS. Easy install, no time limit. Standard install is capacity limited but I think is easily hackable to go higher by changing the size of the virtual disks.

Open Filer: Supports both iSCSI and NAS. Easy install, no time limit. Not capacity limited.

NetApp Filer: Supports both iSCSI and NAS. Not strictly a VSA and you install on top of a Linux build. NetApp could do with shipping it as a proper VSA. Severely capacity limited. Good for playing with OnTap but not a serious tool. Only downloadable by NetApp customers!

Nexenta: Supports both iSCSI and NAS. Comes in a variety of versions; I'm using the developer's edition which is not time limited. Easy install, no time limit. Capacity limited at 1 Terabyte.

Lefthand VSA: Supports iSCSI only. Comes in OVF format, needed to use VMWare convertor to get it into a format useable by VMWare Workstation. Install has a gotcha, need to make the virtual disk you are going to use to store data on sit on SCSI id 1:0 or it won't pick it up. Time limited and only supports a single virtual disk but you can change the size of this, supports capacity up to 2 Tb.

EMC Celerra VSA: Supports both iSCSI and NAS. Pig of an install initially, no time limit. Not capacity limited as far as I can see. Chad's blog is a really great place for information and general tips on how to get the most out of your virtual Celerra. He needs to be applauded for the work he has done here.

I've just registered for the FalconStor appliance and I'll see if they come back with an okay. I can't see why they won't.

Early feel on the performance is that the Sun and Nexenta VSA's are easily the fastest; both are based on Solaris with ZFS. All of the VSA's are useable tho' but NetApp need to step up a bit; repackage and support increased capacity.

Certainly in the SMB space, this is going to be an interesting area but I can see potential in VMWare environments as well where a VSA could be provisioned per blade enclosure for example keeping I/O in the enclosure.

If any vendors reading this have any more VSAs they want to tell me about, I'd be happy to look at them and if any of them want to give me a registration key so that I don't run out of time, it'll save me re-installing.

I've also been toying with my own 'VSA' build using Linux but not only does it support iSCSI and NAS; it also has the OpenFCoE code as well. It's ummm...fun, however I wouldn't trust it with my data let alone yours!

Small Stuff

I've been recently reading Dave Hitz's autobiography which covers a lot of the history of NetApp, how it came to be etc. It's an excellent read; not especially technical and gives you an insight into how a start-up came to be one of the largest storage companies in the world.

One of the things which was interesting though is how far NetApp have come from the starting vision; a company which was going to make cheap, small, network-attached storage appliances based on commodity hardware. How far are things like the 6080 and the 3170 away from this vision; in fact, NetApp seem to be surrendering the low-end of the market and have very few product offerings in this space.

Fortunately, there are some interesting smaller vendors who are building things on commodity hardware; this caught my eye recently, the QNAP TS-809 Pro, fully stacked it works out at about £250 per terabyte for a 12 terabyte system. It supports iSCSI, the normal NAS protocols and you can run a whole raft of other stuff as well.

I'm not suggesting that vendors like QNAP are going to be challenging NetApp et all any time soon but perhaps if you replaced the Operating System with OpenSolaris with ZFS or wait for BTRFS under Linux, the vision of Storage Appliances built cheaply out of commodity hardware and not charging crazy overheads may come true. Pity NetApp currently appear to have lost that vision, it was a worthy one. EMC were always about building storage arrays for the big boys; NetApp was supposed to be building storage for the rest of us.

Don't Be Blinded by the Flash!

TSA has done a good if slightly skewed write-up on the current positioning of Enterprise Flash disks and what the various vendors are doing with them. And at the moment, they are all using them as faster spindles; as faster replacements to spinning rust. In the same way that I could remove my laptop/pc drive and replace it with a SSD; I can do it in an array at some extortionate price.

I know there’s been a fair amount of tweaking etc to get them into the various arrays but it doesn’t appear to be the proverbial rocket science! So can we have some rocket science now? 

We’ve actually looked at leveraging EFDs in our current environment but at the moment as well as the initial expense; the actual retrofitting into the environment is either a lot of work or it means buying a lot more SSD than we would possibly need. The layout in a shared array becomes arcane and possibly more complex than a mere ‘bod can hold in their heads. This does not mean that we won't do it but it's not a magic bullet; it'd probably be a lot easier if we were not looking to implement into an already running business critical environment.

I think if the industry simply looks at these as faster disk spindles; we are missing a huge opportunity to make storage a lot more efficient. We need to understand a lot more about what actually goes on at an application layer; for example, which individual files are hot and need to be stored on faster storage. In a hot LUN (a concept which must die); what percentage of the data is actually hot, is the whole LUN a hotspot or is it a tiny chunk of that LUN? If it's a tiny proportion of that LUN, is it not really wasteful to put the whole LUN onto this expensive resource.

Things like Sun’s ZFS hybrid disk pools might help a lot; we need to see more collaboration between the Operating System/File System and Storage Vendors to get this all to work efficiently and cost effectively.

No, you don’t need to make huge changes to use EFDs but to use them efficiently and not simply give the greedy storage vendors even more money; you might want to think about making huge changes.

Of course, by the time we've fixed the problems, EFDs will probably have come down so far in price that the economic issues will have gone away. So perhaps the solution is for all the vendors to take the hit now and bring the costs of EFDs down to the cost of traditional disk?