It Cuts Both Ways

You can't have it both ways and it appears that some in IBM want to have it both ways. IBM historically have enjoyed benchmarketing and have historically kicked EMC for not submitting Symmetrix for SPC. EMC have responded that customers don't run SPC as a workload and it does not realistically reflect a true customer workload. At which point the whole storage industry blows a raspberry and calls them chicken.

But read this comment from StorageBuddhist on my blog entry 'Wither XIV'

"The closest you will get to an XIV benchmark is probably my blog post from April http://storagebuddhist.wordpress.com/2010/04/06/xiv-ms-exch-2007-replicated-performance/ but being a non-trad box, XIV has a very non-trad performance profile. We recently heard from a customer who has got more than 100,000 IOPs out of their XIV in the course of normal business duties - something I had previously assumed was only possible in the labs." 

I'm trying to work out what he is saying; is he saying that XIV is only suited for non-traditional workloads?  Why won't IBM publish standardised benchmarks for XIV, they do for everything else they ship? If they could find a way of benchmarking the free pens they hand out, they would! I know he doesn't mention SPC but that's the standard benchmark for storage that IBM support.

You can't call EMC out for not submitting their arrays to SPC and then not submit one of your own because you know it will perform poorly.

Either SPC is a valid storage benchmark or it's not! Or perhaps XIV is only good for running some workloads? You see if that is the case then no matter how simple it is to manage; it does mean that to cover all my workloads, I am going to need multiple types of array and that builds in complexity to the environment. And at that point, does not much of XIV's selling point go down the toilet? I will still need a traditional storage team to tune and manage those applications which need a traditional array to perform and don't only perform well when run from cache. 

Do I think EMC should submit their arrays to SPC; actually, I do. Do I think IBM should submit all their arrays to SPC and not cherry-pick which they do, abso-blooming-lutely I do. Or IBM should come very clean as to what applications those arrays they do not submit to SPC are not suited for. If they do not, they call the whole validity of the SPC into question.  

Comments

[Martin G]

Yet they will still publish that top speed of a tractor. Sorry, if IBM are unwilling to publish figures for their whole range; we can only assume that they have something to hide.

And lets be honest, if a Ferrari could do the job of a tractor at a comparable cost, you'd buy the Ferrari wouldn't you?

IBM have been quite willing to talk about XIV doing ridiculous amounts of IOPs but generally unwilling to submit it to any kind of public benchmark. Do I think IBM have run SPC against XIV? Yes, I do and does it expose XIV's serious performance limitations? Of course it does. Perhaps it is a carthorse or a tractor; time to admit, it is only good for ploughing fields then.