Storagebod's Blog

Automated Questions

The Storage Buddhist's blog about IBM Easy Tier with SATA and SSD here started me thinking. 

And I quote from his blog

'Easy Tier was left to automatically learn the SPC-1 benchmark and respond (again, automatically)'

Very cool and IBM get some good results from this but I have a few questions and this goes out to all vendors who are doing automated tiering of some sort (I include NetApp's PAM here)

1) How long does the array take to learn the workload and how quickly does it take to achieve an optimal state?

This is very important because it impacts any new application deployment. Do I deploy to the fastest disk and artificially throttle the disk to required SLAs? Users are funny you see and if they discover that application performance is degrading over time, even if it is within the SLAs, they generally complain. Or do I deploy to the slowest tier and let the array tune itself to meet the required SLAs? Now, this might be the worst possible time to impact performance and may impact the user acceptance of the application?

Do I have to come up with some artificial way of simulating the production load and run that for a period to enable the array to tune itself prior to any real users? If so, that has impacts on my ability to respond quickly to my user's demands. And my dynamic data-centre becomes less dynamic than I want.

2) How does the automated tiering impact replication?

In traditional replication technologies; one tries to keep the layout of the local and the remote array in sync. This is challenging enough in a non-dynamic environment and is often a manual task. How do you keep the array layout in sync if the array constantly changing it's layout to reflect the load on it? How can you be sure that your remote recovery array is in an optimal state? Do you want to? I suppose the answer is very much dependant on my first question? How long does it take to reach optimal state? Is it hours? Is it days? What is the delta between optimal and non-optimal performance?

If you have to keep the arrays completely in sync; what is the impact of sending array layout changes to the remote array? How much additional network bandwidth do we require?

3) What is impact on restores both from tape and from snaps/clones?

The backup application has no idea how the underlying physical structure has been changed; it will just sequentially restore the blocks on the LUNs it can see. However the array has no idea if the blocks being restored are the hot blocks it moved onto SSD or the cold blocks it moved onto SATA. It could be a right dog's dinner and obviously it will see a completely different I/O pattern in a restore scenario; you might not want your array self-tuning itself at this point.

I suspect that the answers to these questions could be quite complex but I am certainly interested in how the various vendors mitigate some of the risk that I highlight. Automated Storage Tiering is still in it's infancy; I think we've a lot of lessons to learn. 

Or do I worry too much? 

Find Your Level

Bas raised a question about vendor lock-in on Twitter and discuss it in his blog; generally if lock-in is inevitable, why moan/fret so much about it. Lock-in is one of those things I do fret about a lot. 

Firstly, I think lock-in is bad in principle; in the ideal world, lock-in would not exist! Everything would interoperate and there would be no migration issues at all. You could take any application and bring it up anywhere, all features would be universal and life would be simple. 

Secondly, I am pragmatic and quite simply know that this is never going to be the case. I am a pragmatic idealist! 

So with those two thoughts in mind; I would like to take the discussion a little further.

Imagine one day you are sitting in your abode; the doorbell rings and a shiny-suited salesman introduces himself. He's working for a new power-company and they can supply electricity at half the cost and it will also make your house smell nice! The only catch is you'll have to replace all your wiring and none of your appliances work any more. You will have to replace everything but it will be cheaper, it will be better! And of course, you'll only ever be able to buy this electricity from this one supplier. If you ever want to change back to the standard electricity, you are going to have to change everything again!

You might think about it but I suspect you'll probably decide to stick with your standard electricity. You have chosen your lowest common denominator.

We make these kind of decisions in our personal lives all the time; now as an idealist, I really don't like the walled garden approach that Apple have taken with the iPhone and iPad but as geeky pragmatist, I've decided to compromise. 

The question is where does that point come in IT? It will vary from deployment to deployment. 

And there have been various attempts to build standard environments; look at the promise of Java and the Java Virtual Machine. Write Once, Run Anywhere but that rapidly turned into Write Once, Debug Everywhere; Java implementations differed slightly, different features exploited the underlying hardware. People's lowest common denominator was not quite where Sun expected it to be. 

It is also entirely possible to write standard 'C' which will compile and run anywhere but you might find yourself limited as to what you can actually do. 

It would also be possible to treat a storage array as a dumb JBOD and utilise none of the features that it provides. 

All of these things would enable a great deal of portability between systems of all types but you would probably find that you weren't getting a lot useful done. 

So perhaps there might be a better way, perhaps a hypervisor to provide abstraction between the underlying hardware and the operating system? The hypervisor could ask the underlying hardware as to what features that it can provide and then provide access to those features in a transparent method, it might even implement the missing features in software. You have removed the lock-in from the hardware but you are now potentially locked into the hypervisor upwards.

Standards might protect you, there's one for everyone.

Only you can decide where your lowest common denominator is; you might not have one and everything will be best of breed and a custom build. You might insist all applications have to run on top of a standard stack. You might insist that everything should be able to seamlessly run in the public and the private cloud. It could be that you decide that you must have company 'X''s unique feature which no other vendor has and is ever likely to have. 

All of these decisions will impact how locked-in you feel and more importantly how hard it is to get out the lock-in. As I say, in the ideal world, migration would be seamless and universal. 

But that'd be really boring wouldn't it!?

You decide how exciting you want your life to be!  

Taking the Proverbial

What is the point of an 'IT Project'? 

Is it to deliver infrastructure?

Is it to deliver applications?

Or is it to deliver a service to the business? 

Actually, most 'IT projects' shouldn't be called 'IT projects'; perhaps they should simply be called 'Service Delivery Projects'. Let's forget about the delivery of applications and infrastructure; we should simply focus on the delivery of service to the Business in a sustainable, cost-effective model as quickly as possible. Then let's look at how we deliver such services. 

This requires IT to work as single department not as warring factions under the titular head of the CIO; without applications to run, infrastructure has no value and without infrastructure to run on, applications are useless.

Some services should be simply be categorised as 'Oxygen'; these are the services that almost every Business needs to survive; email, desktop services, collaboration tools, backup, archiving; all of these can be delivered in an off-the-shelf manner. These are the services which may most easily lend themselves to an outsourced or cloud delivery model. 

Some services are those which require a certain amount of development, probably customisation of off-the-shelf applications; in many cases these are such things as CRM, Workflow management tools, web services etc. Every company probably needs them but every company will use them in different ways and have different requirements. The delivery model of such services is often the most contentious; in-house, out-sourced, cloud? All of these have potential.

And then there are those services which make your Business special; you might have a lot of these, you potentially have none. You might not actually need IT to produce your core product, an Artisan Baker probably doesn't need a lot of specialised IT to bake bread; whereas a Satellite broadcaster needs a lot of specialised IT kit to put television programmes out. These could be bespoke applications required to deliver your service which for whatever reason need a non-standard stack. These are the services which you will most likely decide to keep in house; these are your core.

Then you need to review these services and what has the most impact of the speed of delivery? Is it the provision of IT infrastructure? Is it the development of applications to deliver the service? Is it the process of building business models to support the decision to progress with a project? Does it take longer to develop the ROI model for a service than to deliver the service? Is it the procurement process? What is actually the biggest cost in delivering the new service? 

I'm not actually convinced that the delivery of IT Infrastructure is either the biggest implementation cost for many services or has the most impact on delivery timescales. I think often it can be perceived as the largest cost and the biggest impact on delivery timescales. The former is often because we tend to buy IT infrastructure in large chunks, storage has been especially vulnerable to this process.

(I suspect the costs of storage get focussed on so much simply because the CIO and the CEO often have to sign off on these large purchases; buying a server here and server there hides the costs. Utilisation of most storage arrays is certainly better than the utilisation of most non-virtualised servers and often better than that of virtualised servers. This might be key to the reason that many storage vendors are trying to push a 'pay as you use' model. It might well help to hide storage costs) 

And delivery of Infrastructure is often the last thing to be done on a project, often any delay further up the chain means that delivery of Infrastructure is often rushed and gets blamed for all of the other woes of a project. Often in a well-run and defined delivery, the development of application and the delivery of Infrastructure are carried out in parallel and almost without exception, the Infrastructure sits idle waiting for the deployment of the new application.  

But still we focus on speeding up the delivery of infrastructure because actually it is one of easiest things to speed-up. It should be a 'crank the handle' process and we do need to get better at this but I am not sure at the end of the day that speeding up the delivery of infrastructure is going to speed up service delivery dramatically. 

Of course as an Infrastructure guy, I would say that but I have run a development team in the past and very rarely was I held up by the delivery of Infrastructure. And if I was, it was often due to us being unclear as to what we required from the Infrastructure. 

Now what might dramatically change the cost-base of ongoing service is a more dynamic infrastructure which is easier to manage which can scale up and down easily. And building applications which can dynamically request and release resource as required. But will it change the speed of delivery of service? 

The most significant change to the speed of delivery of most services would actually be the five P's! 

Planning Prevents Piss Poor Performance. 

Nothing I have written above actually says that the various block delivery models are without merit but don't expect miracles from them either. Examine the whole service delivery model, not just that of Infrastructure; Infrastructure is a tiny piece of the delivery model, let's not forget this.

Future Imperfect?

The Anarchist posts an account written by one of EMC's engineers imagining the day in the life of a future application administrator, managing storage has now got so simple that we no longer need specialist storage administrators. It's a compelling vision but how likely is it to come to pass? 

That's a hard question to answer, EMC certainly believe that they are going to announce technology that will enable this and all indications are that they are going to be setting out more details at EMC World. It'll certainly be interesting to see how ready the technology is and how near to ship it is, let's hope that they can do better than they have with FAST2.

But as I keep saying this is not about technology, it's about people and process. 

Firstly, EMC is not the storage world; there are plenty of companies out there who are not running EMC storage or at least are running EMC storage and something else. If  this is an EMC only tool with a huge cost to migrate into; there will be resistance and it's not only a case of migrating into; there is the cost of getting out of again. Technology refreshes are not just upgrades to the latest from the incumbent vendor. 

I suspect that somewhere along the line, there will be some interesting changes in maintenance costs; you will be incentivised to upgrade to the latest greatest; I mean it is easy, why wouldn't you and if the maintenance costs for year's 4+ are high; well, the TCO 'savings' would stack up on paper.

Anyway, let's just say that you look at the figures and you decide that actually this all stacks up for you over the next 5-10 years. So off you set on this journey. 

Firstly, you've got define a service catalogue; now this isn't quite as easy as you might think; yes, I can define a service but what does that actually mean in Business terms? What does 5000 IOs per sec, 5 ms response time, 500 Megabytes per second really mean to a Business User? Do you review all your current applications and define service tiers around those? RTO and RPO; we talk to the Business about those already and still we struggle to get meaningful answers. 

Nothing spoken about in Anarchist's blog entry really addresses these problems. 

Secondly, how does this new dynamic world fit into the current processes? Now, we can sit and moan about whether the current processes are fit for purpose; whether the current change, configuration, problem and incident management processes really work. But many of these processes have been in place for years and actually do a pretty good job with today's technology. 

But we aren't talking about today's technology? Sorry, this is not a black and white situation; you will not suddenly stop using today's technology and if there is one thing which will cause chaos, that is having two different processes in place (unless you try to run as two different IT organisations). This is a non-trivial problem. 

I also love the comments about chargeback. Chargeback in nearly every organisation I've for/with has been one of those things which seems like a good idea until the Finance Department get involved. We get into complex areas about Capital versus Operational Expenditure; what does the chargeback cover etc, etc.

EMC's vision is marvellous and compelling, the technology is the least of the problems but I will add something on that. I seem to recall that we were supposed to have no application developers by now; that code-generation was going to get so simple that a business user would be able to define their applications based on business rules and the business application would be generated. I suspect that there are more application developers now than there have ever been.

Pancakes!

So the IT industry is starting to look like IHOP with stacks everywhere from Hitachi to Oracle, everyone is at it. It seems that infrastructure stacks are the new thing and everyone has to have one. And if you don't have one, you'd better announce one just in case.

So what is going on? Is this an admission that the heterogeneous infrastructure has failed? Is it an admission that the vendors are incapable of agreeing and implementing standards in a standard way? And what blame if any can be laid at the door of the humble infrastructure techie?

I think that firstly, heterogeneous infrastructure has been altogether too successful at emphasising the differences between infrastructures. Standards are implemented in differing ways and extended to give competitive advantage. Just take Fibre Channel, for what is basically a simple protocol; it's a joke! You put a fibre-channel switch into Interop mode and loose half the features which you paid for. Heck, half the time, they'll try to charge you extra just to put the switch into Interop mode. Yet half the features that you 'loose', you'll probably never miss anyway! I mean who is going to miss 'read-only' zones or LUN zoning on the MDS? I suspect pretty much no-one. So we've ended up in the situation where we've only got two SAN switch vendors with one pretty damaged, it's not a healthy situation really.

And of course the various implementations of standards have lead to compatibility matrices which are now longer than the manuals which come with the kit that they ship with. We've built an infrastructure with such complex dependencies that it is becoming impossible to manage. So now the very vendors who have caused this problem are coming to the rescue with certified solutions, guaranteed interoperability and reduced complexity (and a greater share of your infrastructure budget).

Well, don't expect me to cheer this heroism and I'd want a good look at the small print before I bought into any of these stacks. I want to know how long a stack was going to be supported for, how long it was going to be maintained, guaranteed non-disruptive migrations when I decide to upgrade, guaranteed interoperability between version 1 of a stack and version x of a stack.

I think ultimately the move to vertically integrated stacks is an admission of failure on the infrastructure vendor's part but as infrastructure techies we can take some of the blame; see the shiney tech! Pretty isn't it?

#storagebeers - the future

#storagebeers are a great social event and we even now have discussion of a spin-off in #virtualbeers which is being arranged by the London VMWare community. 

But they are just beers and I was wondering if there is any appetite to make it something more; I'm thinking along the lines of an unconference. 

So is anyone interested? Are there any vendors who would perhaps be interested in hosting? What would people like to see as items for discussion? Is there anyone interested in helping me get this off the ground? 

I've got no idea how to go about getting this off the ground BTW! I'm just floating an idea!

Diaspora

James Gosling's departure from Oracle was I suspect entirely predictable; he was never going to sit easy in Oracle after the freedom that Sun gave him. That it happened so quickly after the event is slightly surprising but probably shows what a cultural mismatch Sun and Oracle are deep down and perhaps this is a sign of things to come. 

Obviously one man's departure does not seal the fate of a company but Gosling was one of the talisman techies of Sun; will his departure herald a diaspora of the technical brains from the old Sun? Actually, I really hope so; I really hope that we will see a raft of new companies spring up from such a diaspora and that we see some new ideas and technologies come about. 

Some might find homes in the other technology giants but here's hoping that just a few strike out by themselves and make new successes. We could do with new cool companies and perhaps VMware won't buy them all!

Actually I was thinking a few days ago, what next? Cloud computing is going to happen in some shape or form; we can argue terminologies, we can argue private vs public but it's going to happen now. So what comes next? Is commoditisation the end of journey? Or perhaps just the start of something different. 

Arguably we've come full circle; the mainframe gave way to the PC which has become the mainframe. The personal computer is no longer personal; we talk about VDI which brings conformity and commonality to the desktop. People will strain against that and the truly personal computer will be that which sits in our pockets and not on our desktop. 

But will these devices develop into something akin to the application platforms that the PC has become? Well I'm not sure anyone would have predicted that the PC would eventually evolve into a close relative of the mainframe; so who knows? And perhaps the diaspora from Sun will herald something new? 

Let's hope so....

Odd World

Life is certainly a bit odd in the storage world at the moment. 

Today IBM announced automated storage sub-LUN tiering in the DS8700; so since EMC announced FAST both 3PAR and IBM have announced and are shipping product before EMC? Although IBM's appears to be chubby chunks; not chunklets more chunkosaurus! 

So from IBM, you can hedge your bets with regards to storage tiering

1) Tiering is rubbish, you only need one tier and one type of disk i.e XIV

2) Tiering is rubbish, but you need more than one tier at the moment and we can speed things up with a big read cache i.e nSeries aka NetApp

3) Automated tiering is the way to go i.e DS8700

So which ever philosophy you subscribe too; you can buy from IBM. And if you want something to bind them all together, you can also get an SVC.

And then we have EMC; storage virtualisation was rubbish, what you want is federation! What you do is you get all your arrays and you federate them behind an appliance. But if you want NAS or Object Storage, you're going to need a different device.

And then our friends at NetApp; that EMC fragmented storage stuff, that's rubbish! What you want is unified storage platform! Manage all your protocols from a single operating system. What's that? Object Storage? You want Object Storage? Really, ummm...okay! Here's a company we've just bought; integrated into our operating system? Ummm, no...not really since you ask. But I'm sure it will be, someday. 

HP? Well we can make it all...we have all our own technology, we've bought just about everyone and it'll all be integrated someday. Just don't ask us who makes XP and SVSP. Please!

Knock, knock....Hu's there? Is there anyone there?  Are you sure? I'm sure HDS now stands from Historically Did Storage. 

It's like watching monkeys bashing away at typewriters at the moment! I'm sure someone will save us but I'm not sure who at the moment. 

This Changes Everything...honest!

Val tweeted this and I kind of took exception to it as no where in the linked marketing fluff does it actually say that NetApp enabled Guitar Hero to beat Rockband to market. But it lead me to thinking about something; can a change in technology really lead to this kind of advantage? 

I don't think that the answer is all that clear cut, a change in technology can be the catalyst which enables a change in process; this can lead to an advantage. But is that change in technology actually necessary to enable to the change in process? And does the investment in new technology really pay off and how do you know? 

Marketing fluff is not a good place to start in evaluating whether new technology will actually pay off for you. 

No CIO is going to get up and say to his boss, 'You know that new technology that we bought, well it's a piece of crap! I was totally lead astray by the shiny-shiny lights and boy do I have egg on my face now!'

Equally no CIO is going to get up and say to his boss, 'That new technology that we bought well it's excellent and it has completely changed the way we work! But you know what, we didn't need to change the technology really, we could have just changed the way we work!'

So what happens is a jolly little cosey-up; a nice deal is cut, a bit of marketing fluff is sometimes agreed as exchange for a really great deal on price and everyone's happy. 

Cynical? Yes very! Does it happen? Sure it does, all the time. Whenever I see a press-release or a case-study, I always ask myself what deal was done to enable this; free services, an extra couple of points of discount? 

Sometimes, I think that we fail to see the wood for the trees in IT. If we want to institute major change; we change everything, whether we need to or not. How often do we see the tag-line 'This Changes Everything!'? How often does the question get asked whether everything needs to be changed as opposed to just changing something? As technologists, we look to change technology first and process second.  

All the changes that Chuck et all are talking about, evangelising about are completely worthless without process change but many of the benefits can be felt with a process change. You do not need a technology change to achieve many of the benefits or if you do; it does not need to be a huge technology change.

Actually if you try to do this without a huge technology change; you may reap more benefits. The problem with large technology changes is that you often fail to do anything with the legacy tail; you simply deploy new stuff onto the new technology. The legacy just sits and moulders, costing money to maintain and manage. Of course you could simply just decide not to maintain the legacy, which will save you money in the short-term but when it breaks, you could find yourself spend even more money trying to fix a system which no-one really understands. If you try to change the process before rolling in the new technology, you will probably stand a greater chance of dealing with the legacy tail.

So don't expect miracles from a change in technology! 

Without process change, technology changes are probably just cost!

People Trump Processes Trump Technology

Don't simply accept a vendor's claim that the only way to change your processes is to change to their technology!

When caught up in the new, don't forget the stuff your business runs on today!

Marketing folks even lie in their sleep, their dreams are the lies they plan to tell you tomorrow!

Buy Bycast

I was fascinated and indeed slightly surprised at yesterday's announcement that NetApp had bought Bycast, an object storage company. Why so?

Well I distinctly remember Val saying just after the Atmos launch that NetApp had an object storage strategy and something would be coming soon; I'm sure someone can probably find the exact comment. So if NetApp were already working on object storage, why did they purchase Bycast? Now either Val was being economical with the truth and when he said that NetApp were working on object storage, he meant that they were looking at buying something? Or perhaps, NetApp have struggled internally to develop something; certainly there are rumours that the focus was on getting OnTap 8 out the door and all development resource was really focused on that.

Now this purchase leaves NetApp with something of a problem as to what to actually do. 

1) Do they integrate Bycast into OnTap? I suspect that this would be the preferred route and would allow them to keep the 'Unified Storage' tag but this route is fraught with difficulty and the Spinnaker experience may well have scared them off.

2) Do they simply continue with Bycast as a separate product? This is a much more pragmatic approach but it leaves them open to criticism that they no longer sell 'Unified Storage'. 

It's an interesting position for NetApp to find themselves in.

Object Storage itself is an interesting technology area and one which itself brings a number of challenges. Actually implementing an object store is not especially hard but interfacing the object store with a multitude of different applications is going to be the biggest challenge. Meta-data continues to be challenging with many custom schema being deployed for different uses.

It's still very early days for large-scale deployments of object storage; I suspect that this market has some way to go before any kind of front runner/leader can be declared. I think we are still stumbling over the early fences at the moment.  

It will also be interesting to see how NetApp position the product; it could feasibly be positioned as a direct competitor with OnTap 8. Could they fall into some of internal politics that EMC's Atmos has suffered from with sales-guys worrying that it could cannibalise their traditional revenue streams? 

Or do they use it as a way to compete with EMC's DaaD strategy? 

Do they try to build an application infrastructure around it? 

Interesting times....