Storagebod's Blog

iBlock?

What can Apple teach us about Enterprise IT? Apple and Enterprise IT, words which don't really belong in the same sentence but perhaps we can learn quite a lot about the future of Enterprise IT by looking at Apple and its current strategy. 

Firstly, like many geeks I must admit to having a very uneasy relationship with Apple and it's products; I still keep thinking style over substance, overpriced and under performing kit. So why is my laptop of choice a MacBook, why do I own an iPhone and an iPad? Why am I looking forward to June 7th and Steve's keynote where he'll certainly announce a new iPhone? 

Like it or not, Apple's stuff just works; my MacBook boots up in the half the time of my Windows Laptop (actually it's even faster since I put an SSD in it), applications just work; hardware and software work in harmony because they have been designed in concert to work together. I don't measure TCO for my home kit but the time I save with at least one piece of kit which just works is great. It gives me the time to hack about with Linux, ESX and Windows. And of course, hidden under the covers, there beats the heart of the ultimate geek operating system, Unix! 

And then there's the iPhone and the iPad; Apple have taken control-freakery to extremes; even telling you what languages you can develop in and then controlling the method of distribution and if Steve doesn't like it, it isn't coming in. But the app-store is so unbelievably convenient; installation of applications is just a tap away and despite the fact that Steve's control-freakery is simply wrong, I still happily use the devices and ignore that nagging voice in the back of mind.

Sure Apple's stuff is more expensive but it just works; it's a fairly sad indictment that to get stuff that just works, we are willing to pay more but that appears where we are at the moment. Apple have developed the iBlock or various iBlocks; perhaps quietly and subconsciously, various strategists in the Enterprise Industry have been influenced by this seductive idea that things should just work? 

People are getting used to the idea that there's an app for everything and it's simply a tap away to get. Our users are getting used to this on the iPhones and now their iPads; we can expect that they are going to ask why they first can't get the same service for their desktops and eventually for their enterprise servers. And they'll just expect everything to work and work *now*. 

But a word of caution and take this from the voice of experience; Apple's TCO in a heterogeneous environment soars, it is painful to get it to work with anything else. It wants to do everything it's own way and plays very begrudgingly with others. If you need to do something slightly out of the ordinary, you will struggle to do so. 

Apple is great as long as what you do is what Apple wants you to do in the what it wants; which is why it will always struggle in the Enterprise. Let's hope that the various Enterprise stack vendors learn both the positive lessons from Apple but also take account of the downsides.

Finding Space and How You Can Help

For the last twenty years or so, anyone who has worked in an Enterprise IT department have had various traits drummed into them and if the Business has not got the IT department that they want, it is not just the IT department's fault. 

IT Managers have accounting principles banged into them; almost more so than any other department apart from the actual Finance department. In almost no other department apart from the Finance department are the concepts of TCO, ROI etc really understood; we deal with rapidly depreciating capital assets which also attract huge recurring costs. And unless IT is your core business, these are not seen as adding real value to the Business but a cost. 

IT Managers also become pretty expert in the area of contract law, leasing law and many other forms of corporate law; at various times in my career, I have been involved in drafting complex support documents, negotiating penalty clauses, get-out clauses, payment schedules and other minutiae which I had no intention of getting involved in when I decided to do IT. 

Innovation within Enterprise IT often comes second to day-to-day keep the lights on which is all kind of sad because most IT guys I know have a ton of creativity and ideas on how to improve IT. A lot of people get into IT because it is their passion and their hobby.

However, perhaps the vendors can help with all this and perhaps instead of coming up with fairly trite redefinitions of accepted terms and sound-bites; perhaps they can try answering those questions which they can help with?

So next time an Enterprise IT guy asks the question 'What's the TCO Model?', 'Can you help build the ROI Business Case?'; if you answer the question, you might find that you have a much happier and more loyal customer who will say good things about you? Perhaps you will find a customer who actually has time to listen to your latest product pitch? The New IT will still have to follow all the good financial governance that we are supposed to practise today, so there is no pointing pretending that it doesn't. 

Help give your customers the space to innovate and perhaps you can help turn IT departments into the IT partners which businesses need and want now?  

Protecting De-Duped Data

As more and more vendors introduce primary storage de-duplication, we need to consider the impact this has on data protection and recovery.

Primary storage de duplication does not make disk any less reliable but it does magnify the impact that a failure could have and if we align this with the ever larger spindles and also the ongoing trend to stripe data across many disks, along with automated storage tiering, we are increasing the impact of failure. We are putting more eggs into one basket. 

If we consider the impact of the failure of a de-duped virtualised server environment, this could take out a large proportion of a data centre. Restoration of many thousands of images from tape will not be viable. If we have a truly dynamic environment with virtual machines being brought up on demand; we need some way of storing state and recovery of state. 

But 'Bod, how likely is this to happen? RAID ranks, especially RAID-6 protected do not fail often but they will fail. Is it more or less likely for your data-centre to burn down than a RAID-6 protected rank to fail? Well only you can make the risk assessment for your business but if you have built redundant data-centres, you are already catering for a certain amount of unlikely catastrophic failure but what constitutes a catastrophic failure may be changing. 

Replication will have to become standard practice for nearly all environments as will seamless automated storage failover. You may even consider local array replication depending on distances between data-centres and complexity of individual component failover. 

Yet even with this overhead, reduplication will still make a lot of sense for many workloads; just make sure you do proper risk/impact assessments.  

 

Dealing with Legacy

Storagezilla's post on Clariion LUNs demonstrates aptly EMC's and many other vendor's problems with dealing the legacy; when I see gloating posts from the newer vendors, I feel very tempted to save the posts in order to bring them up in ten or fifteen years when they themselves are struggling to cope with an established customer base and legacy arrays.

Dealing with legacy is a big deal for all of the vendors; if you move away from the legacy, you potentially upset your customer base but if you continue to support the legacy, you find yourself building on more and more cruft. Customers with long established process, procedures and scripts are loath to change them. 

And indeed, even the vendor's own staff may well be treating the new features with suspicion. How many times have you sat in a meeting with a vendor's staff who tell you that they wouldn't touch that release of the code yet as it's too new and not 'Enterprise' ready. 

Yet vendors spend a huge amount of money QAing their code; 50% of product development time is often QA. However, we often get code which is still not bug-free and still often has some whopping bugs in it but in many ways, we as customers do add to the problem. We insist on using deprecated commands, not trusting the new features and generally insisting that the vendor supports features from 10-20 years ago. 

All of this adds complexity into the code-base; inefficiencies and mis-understandings creep in, bits of code become fossilised and no-one knows what they do anymore. 

However how does a vendor take its customers on a journey allowing them to transition to a newer, better way of doing things? I'm not sure but certainly their own field-staff need to show confidence in their own products; they need to redouble their efforts to ensure that nothing ships with service impacting bugs. Some serious thought needs to be applied to creating tools which migrate scripts and allow customers to move into new features. But there will always be customer resistance to change.

EMC's problems are fairly horrible for them to deal with; their plethora of product lines does make dealing with this legacy harder. And at present, we just see that product-base expand; even if they just turn all of their products into personality modes which run on a base platform, they will still have an ever expanding mountain of code to deal with.

To be honest, I think that at some point EMC will have to bite the bullet and consolidate their product-sets but which one? There are significant overlaps and the overlaps are getting worse not better; the low-end Clariion overlaps with the new rackmount Iomega and the high-end Clariion overlaps with the low-end Symm. And if we throw VPLEX into the mix? 

EMC are not the only company with this problem but they are the company with the largest problem due to the size of their install base, the maturity of the install base and also the fact that they make all of their own stuff. They have also been very good about ensuring a certain level of compatibility and retaining features for almost no good reason. 

Yes, they do deprecate hardware; I don't really blame them for this, the CX4 for example is significantly more powerful than the CX3. And with VMAX, it is a completely different processor architecture to the DMX.

Companies like IBM and HP, who at first look to have similar issues, do OEM at least some of their product base. And IBM were very ruthless when they moved from ESS to the DS8k range; changing CLIs for example. But they had less customers to piss off really!

But to poke at EMC, for at least trying to support their legacy and doing the right thing by the customer; is a little small. When you've had products in existence for as long as EMC and others and you find the magic bullet for dealing with legacy; then you can poke.

And yes, NetApp et all; your unified storage approach helps in that it reduces the number of code-bases that you have to support but don't pretend that there are not some wierd and wonderful legacy structures that you probably would rather not support. 

Beer, beer....we want more #storagebeer!

Yet another #storagebeers to be held on Tuesday after the SNIA Academy. This is an important one as we hopefully have not one but two special guests from the blogosphere. There will be other bloggers there and I'm not saying that you aren't special, you all are but these two are travelling some way to be with us!

Firstly, we have Alex MacDonald of NetApp; hopefully Alex will be joining us from the frozen North; okay Scotland but it's been so cold recently, I think it counts as the frozen North. 

And secondly, we have another NetApp blogger in the form of Val Bercovici who will be joining from warmer climes! 

So if you want an informal and hopefully wide-ranging discussion from these two members of the Storage Blogosphere; please come along! 

We shall be meeting in the Chesire Cheese which is close to the SNIA venue and also to Tower Hill tube. Please note, not the more famous Ye Olde Chesire Cheese on Fleet Street! 

p.s to make #storagebeers entries easier to find, I have added a #storagebeers category!

New Device or Future Array?

Okay, some more riffing on the subject of VPLEX; just idle speculation and I expect either no comment or complete denial from EMC. 

In VPLEX, do we see the future of EMC storage? Or more accurately, do we see the future of Symmetrix? Is it the beginning of the end for Symmetrix and more importantly, Enginuity. The final break from the past?

It's certainly an interesting thought; firstly despite regular denials from EMC; bushfires regularly break out between the Clariion and the Symmetrix camps. Should all the development efforts be focussed on Flare or Enginuity? Some people might say that at times, there have been more than bushfires. This is to be expected, when development teams are competing for finite resource, this sort of thing happens. But this never really gets resolved, especially when you have two successful products like Symm and Clariion. 

Perhaps what is needed is a break from the past? Could VPLEX be this break? Well certainly from what has been announced so far, probably not. But if we look at the various EMC blogs where it is suggested that they might add things like snaps, clones etc; this seems a possible way forward. 

VPLEX as an array controller may well be more interesting long-term as opposed to this 'new category of storage device'; a device which has a much looser coupling with it's back-end disk than the current range of storage arrays. There are reasons why EMC might want to do this or at least customers might be interested in such a device. 

I posit that array controllers are actually changing quicker than the back-end disk these days; sure, disks are getting bigger but this is not entirely beneficial to especially the Enterprise storage market but if I want to take advantage of the latest greatest features of EMC's latest greatest array; I have to rip out both the array controllers and the back-end disk. 

What if I didn't have to do that any more? What if I could upgrade the array controller completely separately from the back-end disk? What if this was a completely non-disruptive upgrade? What if in order to do a migration, I didn't have to temporarily have twice as much disk on the floor as I do in normal operation? 

Perhaps at that point EMC have actually built a truly modular storage array?  And what if EMC can finally head towards a unified code-base for block storage? Yes, they might have to live with three code-bases for a period of time but it might actually be a worthwhile investment for them or perhaps they are happy to continue with a multitude of code-bases.

Just thinking what I might do in their situation...

V(per)PLEXed?

So we have VPLEX and despite some scratching of heads as to what it is; it is really quite simple, 

'storage access is further decoupled from storage physicality'

And this really is nothing especially new; decoupling the storage access from storage physicality has been going on for some time. Servers are getting further and further away from their physical disk. We have been adding abstraction layers to storage access for some time, the big question is whether we need another abstraction layer? 

Actually I think that the additional layer is useful; the ability to present 'real LUNS' from 'storage arrays' as a single 'plexed LUN' and keeping these LUNs in-sync might actually be useful in a number of use-cases. I can see it simplifying and transforming DR for example; I can see it making migration a lot easier and I can see EMC heavily leveraging their VMWare investments. I've said it before and I'll say it again; ever since EMC spun VMWare off, they have acted more in concert than when VMWare were wholly owned. 

Is it useful enough to warrant EMC's claim to have invented a new class of storage device?

I think I'll let the vendor bloggers rip themselves to shreds over that.  

I also think it is interesting that they have at long last decided to pretty whole-heartedly support third party arrays; if anything, this makes it an interesting announcement for EMC. Will they sell any? Well, it's going to be an uncomfortable experience for your run-of-the-mill account manager when faced with a Storage Manager who says 'Well you've just re-invented SVC/USP-V etc...you told me that they were rubbish, so why is yours any good?'

I think the heavy-hitters in EMC are going to be very busy supporting their account-teams.

Why I Don't Ignore NAS?

Stephen's blog entry about NAS strikes so many chords with me that I find it hard to disagree with very much he writes in the entry but I am going to disagree with the central premise/question; you should not ignore NAS if you work in a Storage Team. You may hate it but you should not ignore it.

If you ignore NAS as the storage team, you will find your users going very much their own way and you will find that you have a half a dozen little NAS deployments deployed across your corporate network. This causes headaches for everyone; security, network teams, back-up teams all suffer when this happens. And when it all goes wrong, who is going to get called in to rescue the situation? Well, that would be the storage team and who will still get the blame, that will be the storage team.  

Stephen talks about primary data-centre applications but what does this mean? I think that we all have our own prejudices about what constitutes a primary data-centre application but I am going to argue that this might not be a useful definition any more. If the data has value, it needs to be in a secure, supported environment. 

However, it is worth looking at the problem that NAS is often used to solve; NAS is often seen as the simpler, more flexible and agile solution. In smaller SME environments where you do not have a dedicated team looking after storage and you only have a few systems; this can very much be the case. But as soon as your environment grows and increases in complexity, often interoperability issues start to raise their ugly heads. 

Mostly this tends to be in the CIFS/SMB area in a mixed-environment. If there is one thing that you can currently do to avoid unwarranted complexity in your environment, that is do not deploy SMB in a mixed-environment. And even if some vendors allow you to do so, do not deploy NAS shares using SMB and NFS to share out the same volume; the security models will eventually cause you to cry.

A pure NFS environment generally works well; a pure Microsoft SMB environment generally works well. But they need looking after and the storage team needs to work closely with the server teams; the storage team needs to understand more that pure storage, they need to understand the operating systems that they are talking to. I think this is why often storage teams try to avoid NAS, they really want to focus on spinning rust not what the users want to use it for. 

For the storage team, NAS is not simpler; it is more complex. So is it more flexible and agile?

Flexibility can probably summed up as shared-access to common areas. This is harder to achieve with traditional block storage and can bring you into the world of clustered file systems; these have traditionally been fairly complex and arcane to manage and set-up. They are getting better, GPFS is the one that I am most familiar with and it has come on massively in the terms of useability over the past decade. The only problem with most of the clustered file systems is that they are not available for all operating systems but with the growth of Linux and Windows in the data centre, this is becoming less of an issue.

Agility, it is probably quicker to throw up a NAS share than to present traditional block storage. It can use the same infrastructure as your already in place IP network. You do not need a traditional SAN and you do not need to put in extra cards in your servers. You can just put a NAS device on your network and start sharing. iSCSI, however could also be deployed in the same way, just as quickly and arguable more simply as you do not need to worry about integrating networked file-systems into your existing security domains.

Obviously this is rather fraught with danger but many NAS deployments start this way; this agility leads to a long term fragility in the NAS deployment. A NAS deployment should actually be designed with as much care and attention as a traditional SAN.

It is also possibly fair to say that the NAS vendors have paid more attention to their management UIs than the traditional block storage vendors; they have worked to make it simple to configure and administrate. However, the block vendors are now catching up in the simplicity stakes. 

So if block storage can be less complex, as flexible and as agile as NAS deployments; perhaps we can ignore NAS? But if we had ignored the challenges that NAS had brought to the traditional block model then we probably would not have seen the improvements which have reduced complexity, increased flexibility and agility in the world of block storage. 

And with convergence at the network layer coming in the form of DCB; neither NAS or SAN storage can sit on their laurels. I wouldn't ignore either as technologies in my data centre but I wouldn't ignore RESTful storage or clustered file-systems either. 

!EMCWorld #storagebeers reminder

I am fairly sure that someone will be running a #storagebeers event at EMCWorld.

This is a reminder to those people who aren't attending and who are in and around London. There will be a !EMCWorld #storagebeers event on May 13th at the Counting House nr Bank Station in the City of London. Event should start about 18:00.

Please come along, I am hoping that we can have some discussion about future #storagebeers events as well discussing the inevitable EMC announcements.

Also, I want to remind everyone that there will be a special #storagebeers event on May 18th after the SNIA Europe Academy event, also in London. This time we should have a special guest in the form of Val Bercovici, NetApp's Cloud Czar. Venue TBA later on this week.

Policy Matters

It's kind of appropriate to be blogging about matters of policy the day before a general election in the UK. However, the inspiration for this blog is not the UK general election but some comments made about automated optimisation and predictable performance. In my previous blog here, I think the comment from Hector about automated movement being not desirable in conservative environments is only half right. 

Firstly we need to examine the current situation and for this I am going to make one assumption; your environment is not so conservative that you dedicate an array per application; if you are that paranoid about predictable performance, you probably do not want to be running shared infrastructure of any sort. So taking away that edge-case, we should consider what is now normal in large environments and that is shared infrastructure; certainly shared network and storage infrastructure is pretty much the rule. 

Once we accept that fact; we no longer really have a predictable environment, complex interactions between applications can cause unpredictable outcomes. You push down here and something pops up there. The more applications we add, the more complex and chaotic the system becomes. So what do we do?

Traditionally, we have relied on human intervention to manually optimise the environment; endless poring over spreadsheets and manual moves. Storage performance tuning is a black art but with storage infrastructures now measuring in many thousands of spindles and application instances heading that way too; even the most efficient and effective storage expert can only struggle.  

The first tranche of storage optimisation tools were never really trusted and were generally only run in an advisory mode; still requiring a human to make the decision. But with infrastructures so complicated, can a human really out-think a machine? I am no longer convinced. 

However, I am also not convinced that telling an array to just 'optimise' itself will work either. An array has no idea about the priorities and the value of the various applications on it, what it believes is balanced may well not meet the service levels required. Some applications will be more tolerant of latency than others, some might need more throughput that IOPs etc.

What is needed is a policy engine; you set the policies for the applications as agreed by the business and the array does it's best to meet them. Defining these policies will become the new black art; it will require a deeper understanding of many aspects of the Business and the end-users than simply making the array go fast.

Triage policies will need to be defined for exception handling i.e. in the event of the array not being able to meet the policies defined does it

    a) degrade all applications equally?

    b) degrade some applications significantly and leave others unaffected?

And these policies may well be time-based; at two o'clock in the morning, your performance requirements may well be significantly different to that at nine o'clock when everyone is trying to log-on. 

Policies can be event-driven; a file-based restore/backup will have a very different I/O profile to normal processing, so perhaps you want to turn off any kind of optimisation for that application when that is happening? Perhaps you might even have a special policy to cater for restores? An application upgrade, a data refresh, a data fix; all will have different impacts on the array. 

In fact, the whole area of policy-based storage management will not devalue or even de-skill the storage performance expert; it will allow them to focus on what really matters and respond more quickly to business demands. 

Policy-based storage management does not need to be any less conservative than manual optimisation; you set the policies as conservatively or liberally as you like. Your policy might just be let the array do its own thing. Or it might be to legislate for every eventuality. But in the background, the array will be doing it's best to comply and at least it'll tell you when it's in breach of the policies set down!

Pity our politicians rarely do the same!