Storagebod's Blog

100% Virtualised? Let's try for 99%

A lot of posts and talks from people involved in VMware and especially when we start talking about the Private Cloud talk about 100% virtualised data centres. And there's always the nay-sayers like me who point out that there are niche applications which currently can't be virtualised. These include applications which run specialist hardware and applications which have real-time requirements; in my world of Broadcast Media, these are often one and the same.

But there a whole bunch of other applications; often niche and often from small vendors which can't be virtualised for no other reason than the fact that the vendor says they can't. And the reason? It's not been tested, often the applications have very restrictive hardware requirements which are basically dictated by the vendor's ability to test against multiple hardware variants and VMware (and other virtualisation technologies) is really just another hardware variant. I have a whole bunch of these where people swear blind that they can't be virtualised, I don't believe them.

So I'm going to have a go; fortunately, as well starting to build a new storage team, I have another job which involves running a test and integration department. Hence I have all the test cases etc for alot of these apps already built, so it should be just a case of opportunistically running these tests against a non-virtualised and a virtualised enviroment and seeing the differences. It's going to be a case of fitting it in when we can but we've managed to scrounge some fairly meaty hardware to build our new virtual environment on.

I still don't think you can virtualise everything; especially in an environment which has specialist requirements; in the same way it would be very hard for some environments to get rid of their mainframes, it will be hard for some environments to get rid of all the non-virtualised stuff and replacing all your non-x86 with x86 hardware. But with some work, we might be able to get rid of more than we can today.

Do you have an Umbrella?

Every now and then, we have 'interesting' conversations at work about morbid subjects; recently we had a discussion about another 9/11 and whether our systems could cope with it. Not the systems that you get to see but our internal systems and how we scale for peak load and what peak load actually means. Scaling web-servers is pretty easy but other systems don't necessarily scale so easily.

We had recently put a new system in and then Michael Jackson went and died; so that was a good test for the system, which coped pretty well. Tragic events are what generates traffic; that and England winning the World Cup but we have generally have to deal with tragedies not miracles.

But web-sites, phone-systems, networks can all fail when under extreme load; in fact, pretty much every form of utility is massively over-subscribed pretty much by design. If everyone got in their cars at the same time, the road network would melt-down. Flights are regularly over-sold, certainly by the budget providers. If everyone in the UK decides to make a cup of tea at the same time, the power-grid suffers.

This led me on to thinking about the Public Cloud; what level of over-subscription do the Public Cloud providers sustain in the forms of 'reserved instances' and how many instances can they actually support all coming up at one time. How much storage does Amazon actually have?

If you are going to bet your business on the Cloud, you probably ought to know and you probably ought to know what kind of events will cause you to burst and probably what will cause everyone else sharing the Cloud with you to burst. And what you are going to do if the Cloud does burst? Do you know where your umbrella is? Or at least the candles to dry you out.

Many Solutions, Many Problems...

Instead of focusing on product, more and more of us are concentrating on solutions; products don't solve business problems, solutions do; I like this entry on Eigenmagic which looks at the differences between products and solutions.

But concentrating on solutions can be dangerous because next thing you know you have a data centre full of point solutions and a support nightmare. Many vendors like to try and package their solutions as black-box type solutions which require their own OS builds, own hardware, do not support virtualisation, do not co-exist well with others. 

Software appliances which run on industry-standard hardware may well be the answer but even these often have stringent requirements. I have come across software appliances which specify that they only support certain types of server underneath; we need to move away from this and to very high-level requirements i.e how much memory, how much CPU and how I/O is required to support the application; not it requires a specific HP server. 

Solutions are great as long as they are not used as a way to sell me expensive OEMed hardware which I can buy off the shelf from half the cost. And you may claim that the TCO will be lower; let me tell you, often the TCO goes up because I still have to manage the security patching, multiple support contracts, non-standard form-factors etc, etc.

Blue Sky Thinking

After yesterday's slight rant about the mis-use of the word 'Cloud' and everyone is using it to put in front of their product name and claiming what they sell is 'Cloud'; I thought I'd write a more reasonable piece on Cloud. Cloud Storage to be precise!

We often talk about an explosion in unstructured data and the huge amount of growth that is seen there; well, in my experience within the more traditional corporate IT functions, I don't really see it. Yes, there is growth but most of it is due to the mis-use of corporate file-servers to store personal files such as photos, music and movies. But there are only so many documents and work-related content that a single person can produce.

Obviously, in the media space; there is an explosion of unstructured data, content which would have been stored on traditional media is now moving over to digital media but most people do not work for large media companies.

But every individual in their personal lives are becoming content creators and they also consume an ever increasing amount of content. I talked very briefly about storing your life in this blog entry. It is this phenomena which will really drive the growth of storage going forward and it is this problem that needs to be solved.

Service providers are springing up to try to solve this problem and provide service to the end-user; almost every ISP tries to bundle a small amount of on-line storage for their customers. But to get a reasonable amount of online storage is still relatively expensive; to store the hundreds of gigabytes of content that some of us have at an online provider is still prohibitively expensive in general. Certainly if you want access to it in a form which is seamless to all the applications that you want to use, I couldn't for example move my iTunes library to Mozy and continue to access it via iTunes; I still need to keep my on-site copy.

If you think hundreds of gigabytes is an exageration; just look at the growth in personal HD-recorders such as the Flip and almost every camera. People are just going to upload the content and store it most of the time, it won't take them long to be heading towards a 100 gigabytes.

This storage needs to be cheap and online; I am for the moment ignoring the bandwidth issues that many of us face today; this storage is going to have compete with the cost of DAS today and I am talking about consumer DAS; not enterprise drives.

This is the real problem that what I think of as Cloud Storage needs to solve, not the traditional corporate IT storage issues. Not the IaaS issues which are actually relatively easy to fix; all that needs is cheaper than the current Enterprise storage with an easy to use front-end. That is not a hard problem to fix and that is what annoys me most about some of the announcement and pronouncements that have been made about Cloud recently.

Let's start thinking about how we solve those things which are hard, not sitting in our little comfort-zone supplying the same old storage with go faster stripes. Because if the traditional vendors don't; there are plenty of non-traditional vendors who are not tied by legacy thinking who will.

Some of you have already made a start, some of you are pretending. This is a different game to the one you were playing in. The old game is still there but there's a new one to play as well. And what is Consumer today may become Enterprise tomorrow; this is not just a case of trickle-down, there will be significant cross-fertilisation.

There is a big prize to be won but I'm not sure it is the prize that most of you are playing for.

Flexible Thinking

I think that there is some interesting discussion to be had from Hu's latest blog in response to my own comments and thoughts about whether storage virtualisation as demonstrated by the external storage virtualisation devices has a long-term future.

The response is not really to do with virtualisation at all; it is all about aligning IT and Business. He talks about an all too familiar case where storage decisions are made locally by the Business Units and the procurement strategy does not take account of the long-term health of the group; ongoing OpEx costs are not born by individual business units and become the problem of the IT department. The concept and value of shared infrastructure was not really understood by the Business.

But I suspect that these very same Business Units will be happy to use Cloud-based services, Infrastructure as a Service etc, concepts which are built on multi-tenanted shared external infrastructure. Why? Because they can deploy rapidly and flexibly; they don't need to engage the slow and cumbersome IT department and they feel that they are masters of their own destiny.

Storage virtualisation won't bring these Business Units back but storage virtualisation may enable IT to put together a flexible and responsive service catalogue. CIOs need to engage with their customers and understand what they want but they need the support of vendors to clearly demonstrate cost to their customers and to their boards.

The service must be something that the customer wants, customers do not understand why it takes weeks to provision servers, storage and networks. They can go down to PC World and the likes and pick things off the shelf *NOW*.

The more savvy customer knows that they can enter a credit card number into a cloud provider and can provision dozens of servers much quicker than the IT department can and what's more, they can turn them off again equally quickly and not be stuck with kit that they do not need and do not want.

Customers want dynamic, flexible infrastructures which can rapidly respond to their needs; Corporate IT departments along with their vendors have been pretty poor at providing these. Virtualisation and abstracted infrastructures are enabling technologies but Hu's right in that people and processes are key...

As for external storage virtualisation devices, I still wonder if they are the future? They are not necessary to provide the service and if I was building a green-field data centre with no legacy to deal with, I am not sure I would deploy them.

Infrastructure provision is at an interesting inflection point; if you don't understand this and your customers do, you have got a problem.

Stuff Happens!

Somewhere in the world, there are a bunch of sys-admins, DBAs, application specialists, storage specialists, incident managers, service managers, network specialists and probably a whole bunch of other people running round trying to recover service after a data-centre outage.

How much running around, panic, chaos, shouting and headless chicken mode depends on how much planning, practice and preparedness they have for the event. You might not even notice even if you are using the service because if they have done their work properly, you shouldn't.

Outages happen; big horrible nasty outages happen. In a career which now spans over twenty years, I've been involved with probably half a dozen; from PDUs catching fire due to overload to failed air-conditioning to wrong application of the EPO*. I have been involved in numerous tests; failing over services and whole data-centres on a regular basis and for most of these tests, the end-user would not have been aware anything was happening.

So when Amazon loose a data-centre in their cloud, this should not be news! It will happen, it may be a whole data centre, it may be a partial loss. This not a failure of the Cloud as a concept; it is not even a failure of the public Cloud; there are thousands of companies who host their IT at hosting companies and it's not that different.

What it is a failure of is those companies who are using the Cloud without considering all the normal disciplines. Yes, deploying to the Cloud is quick, easy and often cheap but if you do it without thought, without planning, it will end up as expensive as any traditional IT deployment. Deploying in the Cloud removes much of the grunt-work but it doesn't remove the need for thought!

Shit happens, deal with it and plan for it!

* Emergency Power Off switches should always be protected by a shield and should never be able to be mistaken for a door opening button! But the momentary silence is bliss!

So No DMX5

I'm on leave this week, so I'm not going to post a whole deal on this but I actually thought that this announcement was important enough to sit through the video presentation just to see what EMC were going to publically  announce.

So there it is revealed, there is no new DMX! There you go! I told you that there wouldn't be!

I've known a little about about the V-MAX (that's a cool name, pity it sounds like a console) for nearly a year now; I responded to a post on Chuck's blog with a list of things I wanted from an array; Chuck didn't put the post up and just said that they were working on making my desires a reality.

I really wanted Automated Storage Tiering to simplify storage and it looks like EMC have delivered it. The devil will be in the details though but this is the start of something which will have industry wide implications. Automation on the scale that I believe will eventually happen will change the role of the storage management team for ever.

I've spoken before about the ability to federate arrays; I actually look at what much that has been done both by IBM and HDS as moving towards this. EMC have delivered the ability to federate their arrays; pity it can't do third party but hey...this is EMC we're talking about! Technology refreshes and data-migrations will become a breeze!

Building on top of industry standard hardware, this is just plain common sense. Going down the Intel route, just makes sense as part of cost reduction and common components.

Management tools will need to change as part of all this, I'm expecting a refresh of the management suite at some point. Who knows, we might get the long promised re-write of Control Center, I'm not holding my breath though! I might go as blue as the flashy lights on the front of the new V-MAX.

I don't know a whole lot about the details but I know that there have been arrays out in the field for a bit. I don't have one before you ask! And of course, I can't tell you who else I have had very similar conversations with about what they might be doing. Life is going to get interesting.

I am frankly very impressed that EMC taken such a leap. Congratulations one and all!!

Oh yes, I forgot to ask....how much is this all going to cost us?

And yes, I don't for one moment believe that the V-MAX will not replace the existing DMX4 unless we are going to see a DMX5 in the future? Now EMC have surprised us once but a DMX5...not a chance! Actually, I think questions have to be asked about the future of all of the other EMC disk-lines; just think about it!

Damn, this is a longer post than I intended! Oh well, back to the DIY!

Guaranteed to make you laugh!

I really thought we'd got away from guarantees which were caveated to hell and predicated on using the vendor's services but EMC I mean VMWare are at it this now. Chuck and Barry must be fuming; well if they weren't concentrating on their rare cat cross-breeding program allegedly!

It just goes to show really, that you can't keep a good/bad* ^{(delete as appropriate)} idea down. What has kind of amazed me is that VMWare have appeared to put even more caveats in place than NetApp. Can they actually fail? In two years time, will all of the savings be realised? It'd be interesting to see a long term review of these programs to see if the results are actually real. But I suspect in two years time, we won't care and it won't really matter that much.

And it also shows, don't make too much fun out of your competitors' marketing gimmicks, you just might find yourself doing the same thing. Still, it made me chuckle more than a little bit.

Don't get me wrong, virtualisation and consolidation is a good thing and I am sure these guarantees are a good way to get customers thinking about what they are doing. But at the end of the day, they are still about getting you to spend money and you might be able to realise significant savings without spending money.

Have a look at the guarantees and the caveats; the caveats do a nice job at helping you identifying where you should be concentrating your effort and identifying the low hanging fruit.

p.s I know why Barry has been so quiet but what about the other Barry? What is the Madster Inventor up to?

Amazon - The World's Bookshop and IT Supplier?

How did a online bookseller become potentially the most important IT Supplier in the world?

Were their employees not simply selling books but also devouring them to solve their own internal problems? And without Amazon beginning to scare the beejesus out of the traditional IT suppliers, would we have cloud? 

People talk about UCS being Cisco's reaction to HP stepping over the line in the sand. And this may be true! But it is Amazon who people should be scared about; a company which understands global markets, logistics on a massive scale, flexibility and agility. These are all core to a company which has become synomymous with Online Retail. An IT infrastructure which has proved itself to scale very quickly and allows small start-ups run Enterprise class infrastructure without huge capital outlay.

With the exception of the EC2 cloud, Amazon are offering interesting and thought provoking services which could change the way that applications are developed. But I'm not especially interested in these as an infrastructure bod in the here and now.

The EC2 cloud is more interesting, not because it is especially exciting or innovative; it's not really but it is more applicable to the here and now. And I had initially dismissed it as just a mere so-what hosting exercise and to a limited extent it is but it might well turn out to be as important as VMWare and it is this which I suspect is beginning to frighten people.

Lets for example, take an Enterprise which is running all of its core systems on Solaris systems; with the current travails of Sun, merger rumours and a general push to reduce costs, a decision might be made to replatform the core applications to run on Linux.

But do I really want to install a large Linux infrastructure initially? That'll take time and more importantly money; I might not have enough space in my data-centre, I might not yet have decided on my corporate Linux infrastructure standards. As the head of development, I probably don't want to wait for my Infrastructure teams to get their acts together; I want something *now* and I want it quickly.

Step-up the Amazon EC2 cloud; quickly and easily, without jumping through the hoops that the Infrastructure teams want me to jump through, probably without a huge amount of IT process such as change management, I can have a scalable and pretty robust Linux infrastructure.

Yes, I could have done it with other companies but Amazon is the company which appears to have the vision and strategy to take this forward. External clouds buy me the one thing that I cannot really buy, time.

So I start hosting my development environments in the Cloud and because my internal IT infrastructure teams are too slow, I put them in the external Cloud. Sure for the time being, I'll probably keep my production workloads in-house but in the same way that production work-loads started to move to VMware, how long before I'm experimenting with production workloads?

And that is without Amazon's more interesting services which could really change things. It makes you wonder about some of the other non-traditional IT companies, what have they got hidden away? There's some interesting stuff been done by some of the games companies for instance which may well have wider application.

Clouds are fuzzy things and boundaries will become blurred. May we live in interesting times.

Clouds Ahead

Is our vision clouded or is it cloud? It appears at present that there is very little agreement on what clouds are, what advantages they bring and how to use them. Even individual cloud suppliers seem to have non consistent definitions. Take Amazon for example, one of the pioneers in this field, you can have       

• Amazon Compute Cloud

• Amazon Simple DB

• Amazon Simple Storage Service

• Amazon Cloudfront

• Amazon Simple Queue Service

It is the first service which really stands out in this crowd; this appears to be an attempt to redefine flexible hosting as a cloud service. The ability to build a virtual machine, configure it and then host it in Amazon's 'cloud'. I am currently struggling in seeing how this is a cloud as opposed to a dynamic deployment model or perhaps the cloud is simply a dynamic deployment model for virtualised infrastructure?

We are still going to need people to manage this virtualised infrastructure; people who are going to configure the AMIs, people who are going to secure the AMIs, people who are going to carry out capacity planning etc. And I'm not sure we are going to see any less of them.

So I am wondering, what is this cloud that we are all talking about? Is it going to be like virtualisation, a band-wagon that everyone jumps on? Is it going to be a new way of developing applications? Is it going to be simply Dynamic Infrastructure which is simply an evolution of the current virtualisation initiatives.

It's not really very clear, so perhaps Cloud is a very appropriate term. IT Archtitects love clouds, it's a fuzzy thing where magic happens; not sure what the magic is but it's magic!