Storagebod's Blog

NFS, VMware and Unintended Consequences!

When VMWare allowed NFS to be used a valid datastore for ESX, I wonder if they really knew what they were doing? Until that time, NFS was really a Unix-geek ghetto and not something that many people played with. Yes, you were starting to be able to run Oracle and some serious-workloads over NFS but I would argue quite strongly that it was VMWare which brought NFS as a storage consolidation protocol to the fore.

NetApp were doing pretty well but the introduction of NFS into ESX completely changed their outlook; before NFS and VMWare, NetApp were being forced to support block storage to allow them to gain really serious traction in the data-centre. It was an uphill struggle for NetApp to get their block storage play accepted by the market, a mixture of competitor FUD and internal naivety made this a long process.

But ESX over NFS allowed NetApp to become a serious player in the data-centre and pretty much at the expense of their great rival and VMWare owner, EMC. I'm not saying that NetApp would not be the runaway success they are today without this but it moved VMWare firmly and squarely into NetApp's sweet spot.

NFS is now being pushed as a way to simplify your storage infrastructure; something which will cause much amusement for many long-term sys-admins who have seen some real hashes made with NFS in traditional environments. NFS has always worked best in well-designed environments with simplicity kept in mind unfortunately it's 'ease of abuse' has allowed it to be abused time and time again.

However VMWare environments are architecturally simple and it is unlikely that we will see NFS abused in the same way, so here it makes a huge amount of sense. 

Mounts are very easy manage and it is unlikely that you are going to get into horrible cross-mounting situations. The server-focused security model works very well and you only have a very limited number of users to worry about; in more traditional NFS environments, it was not uncommon to find UID and GUID mismatches; confusion often exasperated by the varying implementations of NIS often running along side NFS.

It is almost as if NFS was designed with VMWare in mind. Yet again, this is amusing because x86 virtualisation seriously damaged Sun's marketplace and was one of the factors leading to the eventual demise of Sun as an independent company. 

So VMWare on NFS has arguably damaged the owners/creators of both NFS and VMWare. Funny how things turn out.

True Names

Names have great power and by naming something, you define it and often bound it in ways that you never mean to do. This is never truer than in certain IT functions and teams by focussing on one part of the function, the primary purpose of that team is lost.

I was talking to one of our Infrastructure Designers and he wanted to know what could be done stop a new 'Backup and Recovery' infrastructure that we are going to implement, falling into the pitfalls that so many previous infrastructures had done. And asking the question in such a way, he at least managed to ask the right question, as he asked about 'Backup and Recovery' as opposed to short-hand which is so commonly used that is 'Backup' with no mention of the primary function of the infrastructure.

By loosing the Recovery bit of the phrase, we unconsciously start to focus on the wrong thing; the purpose of the infrastructure is lost and it simply becomes about storing some files without really thinking about why we are storing them. 

So often I hear that 99% success rate is an acceptable metric for back-ups; sometimes it's higher, may be three nines, may be four nines and often it can be lower. However, if you were ask the average IT manager if it was okay for you to wander around a data centre and turn off 1 in 100 machines? What do you think the reaction would be? Probably not positive? And with a highly virtualised data-centre with 1000s of virtual machines? We could be talking significant numbers. And that assumes that the same back-up is consistently failing; what if it is a random distribution? The impact of 1% of backups failing could be impacting easily 20-30% of your server estate at a rough guess. 

Perhaps it's about time, we stopped talking about Backup and we started talking about Recovery. If you called the 'Backup Team' the 'Recovery Team' and talked about the Recoverability of your estate as opposed to simply the amount of data and the Backup success rate; people might take it more seriously. And with modern systems, it should take minutes to recover a system in general; so what's the excuse for not testing on a regular basis? Even if you were not bringing the recovered system up and into production, a simple smoke-test would probably be an enhancement. 

It's time for the Backup teams to step out of the shadows and into the light; the first thing is to rename that team to reflect their true purpose.    

In Real Life!

'He's only a racist thug when he's drunk!'

'He's fine when you get to know him socially!'

'He's only a bully at work!'

'He's really nice in real life!'

I've heard all of these excuses but the last one is increasingly common and is generally a response to the question 'Is 'x' that much of a git in real life?'

We all have different personae depending on the situation but unacceptable behaviour is unacceptable behaviour where-ever you are; so it's about time that we stopped excusing bad behaviour in Social Media on the grounds that it's not real life.

For nearly thirty years, I've lived a portion of my life online; from 300 baud dial-up to 50Meg broadband, from text-based MUD to World of Warcraft; many of my closest friends have been met online and I suspect that more people know me by my online names as opposed to my 'real name'. 

For me, online is as much real as any other aspect of my life. I suspect it is increasingly true for a lot of you too.

'They started it!'

Disagreement, discourse, debate is all part of real life but do we conduct these in the manner of the playground or as grown-ups? And no-one cares who started it! And I keep wondering when one vendor twitterer is going to ask another one 'When did you stop beating your wife?' 

So let's make an effort to raise the current debates from the gutter and back onto the pavement.

'Standards'

Obviously, we all have different standards as to what is acceptable but if you wouldn't say it face-to-face, perhaps you shouldn't say it at all. Start treating online the same way as you do offline and I suspect you won't go far wrong. Some of you might be obnoxious gits offline but the ones of you that I have had the pleasure of meeting are generally charming, personable and I would happily share an evening chatting tech, life and everything with; it would be nice if your online behaviour matched.

'The Internet Doesn't Forget'

And let's just remember; once it's out there, getting it back is pretty hard! And you have a potential audience of many millions. 

Time to draw your horns in, take a deep breath, go outside and enjoy the summer (or the winter if you are an upside-down person). Or perhaps kill a few monsters?

HORDE FTW!

Expotition to the Cloud

One day, Pooh comes across Christopher Robin putting on his boots and Pooh is very excited for Big Boots mean that Christopher Robin is going on an Adventure. Adventures are great and Pooh tidies himself up and tries to look 'Ready for Anything'!

Christopher Robin announces that they are going on an Expedition, this is a thrilling announcement for Pooh as he's never really been on an Expotition and they sound just thing for a 'Bear of Little Brain' to be doing. Christopher Robin tells Pooh that they are going to find an Expedition to find the North Pole and the North Pole is something that you discover and there is no explanation above that required!

Pooh rounds up all his friends and them to prepare for an Expotition to find the North Pole; the message gets rather garbled and Pooh seems to forget half of it on the way round.  All the animals gather, including some who are not invited but who turn up anyway and off the friends set to find the North Pole!

The trip is fraught with danger involving Ambushes (which are not types of Gorse Bush which fall on you but are suprising all the same); Pooh sitting on thistles and Eeyore eating them. Eventually Rabbit and Christopher Robin have to consult about what the 'North Pole' is; indeed they used to know but now they have forgotten but they decide that it must a pole stuck in the ground.

Meanwhile Roo has decided to have a wash in the stream but unfortunately he falls in and although being a brave little Roo; everyone is worried about what might happen. Fortunately for all, Pooh has found a long pole and manages to use it to rescue Roo! 'Hurrah for Pooh', I say!

And what is more, as if by some miracle; the Pole that Pooh has found has turned out to be the very North Pole that they were looking for!

So what does this have to tell us about the journey to Cloud? What can we learn from the animals and Christopher Robin?

I see Christopher Robin as the CIO or at least a senior level manager in a company; he has heard of this wondrous thing called 'Cloud Computing' and has decided that his company needs to adventure towards it.

Unfortunately Christopher Robin knows very little detail and even worse, he asks his friend Pooh to communicate the message; now despite being a really rather marvellous bear; Pooh is not the world's greatest communicator and the message about what they are trying to achieve is somewhat garbled. In fact, no-one turns up to the adventure with any real clarity about what is to be achieved and what the goal really is! 

As the adventure continues and people start to grumble a bit; Christopher Robin has to turn to Rabbit who appears to be acting in a consultative capacity. It is fair to say that Rabbit like most consultants really doesn't know what he is talking about but he agrees with the client's vague ideas anyway.

Meanwhile, Roo decides to do something by himself; which is like the user who decides they are going to attempt to do something and as is sometimes the case, this leads to deep water. Perhaps like a proof of concept application which grows out of all proportion and is running away; fortunately Pooh is on hand with something he's found to rescue the day and everyone is happy. 

Even better, what he has just found is declared to be what they were all looking for anyway. In our theoretical IT company, I would suggest that they have not found Cloud Computing but a hosting company. But close enough and everyone sets off home feeling happy that they have discovered precisely what they were looking for even if they didn't know what they were looking for anyway. 

 

Bedside Manner?

At times listening to some of my fellow Corporate IT bods, you have to wonder why IT exists? I think it must be something to with the amount of money that gets spent on IT which seems to lead some people into believing that IT has some kind of special value and that businesses exist to fund the IT department. 

Seriously, talking to some people; I really think that people believe this and what's worse, is that our customers have picked up on this. So any spend that IT makes is almost immediately a red-flag issue; it's just the guys in IT wanting to spend money.

But how do we change things? And can we change things?

Firstly, it's about time that we started treated our users with modicum of respect; we're all guilty of the 'users==lusers' thinking at one time or another. We like to think that we are the technological elite but many of our customers have really rather sophisticated needs and often they are much further ahead on the uses of technology than we are. 

Nothing has demonstrated this more than the growth of the use of social media in the business; I know that within the organisation that I work for that IT is probably behind the curve on this, the real thrust to use social media has been from the business. In fact many people in IT still see it as just a toy without realising that is replacing email for many of our users. No-one has asked the IT organisation for permission for setting up Twitter accounts, Facebook pages etc; they just do it. 

Then look at the desktop; how many people work at companies who have massively locked down desktops? And when the locking down was done, who made the decision about what applications would be provided? Who is engaging the users to find out what they want? 

Our users grow ever more sophisticated in their demands and too often, the IT department is seen as the brake. And a very expensive brake. 

IT people seem to see every problem in the terms of technology and often they are framing answers in the terms of the technology that they are aware of. And how they make the best of use of technology to suit *their* processes. Too often they start with a technology answer.  Normal people don't especially care about technology; they care about outcomes and how they work better and improve the business and generally they want to be involved in the changes that impact them. 

I think that IT needs to develop a bedside manner; we do deal with a technical subject and at times it is highly specialised, just like medicine and we could learn a lot from some of the changes that the medical profession is undergoing. Arrogant, aloof and generally obnoxious doctors are rarely tolerated these days; yes, there are some old-school surgeons/consultants about but there are less. 

Doctors, like ourselves, are having to cope with the fact that our parish now has opinions and knowledge about our specialised areas. And yes, at times, this is incredibly frustrating; just talk to any GP and they will tell you about the times when their patient has googled their symptoms and come up with some weird and wonderful disease, convinced themselves that they have days to live and panicked themselves when all they have is a cold. But they will also tell you about the well-informed patient with a chronic condition who knows more about it than them, knows the treatment options and cuts out a huge amount of work. 

The trick is to treat both with the same amount of courtesy and also identifying which is which; working with and listening to the patient is the only way to do this. Just reaching for the treatment cabinet is easy but not always the right thing to do. 

And almost always outcomes are better if the patient is involved and sometimes the outcome the patient wants is not necessarily what the doctor wants. We could learn a lot from this attitude. 

More Cloud Lessons from the Bear

In my previous blog, I looked at what Winnie the Pooh had to teach wannabe Cloud vendors; but the 'Bear of Very Little Brain' has much to teach us all and not just vendors. 

Although Pooh has a wide range of diverse interests, it is probably fair to say that somewhere near the top of list is Food. 

So it is a day very much like today, that we find Pooh walking through the forest humming a little hum to himself when he comes upon a hole. A master of logical and linear reasoning; a hole means Rabbit and Rabbit is Company; and Company means Food! 

So after a rather hearty elevenses, Pooh decides that he better be off (truth be known, Rabbit had run out of Food) and it is at that point he discovers a problem. Unfortunately, the consumption of elevenses has resulted in a unplanned expansion in the girth area and when attempting to leave Rabbit's hole; he becomes stuck. After pushing and pulling by Rabbit, there is a stressful moment where Pooh blames Rabbit for not having a bit enough front door and Rabbit blames Pooh for not knowing when to stop eating.

After more pushing and pulling with all of Pooh's friends getting involved; Christopher Robin comes to the conclusion, that the only solution to the problem is time and allowing Pooh slim down over time. Eventually Pooh is freed to wander the 100 Acre Wood again.

Now as a potential user of the Cloud, there are lessons to be learnt from this:

1. When Pooh went for a walk, he did not intend to stop at Rabbit's for elevenses; it was a very opportunistic snack. Many users of the Cloud will end up using the Cloud in a very similar manner. It seems a great idea at the time.
2. Pooh did not check with Rabbit how much Food he had and eventually consumed it all. How many people know what the limits of the Cloud are? It may appear to be infinite but is it? Will the cupboard be bare at some point.
3. Pooh easily got into Rabbit's home because he was less weighty at that point, when he came to leave, he had expanded some what and had not accounted the limited size of egress. It is easy for a Cloud application to generate a large amount of data and getting that data out of the Cloud may be harder than you think.
4. For Pooh to eventually escape from Rabbit's home was a long tedious process and at times a somewhat fractious process. Migrations can also be long, tedious and fractious; they also stop you doing what you'd rather be doing which in Pooh's case was eating, in your case it might be growing your business.

Many of Pooh's problems could have been prevented by planning and management of girth. Many of the potential pitfalls in the Cloud can be prevented by planning and management of growth.   

Pooh and the Cloud

As most people are aware, Winnie the Pooh is one of the greatest minds who ever lived and through his many adventures in 100 Acre Wood, we can learn many things about many subjects. Over the next few weeks, I shall be examining what he and his friends have to teach us about Cloud Computing. 

I think one of the first lessons that can be learnt from the 'Bear of Very Little Brain' is that calling yourself a Cloud and attempting to disguise yourself as a Cloud doesn't often fool the bees that make the honey. In fact they might indeed sting you and you could come crashing down into a prickly gorse bush. 

In fact, Pooh whilst hanging from the balloon, pretending to be a Cloud and getting stung by suspicious bees comes to the conclusion that the bees are 'the Wrong Sort of Bees' who make the wrong type of honey.

Pooh is acting like many a vendor in the Cloud space

1) Rolling around in mud and pretending to be a Cloud Company! 

2) When customers express suspicion about their cloud credentials; express the belief that these are obviously the wrong sort of customers! 

And although none of them have come crashing to the ground yet, I suspect some of them are looking slightly uncomfortable. Let's hope that they have a softer landing than a prickly gorse bush.

In my next entry, I will examine what lessons that we can learn from Pooh and friends about migration to and from the Cloud.

Infrastructure is Software

Chuck has just written a blog which was very similar to a blog that I was working and I agree with a lot of what he says but I'd take it a lot further and there are some interesting conclusions and potentials along the way which could open the market for interesting innovation going forward.

Chuck talks about storage as being software, go read his blog; there's little I would disagree with there at all; well, until he starts talking about EMC products! However, I would go much further and suggest that we are getting to the stage where all infrastructure at a very real level is becoming software. Although I am not totally enamoured with the Intel-focussed monoculture, it has allowed a common hardware platform and it is flattening the playing field when it comes to hardware differentiation.

In fact, to differentiate your hardware is going to become increasingly expensive and hard, so why bother? Yes, there will be edge cases where hardware differentiation will be a key USP but in 90% of all use-cases; an Intel box assembled from bog-standard off-the-shelf parts will be good enough. 

If we then factor in pervasive virtualisation in the data-centre; we have a platform which has become pretty much standardised and commoditised. I would like to see more 'standardisation' in the virtualisation arena but it's not that bad at present and you really do not have that many choices.

So this has some interesting results; it lowers the cost of entry for new players in the market, if they no longer have to spend time developing a hardware platform and packaging their infrastructure product as a device but simply as a 'soft appliance'; they can get it out there a lot quicker. If they can now rely on the virtualisation layer providing them with a common way of accessing hardware services; they can develop a lot quicker and they can try things much faster. The cost of failure is a lot less; it also allows integration with other types of infrastructure to be tested without a lot of expense; this is a plus for both the developer but also for the interested Infrastructure specialist.

The speed to market is greatly enhanced and they can get it front of idiots like me who will download the appliance and have a play and it's not just storage appliances. There are products like Traffic Squeezer which do WAN Network Traffic Acceleration; there are more open-source router projects than you can shake a stick at. 

I am slowly building a virtual Data Centre out of open source or at least free products; I want to see how far it would be possible to get. But I'm not suggesting that anyone would do this for real today, although I can see some Cloud providers having a really good bash at it. This approach probably would not make sense for most companies as a complete strategy but as part of a strategy, it may well be worth considering. There are large companies out there who invest in start-ups simply to develop stuff for them to use internally; the advent as 'infrastructure as software' without a huge investment in tooling up to build hardware means this a very viable approach for the biggest companies. 

Google do it, Amazon do it but often you hear the comments, 'Well they employ very clever people, so it's easier for them!' Well, don't you employ clever people? Or are you saying that all your employees are second rate.

It took me a long time to come round to the idea that commoditisation and standardisation could drive innovation at all levels; I now believe that it could. It's not just about more and more Web applications; it could drive a new wave of infrastructure innovation as well. 

This leaves some interesting conflicts on the horizon for companies like VMware; perhaps VMware might want to get into infrastructure appliances but that would lead them into direct competition with the Mothership. Infrastructure as software; interesting times.

Here's some links of things worth looking at or playing with, it possibly includes things which are not strictly infrastructure but are interesting anyway. Some are great, some not so great; some show great potential. 

Traffic Squeezer

Openfiler

Samba Ctdb

Vyatta

Amanda

OpenDedup

And of course, there is Ubuntu Server; which will let you build your own cloud for free; there are various ZFS-based storage appliances. You can build your own appliances as well, packaging up and integrating components in the way you want. 

One area where EMC have shown great foresight and that is investing in the vSpecialist team and building a team out of diverse specialities because as infrastructure becomes software; the cross-over between the infrastructure disciplines will become even more mandated. Now the vSpecialist team may be very focused on the 'EMC product set' but if I was an SI or another vendor playing in this space; I would be looking at doing something very similar in the near future. 

It's what I do.....

A few people have asked me about the project I'm working on at the moment; now I'm going to have be careful precisely what I say because I suspect much of the detail is still covered under various confidentiality agreements but I think it is possible to talk about it in very general terms with going into details of technology. 

Firstly, most of people are probably aware that I work for a large broadcaster so what I'm discussing may not be entirely relevant to most of you but you it is possible that you might find some of the concepts trigger some ideas/thoughts in what you do, especially around archiving and infrequently changing files.   

Broadcast is an industry which has undergone and is still undergoing rapid change as the consumption model for media changes; people want to consume media on pretty much any device which has a screen and at of a time of their choosing, this brings many infrastructure challenges but one of the largest ones for many broadcasters is that all of our video still sits on video tape and all of our workflows involve physically moving tapes around. And we have a lot of tapes! 

In the new multi-format world this is not a fantastic place to be and also it means that it isn't as easy as it might be to sweat the asset and of course, we have a massive tail as well which if it were more easily accessible, we would be able to do more with. So the time has come to move to a completely digital workflow or should I say workflows. 

The heart of the system is called Tapeless which is a bit of a misnomer as you will see. The plan is remove all of our video tapes and replace them with digital media but also ensure that the resulting digital archive is searchable, scalable and online i.e all of the media handling is automated, no more humans running round the place with tapes. 

The archive will grow 'forever' which brings some interesting challenges but we have no intention of expiring stuff from the archive; feasibly stuff could be deleted from the archive but in general; once it goes in, it stays in. Technical refresh and migration considerations have been considered pretty much from the get-go.  

So if we look at a life of an asset, it's really quite simple.

The asset is ingested into the system from a variety of sources; from video-tape to a multitude of digital sources.

The asset is stored on high-speed disk, this disk is basically a working cache; most content will only be worked on in the first few days of it's life. Content will be deleted from the cache once various thresholds are reached. High-speed disk for us is highly optimised for sequential workloads; random I/O is not that much of an issue for us but throughput is. 

The asset is simultaneously stored on tape; as I said, tapeless is a misnomer.

The asset is also stored in a browse format on disk; this is the equivalent to thumb-nails of photos. Browse copies are never deleted from disk and this area will also grow forever. This will be viewable from a standard desktop and most rough edits will be able to be done from a standard desktop before and if they are transfered to a craft edit suite for refinement. 

Metadata about the asset is stored in central application from basic information like it's filename to more detailed information about what the asset actually is. This application also manages how the asset moves about the whole digital environment.

It is important to note that assets go straight into the archive and straight onto tape; this copy is never changed, you do not 'edit' this copy. It is possible that an edited copy could also be stored in the archive but we always have a clean copy.

And also all media assets on disk are accessible from any of hosts in the cluster, this is done via a clustered file-system. 

So there you go, it's all rather simple. We have minimal backups to worry about, we have predictable workloads or at least easily calculable workloads i.e we know how much a bandwidth a videostream will use and our only real variable is the users and the workflows they use. This is probably our biggest challenge as the various teams will work in different ways and put different strains on the system; we expect sports content to be handled in a very different way to drama for example. 

People do ask why the archive goes to tape as opposed to a massive disk farm but putting it on disk will not give us a huge amount of benefit for a massive increase in costs and tape is really, rather good at sequential workloads dealing with large files.  

People also ask why a clustered file-system as opposed to NAS; application requirements is the easy answer and we are already moving into unknown territory for some of our application vendors without throwing an additional variable into the mix. 

I suppose one of the things which people might consider in more general environments is consider archiving earlier than later and treating faster disk as a cache tier.  

And I wish I could take credit for designing the system but other people can take the blame for that; my team just have to make the storage bit of it work. 

Clusters, WNPoT and Great Blogs...

I was researching for a blog entry I was going to write about clustering, clustered file-systems and positing whether the future of x86 virtualisation was a Single System Image hypervisor allowing seamless automatic migration of virtual machines between hosts and whether we could see some kind of automated tiering for applications or whether just faking SSI with clever load-balancing/migration technologies might be good enough when I came across Greg Pfister's blog here. 

Greg wrote 'In Search of Clusters' which was pretty much the Bible when I was working in HPC and clustering; I don't know where my copy is any more unfortunately and I note that the publication date was in 1997; so it's probably a little bit out of date. But his blog is a great read and I can recommend it to anyone who is interested in virtualisation and Cloud; it'll fill the gap until he writes another book. 

And he points to another great blog written by Charlie Stross who has come up with the acronym 'WNPoT' which stands for Wonderful New Piece of Technology which I suspect is another way of saying 'Awesome Sauce'. 

p.s Yes, I suspect we will see some attempts at a SSI hypervisor; IBM have a statement of direction for z/VM leading down that route, so I expect some brave soul to try to do the same for x86. But for the time being, I think faking it with some good tools might be good enough.