Storagebod's Blog

Bod's Annoyances: Server Virtualisation

Server Virtualisation? Is 'Bod simply being obtuse; yes it is entirely possible but before you write this off as lunacy, I do have my reasons. 

Firstly, server virtualisation is not actually a bad thing; it's been around for many years but the landscape today is actually more than a little concerning and I wonder if it will come to bite us really rather nastily over the next decade. The sheer number of virtual servers being built leads me to wonder about the long term sustainability of the current infrastructures and with the current virtualistion==cloud lazy thinking; I am really rather worried.

So how did we come to such a position? The seeds for this were sown in the early eighties when IBM launched the personal computer; prior to this (and yes I am ignoring the early adventures in personal computing by Apple, Commodore, Radio-shack and the likes), computers were large, complex and extremely expensive beasts managed by the computing department; access was very restricted and not many people had access. But with the personal computing revolution; our relationship with computers changed, indeed we actually started to develop a relationship with computers. 

One person mapped to one PC; that's the very definition of personal computing but as these devices became more powerful, it became pretty obvious that they were capable of so much more but the big, multi-user, multi-tasking operating systems which ran on the large corporate mainframes were not available for these personal computers. So we ended up running larger and more complex applications on these devices but using the operating systems available on the desktop; these were single-user operating systems with limited multi-tasking capabilities. So an application per device paradigm became the norm and even when multi-tasking became available and more common; the paradigm had become firmly embedded. 

And the lack of multi-user capablities really hampered the development of a centralised management and secure environment; so it became even more embedded. Add into that, a growth in CPU power far outstripping our ability to use it for a single application; we ended up in the position that we had rampant physical growth but with ridiculously low utilisation figures.

So in order to use this power and to manage it effectively; we have looked to server virtualisation to fix this problem allowing us to build virtual servers which more effectively use the available CPU or more to the point allow us to run multiple single application servers on a single physical server. It also allows concurrent management of these applications to be more easily achieved. Even the rise of the Intel-based Unix systems has not done little to break the paradigm of one application per server but instead of a physical server, we now talk about virtual servers.

It doesn't have to be this way and we only have to look at the way that virtualisation is used in the mainframe environment; you don't talk about running 1000s of server instances, you are more clever about how you virtualise. You may virtualise an environment but it will still only run a few actual virtual machine instances.You may group related applications into a single machine instance but not have those applications running as separate server-instances.

We could do this in the Intel-space; we should move away from the default position of one application per server instance, we should move to the position where we understand why we do so. There are applications which certainly work better when run in massively distributed and parallel environments but this is not every application. A database server can run more than a single database instance; a web-server can run more than a single web-server.

Okay, it's not server virtualisation itself which annoys me but the unthinking use of it and the traps that we are busy building for ourselves.

Turning desktop operating systems into powerful server systems probably was not the wisest move that the IT industry made but the short-term gains were probably too attractive, can we hope that we don't do the same thing with mobile device operating system? Actually, I am not that hopeful really; history does unfortunately have a habit of repeating itself.

Supporting Role?

If you work for a vendor and after reading this you feel that this entry is about your company, then you are right! And if you work for a vendor and after reading this you feel that this entry has nothing to with your company, then you are totally wrong!

I've been working in IT now for more than twenty years; I've done a variety of roles but a big chunk of my career has been running support teams and I've done this both sides of the fence i.e both at an end-user but also at a reseller. I know a lot of people who have worked in support on both sides of the fence and pretty much all of us have coming to the same conclusion; over the past ten years or so, vendor support has got markedly worse! Of course, it may be that we are looking at the past through rose-tinted glasses but I do not believe so. I would also say that this is pretty much across the board; this is not a storage problem but it impacts hardware and software support across the board.

There are a variety of reasons for this decline in quality and ironically, it is some of the services that we as end-users have demanded that may have driven the decline in service. 

The Internet is a big driver in the decline of quality; there are huge amounts of support information and detail on line and we as end-users have demanded that the vendor support databases are put on line; unfortunately, this information is often presented in a manner which it is pretty much impossible to effectively search and you find yourself trawling through screens of unrelated sludge before you find the answer you are looking for. But because all of this information has been provided in a 'self-service' portal, this appears to have been used as an excuse to reduce the number of qualified people that you have in a support centre. 

The Internet has also further enabled the stock holding questions from the support centre and having worked both sides, I know that these are holding questions in general.

1) Are you at the latest level? 

2) Can you send me the logs?

3) Can you send me the configuration files?

Now all of these are valid questions in many circumstances but too often it feels that you are dealing with a robot who is working off a script! This is a Support Centre not a Call Centre; there is a difference! And my (and many other's) responses to the questions are

1) Why? Will it fix my problem? Where in the release notes does it talk about something even vaguely related to my problem? And if I'm in a really sceptical mood, can you send me the piece of the code which fixes the problem. And you realise that this is a live production system and just applying fixes is unacceptable, I will get asked all the questions I've just asked you by the change board. And if it makes it worse, it's my job on the line.

2) Why? Will that log help you diagnose the problem because I've already looked at the log and I can't see anything. Do you have any idea how frustrating it is when you have got a disk off-line and marked failed to be asked for logs from the array?

3) Which files do you want to see and why? 

And often, I've already attached the information to the ticket already; so you asking for them again shows that you have not read the ticket or are just working off the script.

Constantly in conversations with our various hardware engineers from all vendors, we discover reductions in headcount, experience engineers being retired early etc and territory coverage per engineer being increased. We are not buying less hardware, we are buying more and there are more systems out there to go wrong. Now arguably, systems are becoming more reliable but there are more of them. And in the world of storage, we have lots of moving parts; disks spin, tapes spool, robots move and these are all things which wear out and break. Disks get bigger and the potential impact of a disk failure and the resultant rebuild times gets ever larger. 

Talking to people who work in the support centres; it appears to be more important to keep the queue within the targets than solving the customer's problem at point of first contact. There is no longer time to do follow-up calls; for example, calling the customer who had the severity 1 call to ensure that they are happy with the fix. 

This is just the tip of the ice-berg and I could rant on and on about this subject; I was ranting about the decline in service years ago and yet it really is not getting better. For example, I am personally aware of four companies this year who have experience major outages due to problems caused by vendor support; it may be that now I have a fairly high profile in the industry that people tell me their war stories but it seems we are on an upward trend. 

I think it's about time that the vendors started to review what and how they are providing support (fix your websites or at least put a decent search capability on it, it is pretty awful that generally I find myself using generic search engines and the site: directive to search your site); it is also about time that they started treating support centre staff with respect and giving them time to do a great job. 

Apple Abandons Enterprise?

So with the announcement of the withdrawal from sale of the 'Xserve'; have Apple finally abandoned the Enterprise? Actually to abandon something, you would have had to have been serious about it in the first place and Apple have never really been serious about the Enterprise and certainly the Data Centre. Pointing people at the Mac Pro or the Mac Mini is not really a serious suggestion is it? 

Apple have never had and I suspect are never likely to have a serious Data Centre presence in the world of Corporate IT but are we going to see Apple attack this from a different angle. We all know that Steve Jobs loves the ability to control the whole stack from top to bottom and I don't think he is yet ready to completely abandon any ambition that Apple have in the Corporate IT space; indeed, I think that he has been heartened by the user-driven success of the iPad and iPhone in the Corporate space and there is a realisation that he cannot compete with the traditional corporate suppliers on their own ground. So, he's going to try to move it to his ground.

The creation of the OSX app-store is simply the first shot across the bows; the ability to manage the applications on your Mac device in the same way as managing them on your iPad, iPhone (or indeed your Android device) will put a huge amount of pressure in Corporate IT suppliers to provide the same desktop experience to their users. I suspect Microsoft are going to come under an increasing amount of pressure from their corporate customers to provide similar functionality to allow application management by end-users. 

But the current Apple app-store does not really provide corporate customers with the level of control that they require and also, it will be completely unmanageable at scale; for this to be a true corporate play, we are going to need corporate app-stores where a corporate end-user can log into and download/install the apps which their employer have licenses for. However, once you get used to consuming applications in such a model; it is simply a better way of managing things. 

And I wonder if this will be Steve's next play for the Enterprise? Or perhaps Apple are truly about to dump the Enterprise and become a consumer-only company. If he doesn't do it; Google will do it as part of Chrome OS, if that ever sees the light of day as a non-beta product.

I suspect even the faithful of the media-world are getting a little nervous with the current behaviours of Apple; for example no FCP refresh until 2012 in a technology area which is changing incredibly quickly with new demands being made all the time is a little worrying for many of them.

2011 is going to be an interesting year for Apple watchers.

Apple's Debt

Apple owes a huge debt to Windows and the PC industry; in fact, without the PC enthusiasts, Apple would have some really interesting problems. People like to point out that Apple has moved over to commodity hardware; which it has mostly but why has that hardware become commodity? It’s because of the PC. 

Steve Jobs makes a big thing about controlling the whole stack, both hardware and the operating system come from Apple; in this way, Apple get to build devices that simply work.  But the world of the PC massively subsidises his hardware development costs; for example, without the PC and especially the PC games players; where would Apple’s graphics cards come from? And there are many more examples in all of Apple’s hardware. 

It would be nice for the fan-bois to acknowledge this debt; there is no doubt that Apple put together some really nice machines, this entry is being typed on a new MacBook Air which is simply the most gorgeous laptop that I’ve ever owned. Is it revolutionary though? Not really, it’s just a really great instantiation of the laptop. And yes, it’s probably overpriced for what it is but then again, you can say that for many things. 

Perhaps including the integrated stacks some of the vendors are pushing…they need to stick Apple logos on them I guess!!

 

Agile IT leads to Agility?

Leadership, management and agility are becoming important watchwords in the role of IT delivery and it is these three concepts which need to drive any IT organisation forward over the next decade or so. It is the third of these that is probably the most important.

Leadership and management can almost be interchangeable when an organisation is in a steady-state and lets be honest, most IT organisations have been in some kind of semi-steady-state for the last decade. IT Delivery became very procedural in a glacial sort of way.

Radical change in many IT organisations is simply shuffling the cards around without adding anything else in; perhaps moving what already is being done to a third party. In fact, arguably the trend to outsource can be pretty much attributed to the steady-state of the organisation. IT became something to be managed as opposed to something to lead. IT leaders who want to change things rapidly and quickly often find themselves at conflict with their own internal organisations; it becomes easier to accept the status-quo as opposed to embracing the change. 

Arguably, IT's success in becoming ubiquitous in many organisations has become it's biggest blocker to change; IT has become so large and so important that it's ability to respond to the business has become seriously compromised by this importance. Look at the sheer amount of work it takes to roll out a new version of Microsoft Office to an organisation? Okay, it is a moot point whether upgrading Microsoft Office buys very much to an organisation but lets look at the number of corporates who are still running with huge amount of risk because changing from IE6 to a later version is allegedly so complex. 

This apparent complexity has lead to an inward focus on the delivery of IT and paradoxically this focus on the delivery of IT has significantly damaged it's delivery. It has become more important to the manage the delivery than to deliver. 

However, we have already seen changes in the way that applications are being delivered and developed; agile development techniques are probably used for more than 50% of our applications internally. I suspect that this is an industry trend and your mileage will certainly vary, yet I remember when I first ran an agile development team nearly ten years ago that this was considered to be extremely unusual and that the 'Agile Manifesto' was some kind of hippie movement. 

The move to agile development techniques has been remarkably quick compared to say that of take up of object orientated techniques which can only be attributed to that there is something right and easily implementable about the techniques.  You can find the Agile Manifesto here.

Agile development techniques have some key points that I believe point to the direction that all IT organisations need to take.

• Customer involvement in writing the stories which are to be delivered are key; every successful agile development has customer involvement at it's heart. Don't hide your people away from the Business; introduce them slowly if needed but stop making every communication between the IT organisation and the Business a management function.
• Leaders can develop without been forced into some management cookie-cutter mould. A great developer gets to work with many different developers and gets to disseminate their greatness but there is no pressure on them to become managers or team-leaders. 
• Teams should be allowed to organise themselves and define their own structures. They will often surprise you and if you allow them to break down organisational silos, you will find that their effectiveness is significantly increased. 
• The acceptance that no plan survives contact with the enemy. This does not mean that there should be no plan but there needs to be flexibility and options. 
• Simplicity and the art of doing the minimum to deliver is essential. For a long time I have preached the concept of 'Constructive Laziness'; do nothing which does not need doing and do what needs doing in the easiest manner. 

 How many organisations are ready to embrace this? I think this where the real difference between Leadership and Management will show, to implement such principles means taking up the mantle of Leader and dropping some of the comforting tropes of the Manager.  

Gladiators - Are You Ready!

Last week at #storagebeers, myself and Chris Evans were chatting and I was saying that you couldn't run your business with HP but you could with IBM or Oracle; although that wasn't quite true as IBM and Oracle still needed a network division (but then IBM go and buy a network company). 

So HP announcing Leo Apotheker of SAP as their new CEO seems to indicate that HP realise this and are planning to start going after the application space. I guess this is all going to be about owning the whole stack! 

It's going to be an interesting ride folks. Snorage may be snoring but the stack-wars are going to get interesting. I hear IBM are going to bring back the old dress-code as we head back into this strange 60s timewarp! 

 

Special?

One of the 'innovations' which EMC are to be congratulated on is not a product and no, it's not their Social Media policy; it's the concept of the vSpecialist or as I prefer to think of them, 'vGeneralists'. I think as a model for end-user departments, it is a concept that has merit and one which more companies should explore. But in many ways, it's not a new idea; there are many aging mainframe bods who in many ways fit the mould as well; many mainframe system-programmers could turn their hand easily to most of the disciplines within the infrastructure.

 
We did have our own specialisms; I certainly never worked in a mainframe storage team or network team but I am certainly of no doubt that when I was working in the world of mainframe, I probably could have done the day-to-day job of any of those. And even in my brief spell as an mainframe application programmer, I was certainly expected to write my own JCL*, plan job schedules, allocate the right sort of disk and create many of my tools. So arguably, in the era when mainframe was dominant, we had both 'vSpecialists' and 'DevOps'; so it is great to see it coming back round again.

I long for the day when we don't have Storage Managers, Network Managers, Server Managers and all the layers beneath them; I'm not sure what we should call this new breed; perhaps we'll end up with System Programmers, Systems Managers and the likes again working providing platforms and not infrastructure siloes. I long for the day when the silos which were never there in the past can be removed.  

But I bet those bloody DBAs stay a breed apart!!!

p.s Yeah, I never wrote my own JCL, I only edited someone else's...but hell, we all know that there has only been one original piece of JCL written ever!!

 

 

Storagebod likes this!

I really like this blog entry by Vince Westin; it absolutely confirms a belief that I have held for sometime! Application vendors have shares in storage companies! 

Poorly written SQL statements have probably driven more high-end array purchases than any salesman working for a vendor. Actually I suspect that poorly written SQL statements have driven more Enterprise IT purchases than anything. 

It is actually nice to see a vendor rather than recommending an expensive upgrade actually doing the sensible thing and looking for the underlying problems. 

Still perhaps Oracle purchased Sun because they absolutely realise that this is the case; they can now sell you both a nut but also a very expensive sledge-hammer to crack the nut. 

I suppose if I have one big concern about Cloud is that if we allow people to treat resource as abundant and cheap enough to waste; will code-bloat and poorly written code proliferate?

 I am convinced one of the major drivers of storage growth is that desktop storage is now so cheap; it can be wasted but the same is not yet really true of Enterprise Storage; a terabyte of 'Enterprise SATA' is very much more expensive than a terabyte of 'Desktop Storage' but try telling the average developer or user that? 

Bedside Manner?

At times listening to some of my fellow Corporate IT bods, you have to wonder why IT exists? I think it must be something to with the amount of money that gets spent on IT which seems to lead some people into believing that IT has some kind of special value and that businesses exist to fund the IT department. 

Seriously, talking to some people; I really think that people believe this and what's worse, is that our customers have picked up on this. So any spend that IT makes is almost immediately a red-flag issue; it's just the guys in IT wanting to spend money.

But how do we change things? And can we change things?

Firstly, it's about time that we started treated our users with modicum of respect; we're all guilty of the 'users==lusers' thinking at one time or another. We like to think that we are the technological elite but many of our customers have really rather sophisticated needs and often they are much further ahead on the uses of technology than we are. 

Nothing has demonstrated this more than the growth of the use of social media in the business; I know that within the organisation that I work for that IT is probably behind the curve on this, the real thrust to use social media has been from the business. In fact many people in IT still see it as just a toy without realising that is replacing email for many of our users. No-one has asked the IT organisation for permission for setting up Twitter accounts, Facebook pages etc; they just do it. 

Then look at the desktop; how many people work at companies who have massively locked down desktops? And when the locking down was done, who made the decision about what applications would be provided? Who is engaging the users to find out what they want? 

Our users grow ever more sophisticated in their demands and too often, the IT department is seen as the brake. And a very expensive brake. 

IT people seem to see every problem in the terms of technology and often they are framing answers in the terms of the technology that they are aware of. And how they make the best of use of technology to suit *their* processes. Too often they start with a technology answer.  Normal people don't especially care about technology; they care about outcomes and how they work better and improve the business and generally they want to be involved in the changes that impact them. 

I think that IT needs to develop a bedside manner; we do deal with a technical subject and at times it is highly specialised, just like medicine and we could learn a lot from some of the changes that the medical profession is undergoing. Arrogant, aloof and generally obnoxious doctors are rarely tolerated these days; yes, there are some old-school surgeons/consultants about but there are less. 

Doctors, like ourselves, are having to cope with the fact that our parish now has opinions and knowledge about our specialised areas. And yes, at times, this is incredibly frustrating; just talk to any GP and they will tell you about the times when their patient has googled their symptoms and come up with some weird and wonderful disease, convinced themselves that they have days to live and panicked themselves when all they have is a cold. But they will also tell you about the well-informed patient with a chronic condition who knows more about it than them, knows the treatment options and cuts out a huge amount of work. 

The trick is to treat both with the same amount of courtesy and also identifying which is which; working with and listening to the patient is the only way to do this. Just reaching for the treatment cabinet is easy but not always the right thing to do. 

And almost always outcomes are better if the patient is involved and sometimes the outcome the patient wants is not necessarily what the doctor wants. We could learn a lot from this attitude. 

Infrastructure is Software

Chuck has just written a blog which was very similar to a blog that I was working and I agree with a lot of what he says but I'd take it a lot further and there are some interesting conclusions and potentials along the way which could open the market for interesting innovation going forward.

Chuck talks about storage as being software, go read his blog; there's little I would disagree with there at all; well, until he starts talking about EMC products! However, I would go much further and suggest that we are getting to the stage where all infrastructure at a very real level is becoming software. Although I am not totally enamoured with the Intel-focussed monoculture, it has allowed a common hardware platform and it is flattening the playing field when it comes to hardware differentiation.

In fact, to differentiate your hardware is going to become increasingly expensive and hard, so why bother? Yes, there will be edge cases where hardware differentiation will be a key USP but in 90% of all use-cases; an Intel box assembled from bog-standard off-the-shelf parts will be good enough. 

If we then factor in pervasive virtualisation in the data-centre; we have a platform which has become pretty much standardised and commoditised. I would like to see more 'standardisation' in the virtualisation arena but it's not that bad at present and you really do not have that many choices.

So this has some interesting results; it lowers the cost of entry for new players in the market, if they no longer have to spend time developing a hardware platform and packaging their infrastructure product as a device but simply as a 'soft appliance'; they can get it out there a lot quicker. If they can now rely on the virtualisation layer providing them with a common way of accessing hardware services; they can develop a lot quicker and they can try things much faster. The cost of failure is a lot less; it also allows integration with other types of infrastructure to be tested without a lot of expense; this is a plus for both the developer but also for the interested Infrastructure specialist.

The speed to market is greatly enhanced and they can get it front of idiots like me who will download the appliance and have a play and it's not just storage appliances. There are products like Traffic Squeezer which do WAN Network Traffic Acceleration; there are more open-source router projects than you can shake a stick at. 

I am slowly building a virtual Data Centre out of open source or at least free products; I want to see how far it would be possible to get. But I'm not suggesting that anyone would do this for real today, although I can see some Cloud providers having a really good bash at it. This approach probably would not make sense for most companies as a complete strategy but as part of a strategy, it may well be worth considering. There are large companies out there who invest in start-ups simply to develop stuff for them to use internally; the advent as 'infrastructure as software' without a huge investment in tooling up to build hardware means this a very viable approach for the biggest companies. 

Google do it, Amazon do it but often you hear the comments, 'Well they employ very clever people, so it's easier for them!' Well, don't you employ clever people? Or are you saying that all your employees are second rate.

It took me a long time to come round to the idea that commoditisation and standardisation could drive innovation at all levels; I now believe that it could. It's not just about more and more Web applications; it could drive a new wave of infrastructure innovation as well. 

This leaves some interesting conflicts on the horizon for companies like VMware; perhaps VMware might want to get into infrastructure appliances but that would lead them into direct competition with the Mothership. Infrastructure as software; interesting times.

Here's some links of things worth looking at or playing with, it possibly includes things which are not strictly infrastructure but are interesting anyway. Some are great, some not so great; some show great potential. 

Traffic Squeezer

Openfiler

Samba Ctdb

Vyatta

Amanda

OpenDedup

And of course, there is Ubuntu Server; which will let you build your own cloud for free; there are various ZFS-based storage appliances. You can build your own appliances as well, packaging up and integrating components in the way you want. 

One area where EMC have shown great foresight and that is investing in the vSpecialist team and building a team out of diverse specialities because as infrastructure becomes software; the cross-over between the infrastructure disciplines will become even more mandated. Now the vSpecialist team may be very focused on the 'EMC product set' but if I was an SI or another vendor playing in this space; I would be looking at doing something very similar in the near future.