Storagebod's Blog

Work for Bod??

Despite what we like to say in IT; it is rare for what we do to have a transformational impact on the day-to-day working of a business, fortunately I am currently in the position of knowing what I do completely changes how the company I work for creates it's product. And even better, I am in the position of being able to recruit someone else to come and join our little band of storage specialists, it's not an army, more a crack(ed) squad.

Over the next twelve months, we will be ramping up the delivery of a file-based workflow for content production for broadcast and at the core of this, is the storage; it is probably no exageration to say that all the company's eggs are being placed in our basket. We will be migrating into our new Broadcast Centre which amongst many other new features will be completely free of video-tapes; when I point out that we currently have over a million video tapes, this should give you some idea of the magnitude of our task. 

And whilst we are delivering this; we will be delivering 3D edit capability, render-farms, core back-up solutions for the Broadcast systems and anything else the business cares to throw at us. The skills that I am looking for are listed in the job advert but core skills are:

• Tivoli Storage Manager - Especially from an archive point of view
• IBM's GPFS - Clustered file-systems are becoming the foundation of the storage we deliver to the business
• NAS - NFS especially but also you need to be CIFS aware and an interest in pNFS would be helpful.
• General Storage - realistically, I don't really care what arrays you've worked on as long as you are ready to take on new challenges and also willing to let go of anything you have done in the past. We are not here to debate EMC vs HDS vs IBM vs NetApp etc; we are here to deliver a function not navel gaze!

Although it is not obvious at first, pretty much everything we do can be applied to the Cloud; no we don't do VMware (yet) but what we do do is build out massively scalable storage solutions based around pretty much commodity hardware. 

We're a small team at present; there's only three of us at the moment and I've no intention of growing it massively. The plan is to keep it small and focussed on doing the right things from day one. You will get the chance to work closely with both the other infrastructure teams but also the business teams. 

If you are interested, please go here and search for position number 02979 or just searching for storage should it.

If you have any questions; you should be able to find how to contact me and I'll be at #storagebeers on Thursday 2nd December. 

 

Compelling Purchase

I really like Compellent; I think that they are a great little storage company, they are a great bunch of people and their latest set of announcements are great; technically they continue to innovate and push their capabilities further into Enterprise territory.

Fluid Data, Live Volume all appear to be great technologies; lets not forget that Compellent blazed a trail with their data progression technologies which probably forced the big boys to announce and in some cases implement half-arsed solutions before they were ready. 

Compellent have also implemented some really rather bonkers solutions like 'Portable Volume' which just make me giggle. And I am waiting for EMC or NetApp to copy that! 

And yet for some reason, they've not managed to get the traction in the Enterprise that they probably deserve and I suspect that the changes that we've seen over the past couple of years means that they will continue to do so.

'Bare is the Stack with no Server'

Now, whether the converged infrastructure stack is going to win out in the long term is probably a contentious issue but it's certainly a door opener at the moment. More and more, CIOs are looking at infrastructure solutions and they are looking for vendors who can do it all. Whether they actually buy those solutions is another question but if you don't at least the ability to do so, it is increasingly hard to sit down at the table.   

'Bare is the Stack with no Storage'  

The answer appears to be smacking us all in the face and that answer is Dell. Yet until today's announcements, I didn't think Compellent were the answer that Dell were looking for. Dell want to be playing in the large Enterprise infrastructure space, building out clouds and dynamic data-centres; whereas I felt that Compellent were really an SME solution. I think that now they are much closer to where Dell want to be and it's almost a match made in heaven.

 Dell need Compellent and Compellent need Dell

There are a number of big infrastructure RFIs floating about at the moment and a number due next year; it seems that there are a number of large Enterprises who are heading into a refresh cycle pretty much at the same time; some of it driven by Cloud and other developments, some simply because they've held off for nearly too long. This is Compellent's chance to get a seat at the table but as a storage only vendor, it's going to hard. There's really only space for one storage only vendor; that's NetApp and unlike Compellent, NetApp's storage picture is much more complete (and yes, they can always partner with Cisco on FlexBlock).

Much as I like the Compellent storage story; like 3Par before them, acquistion seems to be the best way for them to meet their potential.  I really hope that they do succeed and thrive!

Supporting Role?

If you work for a vendor and after reading this you feel that this entry is about your company, then you are right! And if you work for a vendor and after reading this you feel that this entry has nothing to with your company, then you are totally wrong!

I've been working in IT now for more than twenty years; I've done a variety of roles but a big chunk of my career has been running support teams and I've done this both sides of the fence i.e both at an end-user but also at a reseller. I know a lot of people who have worked in support on both sides of the fence and pretty much all of us have coming to the same conclusion; over the past ten years or so, vendor support has got markedly worse! Of course, it may be that we are looking at the past through rose-tinted glasses but I do not believe so. I would also say that this is pretty much across the board; this is not a storage problem but it impacts hardware and software support across the board.

There are a variety of reasons for this decline in quality and ironically, it is some of the services that we as end-users have demanded that may have driven the decline in service. 

The Internet is a big driver in the decline of quality; there are huge amounts of support information and detail on line and we as end-users have demanded that the vendor support databases are put on line; unfortunately, this information is often presented in a manner which it is pretty much impossible to effectively search and you find yourself trawling through screens of unrelated sludge before you find the answer you are looking for. But because all of this information has been provided in a 'self-service' portal, this appears to have been used as an excuse to reduce the number of qualified people that you have in a support centre. 

The Internet has also further enabled the stock holding questions from the support centre and having worked both sides, I know that these are holding questions in general.

1) Are you at the latest level? 

2) Can you send me the logs?

3) Can you send me the configuration files?

Now all of these are valid questions in many circumstances but too often it feels that you are dealing with a robot who is working off a script! This is a Support Centre not a Call Centre; there is a difference! And my (and many other's) responses to the questions are

1) Why? Will it fix my problem? Where in the release notes does it talk about something even vaguely related to my problem? And if I'm in a really sceptical mood, can you send me the piece of the code which fixes the problem. And you realise that this is a live production system and just applying fixes is unacceptable, I will get asked all the questions I've just asked you by the change board. And if it makes it worse, it's my job on the line.

2) Why? Will that log help you diagnose the problem because I've already looked at the log and I can't see anything. Do you have any idea how frustrating it is when you have got a disk off-line and marked failed to be asked for logs from the array?

3) Which files do you want to see and why? 

And often, I've already attached the information to the ticket already; so you asking for them again shows that you have not read the ticket or are just working off the script.

Constantly in conversations with our various hardware engineers from all vendors, we discover reductions in headcount, experience engineers being retired early etc and territory coverage per engineer being increased. We are not buying less hardware, we are buying more and there are more systems out there to go wrong. Now arguably, systems are becoming more reliable but there are more of them. And in the world of storage, we have lots of moving parts; disks spin, tapes spool, robots move and these are all things which wear out and break. Disks get bigger and the potential impact of a disk failure and the resultant rebuild times gets ever larger. 

Talking to people who work in the support centres; it appears to be more important to keep the queue within the targets than solving the customer's problem at point of first contact. There is no longer time to do follow-up calls; for example, calling the customer who had the severity 1 call to ensure that they are happy with the fix. 

This is just the tip of the ice-berg and I could rant on and on about this subject; I was ranting about the decline in service years ago and yet it really is not getting better. For example, I am personally aware of four companies this year who have experience major outages due to problems caused by vendor support; it may be that now I have a fairly high profile in the industry that people tell me their war stories but it seems we are on an upward trend. 

I think it's about time that the vendors started to review what and how they are providing support (fix your websites or at least put a decent search capability on it, it is pretty awful that generally I find myself using generic search engines and the site: directive to search your site); it is also about time that they started treating support centre staff with respect and giving them time to do a great job. 

Isilon - A Wrap?

So it looks like that EMC will get Isilon barring any mishaps and counter-bids; I stand by my reasoning here but I've been thinking a bit more about this and I've had a few conversations with people.

I've chatted to a few vendors and many of them were saying that if they didn't work for their current company, Isilon were their company of choice; they thought it looked pretty cool. Not sure how many of them actually had any real idea about what Isilon were about but they thought it was cool anyway. Obviously once Isilon become an EMC company, their coolness factor will decline. (Sorry Chad, but the videos that the vSpecialists tend to record are bad and not in a good way!)

It's very interesting how quickly the whole scale out storage thing is progressing; everyone wants a piece of the action and Isilon certainly have the scale-out NAS story down pretty well; block, let's be generous and say that it is early days yet. But what was once a scientific and media specialised device is becoming mainstream. Isilon however were looking at ways that they could attack the entry point and how they could attack the Celerra and NetApp filer-space; so anyone who believes that Isilon is a cloud-only play from EMC, well I think that they are wrong. 

In fact, I would go as far to say that in the same way that 3Par will become the foundation of all HP's block storage; Isilon will become the foundation of all of EMC's file-based storage; from Celerra to Atmos, this acquistion will impact them all. Atmos with Isilon becomes a more interesting if not compelling offering than Atmos alone; it is there that I suspect that all of the initial integration focus will be. How do you graft Atmos onto Isilon is probably the next question that EMC's Atmos engineering team have to answer; well once they've worked out how to integrate Atmos and Centera and more importantly how to do so without a mouthful of humble-pie.

Of course this leaves us with some interesting questions about C-NX or whatever it might end up being called; like HP and EVA, EMC have some questions to answer. Still the Clariion is a much under-rated block storage array even by a large chunk of EMC and as Isilon really doesn't do block; I think it's future is safe for some time to come but perhaps it's another integration point. 

I also wonder if EMC will spend time moving the BSD-based Isilon product to their internal common Linux platform; let's hope this does not turn into a NetApp Spinnaker re-tread. I don't think that people should get too hung up on the interconnect between the Isilon nodes; yes it's Infiniband today but there's good reason for that, it was probably the cheapest and most effective low-latency interconnect Isilon could use without spend a fortune on specialised 'R and D'. Isilon's smarts are in software development and not really hardware; EMC could really help push them to the next level of hardware.

EMC's biggest challenge though is going to be taking a company which is culturally very different to EMC (I've dealt with both) and integrating it into EMC without loosing the clever folks that they've just bought. Isilon currently feels like dealing with a start-up but a start-up with mostly smooth corners and I don't think that they need to change too much. EMC might have just bought a company which will have as much impact on their storage offerings as their purchase of Data General all those years ago. 

 

Assault on Benchmarking

I've been thinking a bit about benchmarking and benchmarketing; pretty everyone agrees that SPC is a very poor representation of real world storage performance but at the moment, it's the only thing that most of the market supports with one key exception. So I thought I'd come up with my own, so let me introduce the SSAC.

Storagebod's Storage Assault Course

This is a multi-part benchmark and is supposed to reflect the real world life of an storage array. You may bid what you like but all costs including the costs of team who support and run the benchmark must be declared.

1) You must specify an array to run the SPC benchmarking suite; this array must be ordered through the normal ordering process.

2) A short period of time prior to the delivery of the array; you will be informed that the workload for the array has been changed to another random work. However, the delivery date agreed must still be met; so any changes to the configuration must be made without impact to the delivery.

3) You must carry out an audit of the existing SAN environment and ensure that the installation of your array will not cause any impact to the already running environment. In the course of the audit, you will discover that pretty much the whole of the environment is down-level and not currently certified against your new array. You must agree any outage required to upgrade the new array; this may involve you auditing every single server and switch to ensure that key variables such as time-outs on multipathing are set correctly.

The day before installation, you will discover a dozen servers which you were not aware of and at least three of these will be running operating systems which are so far out of support that no-one is sure what is going to happen.

You will be responsible for raising changes, carrying out risk assessments and arranging site surveys, traffic plans and any other supporting tasks.

4) On day of delivery, you will discover that the array is to be installed in another part of the data centre which neither has power and the array will probably fall through the floor crushing the secretarial team below.

You will be responsible for arranging the remedial works and rescheduling all the changes required.

5) Once the array is installed, you will be responsible for powering it up and installing the initial configuration.

6) A Major Production Incident is declared and you will be responsible for convincing everyone that it is not your new array which caused the problem.

7) You are finally allowed to run your first benchmark.

8) You discover that all of the SAN switches have been set to run at the wrong speed and you need to raise changes to correct this.

9) You are allowed to re-run your benchmark again but halfway through your benchmark a Performance Major Production Incident is declared and you are responsible for first proving that it is not your array which caused the problem and then fixing the Incident which is nothing to do with the work you are carrying out.

10) You manage to successfully run your benchmark. 

11) You receive an urgent change request and you must find space on your array for a new workload; this workload has completely different performance requirements but you will told to JFDI.
12) You run your second benchmark.

13) You receive an urgent call from the original application team; apparently they mis-stated their requirement and they are servicing twice as many requests as originally thought. You must expand the environment as an urgent priority. 

14) Through the normal ordering channels, you must arrange any necessary expansion to the current environment. 

15) You must carry out any necessary reconfiguration without impact to the running workloads or agree any downtime with the business to allow you to carry out the necessary reconfiguration.

16) You will recieve an emergency technical notice from your vendor support teams; a fix must be implemented immediately or the array will catch fire (possibly).

17) You must re-audit the environment and carry out any remedial work on servers, switches and applications to support the new code level.

18) You will be informed that the system must now have a DR capability due to government regulation.

19) Carry out any design and certification work to provide DR capability. The order for the DR capability will be held up in procurement but you still need to implement within the government timescales or face severe financial penalties.

20) Re-benchmark the entire environment showing the impact of replication on the environment.

Explain to a non-technical person that the speed of light differs in fibre to that of that in vacuum. Explain in a louder voice that no, the speed of light is not something which can be overcome and no, rival company X has not overcome despite what they might say.

21) A third workload will be detailed; this must be shoe-horned onto the array at no extra cost but with no impact to the additional applications.

22) Benchmark again to prove this case.

23) You are summoned to a meeting to explain and justify your array to a Senior Manager who has just been bought a three star lunch by a rival vendor.

24) You will experience a failure on your array; record how many logs and diagnostics that the vendor support team (and no cheating, you must use the customer route) ask you before agreeing to replace the failed disk.

25) You will be informed that Group Audit have looked at the application that you provided DR for and decided that actually it was exempt, hence you have wasted time and money.

26) Plan to repurpose DR array for a new workload requirement. 

27) A new capability is announced and will require a major firmware upgrade that may require downtime but your support team are not sure. Carry out the risk analysis and plan for upgrades.

28) You are outsourced!

At this stage please disclose:

1) Total Staff costs including time-off for stress related illness

2) Total Capital Expenditure

3) Total Downtime due to upgrades and other non-disruptive activities

4) Any other expenditure due to work required to existing SAN environment and Server infrastructure

And those performance benchmarks, throw them away; you didn't think I was actually interested in them!?

Quick NetApp Thoughts...

It's days like today that I really feel for vendor techies; proudly watching their newest products sail off into the big wide world to be marketed and sold to the unwashed masses. 

They watch their baby pawed and prodded by their rivals; who inevitably tell them that their baby is ugly. And no-one likes that. Chuck is inevitably straight up to the plate and I bet none of us predicted that.

He accuses NetApp of being boring; well, I guess it is really. All of NetApp's hardware announcements are pretty boring, NetApp are not a hardware company. It's OnTap announcements are where the interest is really and I would have liked to see a bigger OnTap announcement, you can't keep announcing the same product. However, the hardware announcements add things which have been asked for by the customer base for sometime and I can't fault them for that.

And some of Chuck's comments are pretty hollow, 

'getting ridiculously easy to win most any performance benchmark against NetApp'

Apart from the industry standard one which EMC won't play ball with. Actually, I had an interesting conversation around the subject of benchmarks with regards to EMC; it might be interesting to take a standard benchmark of some sort; run it on a raw VMAX or DMX or CX, then run it against the same array but this time behind one of the virtualisation controllers say from NetApp, IBM or HDS. It might be kind of fun. 

But really, unless all the industry agree to release their products on the same refresh cycle, we are always going to be in the situation where we are comparing different generations of product. Although IBM, HDS and NetApp have all refreshed this year; so it appears that EMC are contrarian. And perhaps for the next six months or so, NetApp might be in the position where they can say that it is,

'getting ridiculously easy to win most any performance benchmark against EMC, especially CX'

However his comments about spindle counts ring true but I think this is very true for much of the industry including EMC et al; the use-cases for 1000+ spindle arrays are pretty limited in general. Certainly if you load the array with SATA spindles, low-IO rate archival would seem to be the only use but I suspect that there might well be better and more cost effective options. 

For example, many of the value-add features are not really of a huge amount of value in these environments; you are probably not going to be snapping or cloning data on a regular basis; depending on the type of archive, dedupe might be of limited value or not even desirable. 

Also I worry about migrating off of these huge arrays; migrating petabytes of data is time consuming and it needs to be done in a non-disruptive manner. If NetApp came up with a way of changing out and upgrading the heads with no downtime; that would be cool. Of course this is where someone from NetApp pops up and tells me that there is exactly that product and I look stupid but hey, I'm willing to take the chance. BTW I'm talking about data-in-place upgrades; just whip the head out and replace it with a bigger model.

So, yes...boring announcement but no the less welcome and needed. 

Apple Abandons Enterprise?

So with the announcement of the withdrawal from sale of the 'Xserve'; have Apple finally abandoned the Enterprise? Actually to abandon something, you would have had to have been serious about it in the first place and Apple have never really been serious about the Enterprise and certainly the Data Centre. Pointing people at the Mac Pro or the Mac Mini is not really a serious suggestion is it? 

Apple have never had and I suspect are never likely to have a serious Data Centre presence in the world of Corporate IT but are we going to see Apple attack this from a different angle. We all know that Steve Jobs loves the ability to control the whole stack from top to bottom and I don't think he is yet ready to completely abandon any ambition that Apple have in the Corporate IT space; indeed, I think that he has been heartened by the user-driven success of the iPad and iPhone in the Corporate space and there is a realisation that he cannot compete with the traditional corporate suppliers on their own ground. So, he's going to try to move it to his ground.

The creation of the OSX app-store is simply the first shot across the bows; the ability to manage the applications on your Mac device in the same way as managing them on your iPad, iPhone (or indeed your Android device) will put a huge amount of pressure in Corporate IT suppliers to provide the same desktop experience to their users. I suspect Microsoft are going to come under an increasing amount of pressure from their corporate customers to provide similar functionality to allow application management by end-users. 

But the current Apple app-store does not really provide corporate customers with the level of control that they require and also, it will be completely unmanageable at scale; for this to be a true corporate play, we are going to need corporate app-stores where a corporate end-user can log into and download/install the apps which their employer have licenses for. However, once you get used to consuming applications in such a model; it is simply a better way of managing things. 

And I wonder if this will be Steve's next play for the Enterprise? Or perhaps Apple are truly about to dump the Enterprise and become a consumer-only company. If he doesn't do it; Google will do it as part of Chrome OS, if that ever sees the light of day as a non-beta product.

I suspect even the faithful of the media-world are getting a little nervous with the current behaviours of Apple; for example no FCP refresh until 2012 in a technology area which is changing incredibly quickly with new demands being made all the time is a little worrying for many of them.

2011 is going to be an interesting year for Apple watchers.

Apple's Debt

Apple owes a huge debt to Windows and the PC industry; in fact, without the PC enthusiasts, Apple would have some really interesting problems. People like to point out that Apple has moved over to commodity hardware; which it has mostly but why has that hardware become commodity? It’s because of the PC. 

Steve Jobs makes a big thing about controlling the whole stack, both hardware and the operating system come from Apple; in this way, Apple get to build devices that simply work.  But the world of the PC massively subsidises his hardware development costs; for example, without the PC and especially the PC games players; where would Apple’s graphics cards come from? And there are many more examples in all of Apple’s hardware. 

It would be nice for the fan-bois to acknowledge this debt; there is no doubt that Apple put together some really nice machines, this entry is being typed on a new MacBook Air which is simply the most gorgeous laptop that I’ve ever owned. Is it revolutionary though? Not really, it’s just a really great instantiation of the laptop. And yes, it’s probably overpriced for what it is but then again, you can say that for many things. 

Perhaps including the integrated stacks some of the vendors are pushing…they need to stick Apple logos on them I guess!!

 

The Information Business

It strikes me that any industry/profession which is the business of providing information or access to information is currently in an incredible state of flux and change. My friend Phil Bradley who blogs ons where librarians and the internet meet and has written a number of books on Internet search and has been involved in information provision since he studied librarianship many years ago discusses this a lot on his blog and on his twitter feed. And although he claims to read my blog and not understand a word of it; I think that our two professions have much in common and could well become much closer over the coming years.

Librarians in the past have been the gatekeepers to knowledge and information; not only that they have curated and preserved much of the world's knowledge but as we have moved away from the physical representation of that knowledge and into storing that knowledge in the forms of bits and bytes, some of that responsibility has passed into the hands of the humble storagebod. Just look at SNIA's 100 Year Archive initiative; if that doesn't have aspects of the librarian as curator, I'm not sure what does.

And we have a common problem; our users want access to knowledge and information in a multitude of different ways, they want to consume information without the gatekeeper and instead of being the gatekeeper, we need to be the enabler. We are both no longer able to tell our users how they should consume information; we need to listen to them and provide it to them in the form that they want. Those forms are changing all the time; be it video, audio, XML data, printed word, web-portal; almost any form that you can imagine.

Also the nature of information is changing with some dramatic impacts; information is real-time, there is no real lag between creation and publication of the information. The data-sets behind the information change all the time; yet provenance becomes ever more important.

You can't simply replace a librarian with a search engine and let the world get on with it; lets take Wikipedia, it's not a bad source of information as long as you except that it can often be wrong and not only that, sometimes the information put on it is designed to be deliberately wrong. Yes it is peer-reviewed but after publication; if you were unfortunate enough to read an entry which had just been erroneously updated and without checking, you could well have just cited a load of the proverbial. 

Much academic publication is now done electronically on the web but how does the humble student determine which is good source and which is not; also citation (and just Wikipedia) is becoming very dangerous; information moves and vanishes all the time. A perfectly good citation could simply change URL and throw a premise into the realms of conjecture; how do you cite a particular version of website? 

The data-sets that our businesses run on are updated in real-time but too often business decisions are made utilising versions of the data-set which are disconnected from the source and could be months out of date. There are various reasons for this but sometimes it is because we simply make it too hard to take a real-time feed of the information the business requires. So decisions are made on information which could be basically hogwash. But yet, we need to ensure that the information provided is correct and not corrupted in some way; it is not enough to ensure that only the right people have access to the information but also ensure that only the right people can change that data. 

The provision of information in a timely manner and in the way that our users want is a challenge both for the librarian and the storagebod; perhaps at the end of the day, we are all just Information Professionals. I'm sure that we have much to learn from each other; they've been doing it for thousands of years.