Storagebod's Blog

Not a Cloud Post!

Currently I have a whole bunch of blog posts half-written but at the moment inspiration seems to have gone a bit south. So I thought I would post on something completely different although it'll probably mutate into something familiar.

Now anyone who watches my twitter feed will probably have seen a few tweets on Spotify, the streaming music service available in some countries in Europe and coming soon I believe to the States. Although not the only streaming service available; it is in my opinion one of the best, it has a great selection of music covering all genres (I recently discovered that it has a growing classical selection) and it has great clients available for Mac and Windows but no Linux at the moment.

It also has two great mobile clients, one for Android and one for iPhone. If you want to use the mobile client, you must pay for the premium service but if you are happy with your music being interupted by adverts every now and then, the desktop version is free.

Now Spotify and services like it in many ways embody what to me is the real beauty of the Cloud model; a service which can be accessed anywhere from many devices but at the end of the day, the end product is the same, music streamed to my ears.

But this post isn't about Cloud, it came about after a brief MSN chat with a good friend of mine who specialises in all things Web 2.0 and especially getting useful information from the Internet; he's been training librarians and all kinds of other people how to use the Internet for years. And he mentioned that he had recently given a talk on how things like Spotify change things; it breaks the link between the physical instantention of the artifact and moves it completely into a virtual world and in doing so, it changes certain value assumptions.

Spotify for example has millions of tracks, now they are all searchable and I can just search for an artist and play their content; certainly, that's often how I use it but it also has the concept of playlists and publically shareable play-lists and it is these which will become more and more treasured and valuable.

Of course, it would be really useful if I could take Spotify playlist and then point it at another service such as Sky Songs and if playlists were portable. Or even take my iTunes database and point that at Spotify. I guess what we are talking about is portable metadata formats or at least gateways between services, in the Cloud or perhaps just stored locally.

Oh heck, this was a Cloud post anyway! We need to ensure that when we are building services or consuming services that in order to truly leverage the power of the Cloud, that we think about portability and flexiliblity. My playlists are currently locked into Spotify (and iTunes) but I am actively thinking about how I get round this and build a truly portable store; we need to think about this in our work lives as well.

Bambi-based Computing

I found this article in the Register very interesting but not as interesting as the paper it links to.

The concept of building balanced systems to improve efficiency and reducing TCO is one which is elegant and pleasing; I wonder if at times that we in the storage world could do better in this? The paper also points out one of the things that we are very aware of in storage; spinning rust has just not kept pace with the development of CPUs.

It also has a fact in that I was not aware of consciously, 2x2Gb DIMMs draw as much power as a 1 Terabyte disk. So simply chucking huge amount of memory in the form of cache will not solve all problems. I like the new measure that they come up with for measuring efficiency, Queries per Joule. Perhaps we could have a new SPC/TPC measure based on power efficiency.

Woah, Scarey Answers

I've read some scarey technical questions and answers before but this is terrifying; it's a rather blase answer to a question about merging SAN fabrics with switches from different vendors. Folks, if you can possibly avoid this, avoid it!

It is rarely a good idea and if you are using any of the functionality that you have probably paid for, it is going to be a horrible job! You would better implementing a seperate fabric in most cases as opposed to growing an existing fabric with a different vendor's switches.

Believe me, I've looked at in the past and the certification matrices, patches and the added complexity means that any short term saving disappears very quickly! Actually, the lack of interoperability between the various FC vendor's kit is a reason to go for iSCSI for many people.

Conversations and comments

I had a few interesting conversations at IP Expo last week and I'd like to talk about a couple of them as they really highlight what is going on in a lot of IT Departments.

First was a conversation I had with a Senior Sys Admin who was very anti the moves we are seeing in the industry towards simplification; he really didn't want to see his job simplified and de-mystified as he could see that he would no longer have a job or not as well paid job. And all this automation and simplification didn't work anyway; all mouth, no trousers.

Secondly was a conversation with a vendor who acknowledged that in the past that their GUI had not been great and we should look forward to a much simpler UI with ease of use at the fore.

So we have an industry move towards simplification at a time of recession; an easy sell to the CIO but not such an easy sell to the people on the shop floor. I wonder how many of these initiatives are going to fail because of these attitudes; unfortunately, I suspect quite a lot but often broken promises, imperfect products and shoddy delivery has allowed these excuses to stand.

I think that lot of people in IT like the idea that what they do is somehow mystical and complicated; they get a thrill from getting called at 3 am in the morning, they bitch and moan like hell about it but deep down they love it. I've certainly seen this in my day to day work; heroes in IT are all over the place, working silly hours to keep systems up but if you suggest that there might be a better way, there is a look of horror.

I wonder what we can do to change this attitude? Ideas? Obviously, we need to ensure that the tools that are supposed to make lives easier, actually do make lives easier; often not the case.

All the functionality that is provided at the CLI needs to be in the GUI; this means there are less excuses to revert to the CLI. CLIs themselves need to be simplified and made consistent; as functionality has got bolted onto systems, the CLIs have got covered in cruft and are no longer syntactically consistent. 

We need to ensure that the simplification is actually simple; too often the automation breaks and means that we end up doing things manually anyway. It must deliver on its promises not deliver more promises.

Data Loss - nothing new here!

One of the fore-runners of information storage and retrieval has after a long running saga of various failures including security breaches, data theft, data theft, mis-indexing and general mal-administration including corruption and incompetence has finally closed it doors today in dramatic fashion which has resulted in the loss of a great proportion of the world's knowledge. Fortunately there were some distributed back-ups but sadly some data has been lost forever.

When the Royal Library in Alexandria burnt down, a huge loss to the world's knowledge was felt but nobody decided that concepts of centrally stored information in the forms of libraries was fatally flawed. It was also fortunate that there were other libraries, even within the great city of Alexandria; which ensured some books survived. It would have been better if there had been a policy of copying all scrolls and ensuring that these were distributed through-out the world but the concept of Libraries are still with us and no-one is suggesting that libraries are a bad idea.

The recent catastrophic failure of the Sidekick service resulting in thousands of people loosing their personal data should not be seen as a failure of the Cloud; it's not! It's the failure of a centralised service which was apparently run by incompetents!

It is yet another lesson that if you only have a single copy of your data; you might as well only have no copies of your data. So if you are archiving and deleting, you better make sure that you have two copies of the archive or at least the ability to recreate that data. Read your SLAs and ask questions about the data-protection policies of anyone who is looking after your data; both internal and external providers.

That reminds me, better take a backup of my blog and better back-up my gmail account as well.

Thoughts on Filestore

So Symantec have decided to enter the world of Cloud Storage; I'm not sure that simply packaging up a Linux Software appliance with some clustering actually counts but it was inevitable; if it was not packaged as Cloud, it would have been packaged as a basic NAS offering.

Realistically, Symantec needed to do something as NAS is a huge problem to them; if I go NAS, then I don't need VxFS and VVM. Even in their core Solaris market, ZFS will start to damage them. The volume manager as a standalone product is becoming an irrelevance. I've no idea what the penetration is in the Windows server markets but I doubt it is high.

Then there is Veritas Cluster Services; if I use virtualisation technologies such as VMware, then the clustering is part of the Hypervisor and I don't need VCS. Oracle RAC replaces the need at the high-end where VCS had a lot of it's core deployments protecting Oracle databases.

And I'm not sure what impact NetBackUp 6 has had on Symantec's share of the back-up market but it has been a poor release; both 6 and 6.5 have undergone patch after patch with problems always being fixed at the next patch level.

But Filestore is lauching into an increasingly crowded market-place and just comes across as a 'me too' product; I think Symantec are going to struggle with this one, it currently lacks features and it's price point is not as attractive it might be. I see some very tough times ahead for the Enterprise Software side of Symantec; the market's moved and I'm not sure that they can offer me enough.

#storagebeers

If you are attending IP Expo tomorrow and why wouldn't you be as the great blogger Chad Sakac is speaking amongst others; there will hopefully be a meet-up afterwards for beers at the The Atlas pub round the corner.

Please come along and join the discussion; the only rules are no vendor baiting and product pimping! It would be great to put some faces to names. And I really don't want to drink by myself! 

If you have any questions about the job I've posted about; it would be a great place to try and quiz me.

Work in the Sky, not Clouds

Okay, I very rarely blatantly refer to the company I work for; I don't hide it but I try not to talk too specifically about Sky. But I can't really avoid it this time as we've got a couple of storage analyst positions available. These are fairly unusual in that they are on the Broadcast side of the company and oh yes, there currently is not strictly a Broadcast Storage Support team; there's me doing it as an ad-hoc role.

So this is the chance to get in and actually try to do things right; it's a greenfield role and we get to work on defining our own processes and procedures. There's no legacy SAN, no legacy back-up and no legacy apps; in fact there's not a lot at present. There's a few packing cases and empty racks (and a building site).

You'll probably look at the list of skills and think WTF! But don't worry, there is probably about a dozen people in the country with the exact skills we are looking for in the UK, so we accept that we are going to have to train people. This is about attitude and wanting to get things done!

No you won't get virtualisation, you won't get deduplication and you won't get Cloud but all the skills you pick up will be relevant to all of these. You'll get the chance to build a system which will potentially redefine the core business workflow in Sky and you'll get to play with and see some cool stuff. Hell, we might even pay you to do it.

And you'll get to work with me...so it's not perfect!

Do you have an Umbrella?

Every now and then, we have 'interesting' conversations at work about morbid subjects; recently we had a discussion about another 9/11 and whether our systems could cope with it. Not the systems that you get to see but our internal systems and how we scale for peak load and what peak load actually means. Scaling web-servers is pretty easy but other systems don't necessarily scale so easily.

We had recently put a new system in and then Michael Jackson went and died; so that was a good test for the system, which coped pretty well. Tragic events are what generates traffic; that and England winning the World Cup but we have generally have to deal with tragedies not miracles.

But web-sites, phone-systems, networks can all fail when under extreme load; in fact, pretty much every form of utility is massively over-subscribed pretty much by design. If everyone got in their cars at the same time, the road network would melt-down. Flights are regularly over-sold, certainly by the budget providers. If everyone in the UK decides to make a cup of tea at the same time, the power-grid suffers.

This led me on to thinking about the Public Cloud; what level of over-subscription do the Public Cloud providers sustain in the forms of 'reserved instances' and how many instances can they actually support all coming up at one time. How much storage does Amazon actually have?

If you are going to bet your business on the Cloud, you probably ought to know and you probably ought to know what kind of events will cause you to burst and probably what will cause everyone else sharing the Cloud with you to burst. And what you are going to do if the Cloud does burst? Do you know where your umbrella is? Or at least the candles to dry you out.

What a Fraud!!

Sometimes when I'm writing this blog, I feel a complete fraud! Actually I suspect most of the time that I am posting complete nonsense. Many of the technologies that I post on, I will never use in my current role.

1) Virtualisation - neither storage or server virtualisation impinge on my world. And it's currently hard to see that they might. Digital media production doesn't lend itself to virtualisation at present even the flag-wavers admit that they are not ready for such environments.

2) Deduplication - Digital media, enough said. Although Ocarina might have a play, digital media is the one place that deduplication does not really play.

3) Cloud - if we are not ready for Virtualisation, then we are not ready for Cloud. Even Cloud Storage currently has a limited presence in the large, performance oriented media space.

But even so, much of what is talked about still has relevance to me in what I do. I don't need to virtualise to utilise many of the process and procedures which are talked about and being developed. At times we all get caught up in the technology and believe that it is the technology that is something special but really it is only half the story.

Even if you don't see yourselves using the technology, you might find many of the processes and concepts relevant to you; for example, I may not use the general purpose Cloud Storage being provided by many people but I am still interested in how to build meta-data and what meta-data should be.

Arguably, a big chunk of what I am working on at the moment is building a specialised media-cloud which will serve content to a huge variety of applications. If all these applications could talk to the media cloud in a standard way; life would be alot easier (they don't), so I'm interested in the standards which are being discussed.

Virtualisation, actually in the media cloud, we will have very few servers but there may be times when we have to scale very quickly. So I can learn alot from the virtualisation guys on rapid provisioning.

So I'll keep posting on things that I know nothing about and hope that you guys don't catch on!