Storagebod's Blog

Seeding the Media Cloud - Part 2

In Part 1; I described GPFS, focussing on it's ability to massively scale but scale alone doesn't make a Cloud; whatever anyone would have you believe.

GPFS has two features which could allow it to become almost Cloud like. 

Firstly the move to Linux allows it to be based on commodity hardware. GPFS actually doesn't care what it's back-end disk is; it could be direct attached SATA, it could be V-MAX attached via a SAN, it could be iSCSI or it could be EFDs; it really doesn't care. As long as it appears as a block-device, it should be able to use it.

As for network, as long as it's IP; as far as I can tell, it doesn't care either. The faster the better obviously!!

So you can scale-out on fairly cheap hardware.

But it is GPFS's ability to move data around for you which could enable the most Cloud-like attributes.

GPFS like other IBM storage products supports the concept of storage pools. A Storage Pool is simply a group of storage devices, likely with similar characteristics; performance and reliability come to mind. A file-system must consist of a least one storage pool but may be made up of up to eight storage pools including the ability to define a storage pool external to GPFS such as TSM to allow data to be migrated to tape.

There is also the concept of a fileset which is basically a sub-tree of the GPFS global namespace; it allows administrative operations to be granted on a portion of a the filesystem and each fileset has it's own root directory and all files belonging to the fileset are only accessible via the root directory. Please note, this does not allow secure multi-tenancy but conceivably they could become the foundation of a secure multi-tenancy capability.  

GPFS includes within it a policy engine which allows it to automatically manage file data using a set of rules. This allows you to control the initial placement of files at creation time but also move these files over time. 

Placement rules are obviously run when a file is created.

Migration rules are run from the command line on demand but normally from a job scheduler like cron.

The rules are written using an SQL like language and can act on a number of attributes depending whether it is a placement rule or a migration rule; placement rules can work on user, group, fileset/sub-directory or filename. 

So you could do something like

Rule 'mp3' SET POOL 'SATA' WHERE UPPER(NAME) LIKE '%MP3'

Rule 'db' SET POOL 'FC' FOR FILESET ('database')

Rule 'sox' SET POOL 'Encrypted-disk' REPLICATE (2) FOR FILESET ('finance')

Migration rules run on things like Last Modified Time, Last Accessed Time, File Size as well as all of the attributes available placement rules.

So you can do things like

Rule 'logfiles' MIGRATE TO POOL 'sata' WHERE UPPER(name) LIKE '%LOG'

Rule 'core' DELETE WHERE UPPER(name) LIKE 'CORE'

You can also carry out actions based on a date; so you could move all the end of year reporting files onto your fastest disk before they were needed and then back off again the next week.

IBM in their typical fashion call this ILM and hence everyone ignores it because ILM is deeply unsexy. But they have the basis of a simple but powerful policy engine. And yet they don't tell anyone about it.

So do IBM have a storage cloud? No, not yet but they are close. The policy engine is simply not powerful enough yet but it could evolve into something and it's been around for long enough that you should be able to trust your data to it. 

And it does have some other nice features, like replication, snapshots, multi-clusters, GPFS over a WAN. I just wish IBM would make it easier for you guys to play with it but if you've got an IBM account manager, hassle them for an evaluation copy. 

If they try to tell you how complex it is...they're living in the past. It's much easier than it used to be and you can build yourself a small virtual cluster, quickly and easily in the virtualisation environment of your choice. I'd suggest building on top of Centos 5 but just remember to edit the right files to get it to pretend to be proper RedHat. 

As I rediscover this well-hidden IBM product, I'll be sure to share my findings.

Seeding the Media Cloud - Part 1

If you are a regular reader of this blog and a follower of my tweets, you will be aware that I work for one of the UK's largest broadcasters and at the moment, I'm working on a massive media archive. I doubt that there is a media company in the world who isn't trying to work out to move away from what is a fairly tortuous workflow which involves a huge number of systems and eventually ends up on a tape. 

When I say tape, it is important to differentiate between video-tape and data-tape; life can get very confusing when talking about digital workflow and you find people talking about tapeless systems which actually use a huge amount of tape but this is about from moving from a video-tape based system to a data-tape archive. 

But this little entry isn't about tape; it's about disk and more importantly, it's about how a build a massively scalable archive with some very demanding throughput requirements. In fact, what we are building is a specialised storage cloud and much of what we are doing has application in many industries. 

The core of this storage cloud is a cluster file-system; more importantly it is a parallel cluster file-system allowing multiple servers to the same data. The file-system appears to the users/applications as a standard file-system and is pretty much transparent to them, There are a number of cluster file-systems around but we have just chosen the venerable GPFS from IBM.

When I say venerable, I mean it; I first came across GPFS over ten years ago but at that time it was known as MMFS (multimedia filesystem) or Tigershark; it was an AIX only product and it was a pig to install and get working. But it supported wide-striping of data and it's read performance was incredible. I put this into what was a large media archive in a UK university (it is fair to say that I probably have more media on my laptop now) but it was very cool at the time.

MMFS then mutated into GPFS and what was already arcane descended in the world of the occult. GPFS was aimed at the HPC community and as such ran on IBM's SP2; I suspect it what at this point that GPFS got it's reputation as incredibly powerful but an extremely complex environment to manage. And running on top of AIX, it was never going to set the world alight. But in it's niche, it was a very popular product. However at about GPFS 1.x; I moved away from the world of HPC and never thought I would touch GPFS again.

In between now and then various releases have come and gone largely un-noticed I suspect to the world at large. However in 2005, an important release was made, GPFS 2.3 was released and Linux support was brought in.  I suspect it was at this point, that GPFS started to make a quiet comeback and move back into what was it's original heartland; the world of media.

So here we are in 2010; GPFS now sits at version 3.3 and supports Linux, AIX and Windows 2008; Solaris was road-mapped but has fallen off and there appears to be no more plans to support Solaris.

Figures are always good to throw in I guess; so let's get some idea of how far this will scale. 

• Maximum Nodes in a cluster: 8192 (architectural); 3794 Nodes Tested (Linux)
• Maximum Size of File System: 2^99, 4 Pb Tested
• Maximum Number of Files in a File System: 2 Billion
• Maximum Number of File Systems in a GPFS cluster: 256

So it'll scale to meet most needs and more importantly there are clusters out there which drive over 130 Gigabytes of throughput to a single sequential file. We don't need anything like that throughput yet.

And of course it meets those enterprise requirements such as the ability to add and remove nodes, disks, networks etc on the fly. Upgrades are allegedly non-disruptive, I say allegedly because we've not done one yet.

It also has a number of features which lead me to feel that GPFS is almost Cloud storage like in nature; it's not quite but it's very close and could well become so.

More in Part 2....

iPad: The Cloud Angle

There has to be a Cloud angle to every technology story these days, so I thought I'd take a stab at it for the iPad! 

Apple needs large Cloud enabled data-centres to support the iPad. Why?

The iPad could quite possibly be for many people the only computer they need; well, not this iPad but whatever comes after the iPad. But even this one will be good enough for some people but if the iPad is the only computer they need and have; they are going to need somewhere to keep all the content that they consume on the iPad and this is rapidly going to exceed the internal storage requirements of the iPad. So I expect fairly shortly that you will be able to back-up and store all the content that you consume, produce in Apple's cloud. I also expect that you will be able to share that content within your household; so if Mum, Dad and kids all have an iPad; then they will all be able to access the content bought.

One of things that has stopped me using iTunes to buy content is the idea that if I manage to trash my iTunes environment; I will end up purchasing all that content again. I am expecting Apple to allow me to back-up my iTunes content to the Cloud for a reasonable monthly subscription. And then I expect that Amazon et al will follow suit. 

So there you go, there's the obligatory cloud angle. I have some some thoughts on the device itself but they will wait.

Controlling Behaviour

Two very different press conferences/product launches happened today; you can't have missed them.

i) the iPad launch by Apple

ii) the completion of the Sun takeover by Oracle

But actually they had a very common theme; control.

Let's take Apple and the iPad and indeed all their products; Apple exert complete control of the hardware that their product runs on; indeed on their mobile devices, they even control the applications that run on their hardware. Some people hate this, they really do not like this controlling element; they go out of their way to do things to break-free of this controlling element. 

But for some reason, we stick with Apple's products; we may hate the company but we love the product; we accept their control grudgingly. We like the fact that we don't have to waste our precious time making things work together. And at the end of the day, we can get out of the relationship with Apple pretty easily if we really decide we don't like them.

Now let's take Oracle and Sun; Larry has looked back at history to the IBM of the 60s and I suspect at his friend Steve and decided I want some of that control.  In fact, Oracle found people who say that are looking forward to Oracle controlling the whole stack? The one throat to choke but I'm willing to be that in big Enterprise computing, no-one really wants this; they don't want to be locked in to a single vendor. We've been there and done that; we have choice, we have competition. 

Yes, at one level, life would be a lot easier with a single throat to choke but we know where that leads and we know if we get too much into bed with Oracle, it's going to be major struggle to get out of the relationship. There's too much at stake to allow Oracle the same level of control we grudgingly accept from Apple.  

More Cloud Witterings...

Must be time for me to witter on about some Cloud thoughts again, someone put up a slide at CloudCamp London last week basically saying something along the lines that in the Cloud all 'Servers are Equal'; you don't get a choice apart from number of CPUs or amount of Memory; that's about as granular is gets.  

So all the servers are equally likely to fail, you don't get any real options at the server-level for building in a resiliency, redundancy; even I/O performance. You can't ask for better network connectivity and you certainly can't ask for faster storage; sure you can have more storage but that storage will perform pretty much the same as anybody else's and that performance may change from day to day; hour to hour; minute to minute; application developers are going to have get pretty smart about how they handle changes in responsiveness from their back-end data-stores.

But perhaps automated storage tiering will come to the rescue? And of course this lets service providers come up with yet another charge; not only a charge for space consumed but perhaps also a charge for the number of IOPs consumed over a period of time. This also changes application design, not only do you want to engineer applications to cope with variances latency but you also want to design applications to minimise I/O. Arguably applications should be designed like that anyway but in the future, there might well be real cost incentives to do so..

Just thinking..

Works in process

Just in the process of finishing chucking in a solution to provide some of our creative types with a centralised place to store their files and works in progress. As is common to a lot of places doing digital work-flow and there is some debate as to whether you can call it a work-flow as it appears to the outsider to be a haphazard set of random events which eventually leads to some content being produced with much of the data currently stored on 

• local disk
• USB disks
• Specialised and isolated SAN islands

And the work-flow has evolved around this. Now in theory going to a more centralised storage environment should allow a more efficient and smoother work-flow but we have discovered one thing, creative types are even more difficult to manage and change than developers; so we are trying to compensate for their processes in the IT layer.

Ideally we'd like to sit down with them and make suggestions as to how to improve the process but like everyone, there simply isn't enough time. But almost counter-intuitively some of their inefficiencies can make us seem much more efficient; for example, we know that each individual will have their own copies of much of the content; we suspect that we might get a very decent return on dedupe. 

But it has left me wondering; we talk about changing process to meet the users needs and we constantly beat ourselves up about not being able to change fast enough (well, at least if we truly care about our customers); do our users fret about doing things differently in order to better leverage the investment made in IT infrastructure? Should they? Because if we are radically changing our IT processes and infrastructures, it might be better done as a collaboration. 

p.s I obviously didn't mean 'chuck in'; it was a rapid and agile deployment aimed at meeting user requirements as they evolved. 

Social Climbing

You cannot have failed to notice EMC's hiring assault on the blogosphere; especially anyone who is blogging on the wonderful world of V. It seems that EMC are aiming to get hold of some of the top talent and who can blame them? But apart from bolstering their technical resource, what else are they up to?

I wonder if some of Chuck's recent blogs give us some insight into what EMC are up to? If you strip away the EMC related rhetoric, you really end up with a compelling vision of a service oriented view of IT. There is actually very little to disagree with; even you if don't really believe in the Cloud; much of what Chuck's blog espouses what many of us have been trying to work too for years. 

So why the hiring spree? Well another common theme in Chuck's blog is that the technology to enable this change is really here and there is nothing stopping us embracing this apart from our own prejudices, either corporate or personal. It is simply a matter of psychology. Although, the real life situation is more complex than this; there is more than a grain of truth in this. 

And I suspect EMC is actually no different; yes, EMC have some people with vision and a real desire for change but is this as prevalent as Chuck would like us to believe? My experience would suggest that EMC is changing but not as fast as people like Chuck want/need it to. So what better idea than to hire in people who have already embraced the vision; dot them throughout the organisation and enable them to be change-bringers.

 The blogosphere is full of people with this vision and this is another example of EMC grasping social media and turning it to their advantage; also, think of the money that they are saving? Recruitment is an expensive game; by careful targeting, EMC are saving themselves a whole bunch of money.

 I hope this doesn't impact some of the excellent independent voices that we have but I can only applaud EMC for their boldness and congratulate those new hires on their new jobs.

$$ per Terabyte is a Good Metric!

Now there has been a lot of discussion about whether cost per terabyte is a good metric when purchasing storage, nearly all of it dissenting (some of the dissent may even have come from my direction) but don't let me ever be accused of running with the herd or even consistency. I'm now going to argue that cost per terabyte is a fine metric for purchase of storage.

Now I wouldn't argue that cost per raw terabyte is a good metric for storage but per useable terabyte it certainly is. Indeed for me, it is sometimes the only metric!

Have I gone mad? Well I don't think so. But I simply have a different use case to most of you; I simply need capacity and throughput; I have a known and very consistent workload, it is pretty much purely deterministic and I don't need any of your fancy snaps, dedupe; replication etc. But I do need throughput measured in gigabytes per second and I need capacity. 

Oh hang on but I am worried about physical space and environmentals and I am worried about scalability. I don't want to end up managing fourteen arrays when another vendor can do it in two and building a new data centre is not an option today; well, not until we've finished the new one. So even in this the most simple of cases where the requirements might be quite straightforward, demanding but straightforward; straight cost per terabyte is still not the complete picture. 

And of course, note I said useable terabyte; useable terabyte is not just about RAID and hot-spare overhead; how much disk do I need to get the throughput I need. Why is life never simple?

But cost per terabyte can be an okay metric, you just need to know what that cost consists of.

p.s if you think your data-growth is bad, save a thought for me and my team; the move to 3D video is going to drive data-growth for us in the broadcast media sector to scary amounts. And not only does it drive data-growth, it is also going to drive raw throughput demands through the proverbial roof. 

Asking the Right Questions...

How often do we ask the right questions of our vendors? Do we ever ask the right questions, I suspect that I could pretty much reproduce a good 80% of everybody's RFP/ITT questions, especially in the storage world. We find ourselves asking the same question again and again; even if we want to ask different questions, we find ourselves lead down the same old paths. And then, we ask the same old questions from our business users. 

Is it any wonder that we regularly find ourselves in quandaries with products which meet neither our needs or that of our business? Lets take availability for example, we demand our vendors supply five nines available kit; then we ask the business, if they want five nines? Invariably they respond yes and then they start to talk our language because that's how you talk to IT. And we start seeing five nines as a business requirement. So eventually it becomes a baseline requirement. We see business requirements poorly translated into IT requirements.

Perhaps we need to learn to ask different questions. Firstly we need to ask our customers what they are really trying to achieve and the impact of not achieving these. Questions may need to be asked in different ways? Perhaps instead of asking whether they need sub-second response on a screen; perhaps we need to show them the differences between different response times and what they find acceptable? 

When gathering non-functional requirements; wait till the end of the meeting to ask the question 'The service has been down since the beginning of the meeting; what is the impact?'. Or have a clock ticking on the screen and ask the meeting attendees to stop it when the 'acceptable' length of time for a service being out has been reached? 

Yes these things are gimmicky but I think we need to learn ask questions in a very different way.

And then how do we ask questions of the vendors? What are the important questions? I don't think it's whether that piece of kit is five nines available, I don't think it's how much storage I get for my pound. Yes these things are important but actually these questions are not hard enough. 

So what are the really hard questions we should be asking?

Disastrous Thinking

As follow-up to my blog here; I'd like to share yet more thoughts on availability and the potential negative impacts on some of the new technologies out there.

How many of you run clusters of servers? HA/CMP? Veritas Cluster? Microsoft Cluster? VMWare Clustering? I suspect lots of you do? How many of you cluster NAS heads? Yet again, I suspect lots of you do? How many of you cluster arrays? Not so many I guess? Certainly in my experience, it is uncommon to cluster an array. And when I talk about clustering an array, I don't mean the implementation of replication.

So, if you don't cluster your arrays; how do you protect against the failure of a RAID rank? Statistically unlikely but it is it more or less unlikely than a loss of data-centre? I'm not sure and the failure of a RAID rank for many people could well mean the invocation of the disaster recovery plan. Why? 

The loss of a RAID rank might well lead to the loss of an application/service and if it is an absolutely business critical service, can you bring it up at the remote replication site in isolation? As a discrete component? If you can, can you cope with increased transaction times due to latency? Many applications now have complex interactions with partner applications; these might not be well understood. So the failure of RAID rank could lead to the invocation of the Disaster Recovery Plan. Actually in my experience, this is very nearly always the case unless the service has been designed with recovery in mind; this requires infrastructure and application teams to work together, something which we are not exactly good at.

But you now take the challenge and make sure that every application can be failed over as a discrete component. Excellent, a winner is you! You know the impact of loosing a RAID rank, you know what applications it impacts, you've done your service mappings etc, etc. And you have been very careful to lay things out to minimise a single RAID failure's impact.

And then you implement automated storage tiering. Firstly, you now have no idea in advance what impact a RAID rank failure may have; you have no idea what applications may be impacted. And actually, the failure of a single RAID rank may well have huge impact. We could be looking at restoring many terabytes of data to cope with the failure of a couple of terabytes and many applications failing.

It will depend on the implementation of the automated storage tiering and I am concerned that at present we do not know enough about the various implementations which will hitting our arrays over the next eighteen months. So despite automation making things day-to-day a lot easier, we cannot treat it as Automagic Storage Tiering; we need to know how this works and how we plan to manage this. 

And perhaps for key applications, we will need to cluster storage arrays locally; that in itself will bring challenges.

I'm still a big fan of automated storage tiering but over the next few months, I would like to see the various vendors start talking about how they mitigate some of this risk. Barry Burke has made a big thing about the impact of a double disk failure on an XIV array in the past; in a FAST v2 environment, I would like to see how EMC mitigate against very similar problems.

I would also like to know what impact of a PAM card failure from NetApp is; does the array degrade to the extent where it is not useable? What kind of tools can NetApp give me to assess potential impact. As Preston points out here; failure of individual components within an array could have significant impacts.

We are heading to a situation where technology gets every more complex and arguably ever more reliable. But we rely on it to ever more greater extents; so we must understand risks and mitigations to a much greater amount than we have in the past.