Supporting Role?

If you work for a vendor and after reading this you feel that this entry is about your company, then you are right! And if you work for a vendor and after reading this you feel that this entry has nothing to with your company, then you are totally wrong!

I've been working in IT now for more than twenty years; I've done a variety of roles but a big chunk of my career has been running support teams and I've done this both sides of the fence i.e both at an end-user but also at a reseller. I know a lot of people who have worked in support on both sides of the fence and pretty much all of us have coming to the same conclusion; over the past ten years or so, vendor support has got markedly worse! Of course, it may be that we are looking at the past through rose-tinted glasses but I do not believe so. I would also say that this is pretty much across the board; this is not a storage problem but it impacts hardware and software support across the board.

There are a variety of reasons for this decline in quality and ironically, it is some of the services that we as end-users have demanded that may have driven the decline in service. 

The Internet is a big driver in the decline of quality; there are huge amounts of support information and detail on line and we as end-users have demanded that the vendor support databases are put on line; unfortunately, this information is often presented in a manner which it is pretty much impossible to effectively search and you find yourself trawling through screens of unrelated sludge before you find the answer you are looking for. But because all of this information has been provided in a 'self-service' portal, this appears to have been used as an excuse to reduce the number of qualified people that you have in a support centre. 

The Internet has also further enabled the stock holding questions from the support centre and having worked both sides, I know that these are holding questions in general.

1) Are you at the latest level? 

2) Can you send me the logs?

3) Can you send me the configuration files?

Now all of these are valid questions in many circumstances but too often it feels that you are dealing with a robot who is working off a script! This is a Support Centre not a Call Centre; there is a difference! And my (and many other's) responses to the questions are

1) Why? Will it fix my problem? Where in the release notes does it talk about something even vaguely related to my problem? And if I'm in a really sceptical mood, can you send me the piece of the code which fixes the problem. And you realise that this is a live production system and just applying fixes is unacceptable, I will get asked all the questions I've just asked you by the change board. And if it makes it worse, it's my job on the line.

2) Why? Will that log help you diagnose the problem because I've already looked at the log and I can't see anything. Do you have any idea how frustrating it is when you have got a disk off-line and marked failed to be asked for logs from the array?

3) Which files do you want to see and why? 

And often, I've already attached the information to the ticket already; so you asking for them again shows that you have not read the ticket or are just working off the script.

Constantly in conversations with our various hardware engineers from all vendors, we discover reductions in headcount, experience engineers being retired early etc and territory coverage per engineer being increased. We are not buying less hardware, we are buying more and there are more systems out there to go wrong. Now arguably, systems are becoming more reliable but there are more of them. And in the world of storage, we have lots of moving parts; disks spin, tapes spool, robots move and these are all things which wear out and break. Disks get bigger and the potential impact of a disk failure and the resultant rebuild times gets ever larger. 

Talking to people who work in the support centres; it appears to be more important to keep the queue within the targets than solving the customer's problem at point of first contact. There is no longer time to do follow-up calls; for example, calling the customer who had the severity 1 call to ensure that they are happy with the fix. 

This is just the tip of the ice-berg and I could rant on and on about this subject; I was ranting about the decline in service years ago and yet it really is not getting better. For example, I am personally aware of four companies this year who have experience major outages due to problems caused by vendor support; it may be that now I have a fairly high profile in the industry that people tell me their war stories but it seems we are on an upward trend. 

I think it's about time that the vendors started to review what and how they are providing support (fix your websites or at least put a decent search capability on it, it is pretty awful that generally I find myself using generic search engines and the site: directive to search your site); it is also about time that they started treating support centre staff with respect and giving them time to do a great job. 

Comments

[Anthony Vandewerdt]

Hi Martin.

Great rant... I think you raise many valid points of concern.

One comment, you state in point 1 that:
"And you realise that this is a live production system and just applying fixes is acceptable"

I suspect you meant: "unacceptable" ?

[Martin G]

Edited to reflect what I really meant. It is really unbelievable that after all these years that we still get this a lot with no appreciation of change control within a large environment.

Even if it would fix the problem, 100% guaranteed; there is still often the problem in a complex environment about the impact that it will have on interoperation.

[Martin Grüning]

When I worked for a vendor selling multiprocessor UNIX systems a support contract for 3 years for a 4 socket system was 10.000+ Dollar. Support cost for the same service level on a 4 sockets x86 system is today less than 2,000 Dollar. That is a factor 5 decrease in service pricing. I don't say that your observation is wrong, but you shouldn't solely compare the service that you get but also the price that you pay.

[Martin G]

Okay, if I look at the support costs as a percentage spend based on my capital budget; they have not declined especially. If I look at the number of software calls that my teams raise these days; they have declined and we really only raise them when we are in trouble; so in fact my cost per call is at best constant but probably in all reality, it has increased.

I think that it is also true that I raise less hardware calls. And although there are more systems which are customer maintainable, we generally pay the uplift to have an engineer as our data centres are remote from the support staff and in some cases, they have oceans between them and the support staff.

The more mundane problems, we generally handle ourselves and obviously, there is no longer the calling up and requesting the latest fix pack; we can download those. This means that we actually need a better quality of support person on the end of the phone; without denigrating the people that we speak to, it often takes a lot longer to get to the expert and we are filtered through a number of levels before we get there.

Generally when we call, we want Batman...not his answering service.

[SR]

I agree with these, we have had almost exactly the same experience of a tech drone asking for 50 pieces of information completely unrelated to the trouble ticket in question.

The fact that his happened with a Tier 1 storage vendor is even more disheartening.

We have not experienced support costs going down. They are still at 8-12% of hardware costs, and have stayed that way for at least the past 7 years.

SR

[Support Engineer]

Wow. A bash vendor support article. Here's the real reasons behind the 3 questions as someone who has worked support by choice for a long time. I'm fully qualified to do many other jobs in IT, but I enjoy support.

Q1 - It's impossible for me to keep up with all the fixes that go in to a new release. How about we start out with the best foot forward and upgrade to rule out one of the ones I don't know about. Your change management restrictions are your own doing. If you pay us enough money or gripe loudly enough we might eventually find the specific bug before you upgrade. Guess what you pay for not taking us up on this? Hope you said a quick resolution if it was something already fixed in the latest version.

Q2 - Because no one would ever put in the logs something that you couldn't understand or you would never miss anything in the logs. It's a very arrogant response to this request from someone who wants to help you.

Q3 - There's a few reasons to request this. 1. The people who wrote the code that support interacts with often wants these questions answered. 2. Misconfiguration is a common cause of the problem you're having. But, of course you never misconfigure anything. 3. It helps to reproduce the problem in the lab exactly as you have to test fixing it before running you through a ton of changes on a production system that might not fix the problem.

[Martin G]

Okay, lets just take your first point about change management. You have obviously never worked in a large, complex environment; going to the latest release can potentially have huge knock-on impacts across a whole environment. For example, going to the latest level on a storage array can mean recertifying every server, SAN switch etc...Do you really think I am going to go through all that work if you can't guarantee that your fix will work?

Just send us the release notes if you can't be bother to read them yourself?

[Roger Luethy]

Hi Martin

Unfortunate you are right. The support sites are mostly a nightmare to navigate and to find the right information is a pure game of luck.

[Support Engineer]

I only work on large complex environments for over 10 years.

You're naive if you think any major company puts every change in the product in the release notes. Release notes are marketing materials, not software audit reports.

[Martin G]

Come back after you've been doing this for 20 years then but obviously 10 years is not long enough for you to understand good change management. Or perhaps, you've been working in support for too long and it's time to change path.

Release notes btw are part of the QA cycle for any release...all software should have good release notes which do document all the changes. You may not release those to the end-users but they should exist and if your internal developments are not briefing you properly, I'd suggest you speak to your management.

[I run a QA team as well as a storage team, we use release notes as part of our testing; I suspect yours do to]