In IT, what is a good measure of success?

Whether you adopt ITIL, MOF or CoBIT, eventually you’re going to come across KPI’s, or Key Performance Indicators.  The idea is that you can measure success by identifying indicators (quantitative, directional, financial) which are important to the organisation, and then use these to identify both opportunities to improve, and measure improvement over time.

And why not find a way to measure improvements so you can prove that the all the effort you’re putting in to changing your IT Services is working?  The problem is the assumptions on which most KPI’s are based.  The idea is that the indicator is (a) important to the organisation, and (b) measurable in some way.

I have two problems with that.

  1. ‘The Organisation’ isn’t a thing, it’s a concept, which means finding out what is important to the organisation is difficult at best.  You might be able to identify what’s important to the head of Finance, the person in charge of Estates, or your most senior Academic, but ‘The Organisation’ isn’t something that has an opinion.  Even if the Senior Management Team/Faculty Board/Council can reach an consensus on what is important, this doesn’t necessarily lead to the definition of meaningful indicators.
  2. There is a tendency to focus on the indicators that are easily measured, but these are only part of the picture.  Call response times, calls fixed within SLA, system downtime are all favourites of the IT Service Manager.  However, the effectiveness of IT service delivery is not measured by downtime alone, it’s measured by it’s impact on the effectiveness and efficiency of business processes. If users don’t engage with the systems and use all the functionality they practically can, then your IT Service isn’t performing, but no common quantitative measurement will detect that, unless there’s some serious analysis of business processes going on.

In my experience, there is no substitute for knowing your users.  By this I don’t mean personally knowing every single user (that’s a big ask) but at two levels; in theory and in practise.

The theory

This means understanding functionally what users do, what their role in the organisation is, how they interact with their colleagues and outside bodies.  Understanding their role means understanding their individual requirements, and understanding how users interact gives you the wider organisational requirements.

The practise

You have to have some personal connection with a proportion of your users, to give you a more complete understanding which theory cannot.  Many chose to use committees or user groups to connect with users, which are of limited value, because they’re almost exclusively made up of people who make up committee and user groups everywhere.  Those users are more vocal, inclined to push their point of view rather than their peers, and not representative of the majority of users.  Such groups have their place, but supplement them by interacting with users in their environment, to make sure you get a balanced view across the organisation.  This isn’t something one person does on their own, but something the whole IT department should do (or at least the user facing part).

If you can combine an understanding of how users interact with your systems, with the statistics of traditional KPI’s; then you might have an understanding of how your IT Services are performing.

Richard Bartlett

View more posts from this author

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.