Target practice: New Health System Indicators - What are the chances of hitting the mark this time?

The debate about how best to drive health system performance is over for now, resolved in favour of persuasion and mutual learning, instead of carrots and sticks

Every year during March, a very strange thing happens at my workplace, the University of Auckland. Outside, the sun is out, people on the street stroll by in short sleeves, shorts and jandals. Students would come to class in puffer jackets. They knew from experience that the classrooms were likely to be freezing.

For a few years, I would ring Property Services, the part of the university that is responsible for heating and air conditioning. I would tell them that the teaching rooms are cold. The response would always be – “the room is not cold, because our temperature gauges show that it is 21°C”. They were right. But the reason the students were freezing was because the air conditioning was on full blast to cool the building, resulting in significant wind chill. It’s a salutary story about indicators.

A few weeks ago, health minister Andrew Little announced the details of the Government’s Health System Indicators (HSI) framework at the RNZCGP conference.

This announcement was notable for finally giving Health Targets the last rites, while building on the more collaborative, locally driven approach that characterised the System Level Measures (SLM) framework first unveiled in 2016.

The debate about how best to drive health system performance is over for now, resolved in favour of persuasion and mutual learning, instead of carrots and sticks. However, any regime of health system indicators is only as good as the indicators themselves. And it’s a challenge to find indicators that are up to the job.

The first characteristic of a good indicator is that it needs to stand effectively for something that is valued. This sounds simple but it isn’t. Property Services’ decision to use 21°C to indicate comfort (the thing that is valuable) looks reasonable at first glance. But air temperature is not the sole contributor to comfort.

To put it another way, 21°C is an indicator of comfort, but it is not the indicator. The rate of participation in the bowel screening programme is an indicator of “improved wellbeing through prevention” but it is not and can never be the indicator (because no such indicator will ever exist). Very few indicators, if any, can represent the “whole thing” that is valued.

One of the major issues with the health target regime is that people mistake the indicator for the performance that is valued. As Katherine Collins from the Santa Fe Institute observes, “an indicator is a clue, not a conclusion”.

A second characteristic of a good indicator is the capacity to generate feedback that can lead to improvements. When the indicator improves or deteriorates, we need to know whether this is a consequence of something we did or didn’t do.

When I look at my step count average for the week and it’s trending downwards, I know why, and I know what to do to fix it. When the rate of acute hospital bed days goes down in Taranaki from one year to the next, it’s important to know whether this is because of changes in service design, or because the most recent winter just happened to be much milder.

If most fluctuations are dominated by factors outside the span of influence of health services, there is precious little usable feedback.

Unfortunately, there is a huge tension between these two characteristics of a good indicator. Indicators that stand for something meaningful are referred to as “tin-openers”. They are great devices for stimulating inquiry about important things, but the indicators themselves are also unpredictable and difficult to control. The tin that they open may turn out to be a can of worms.

By contrast, indicators that provide real-time feedback are easy to control but rarely stand adequately for something meaningful. These are “dials” just like the temperature measurements Property Services uses.

The tin-opener and dial analogy are the work of Neil Carter, Rudolf Klein and Patricia Day in their 1992 book How Organisations Measure Success: The Use of Performance Indicators in Government.

Most of the Health Targets were dials – health sector staff worked out how to calibrate them and make sure that targets were met, but the activities they engaged in to do so, often missed the point and did not result in real improvement. Overall, the new Health Systems Indicators contain more tin-openers than dials.

There is a Goldilocks zone, however, and at least one of the Health System Indicators inhabits it.

The child immunisation indicator effectively represents something that is valued, while providing timely feedback about performance. This indicator had a past life as a national Health Target, so it will be interesting to compare progress on this indicator over the next three to four years, to progress made from 2009–2014.

Such Goldilocks health system indicators, like planets, are rare. The last 12 years can be seen as a great testing ground for potential candidates. Apart from child immunisation, no health target indicator returned as a Health System Indicator. But nor did half the System Level Measures make the cut. Amenable mortality, in retrospect, was never going to work, particularly when DHBs had to wait two to three years for the data. But acute hospital bed days, and specific items from the primary care patient experience survey, live on in the new regime. In time, perhaps these will generate both timely feedback and meaningful information.

In the age of big data, where there is no frontier that is immune from measurement, health system indicators of some sort or other are here to stay. I can live with that, but nor do I have high expectations as to whether any regime of indicators will effectively “capture” performance.

We have other ways of evaluating how good (or not) our health system is, that do not require indicators and measurement. They are based on qualitative judgements, also far from perfect, but we must not stop using them.

Tim Tenbensel is associate professor, health policy, in the School of Population Health at the University of Auckland