Audit & benchmarking and how to document, document & document

The Healthcare Commission has produced this report into Acute Mental Health In-patient Care. There’s a HCC press release here.

While the Telegraph highlights:

Thousands of mental health patients go missing from wards

The sectioned patients either escaped from wards, went missing during authorised leave or failed to return to hospital when they should have, a critical report by the Healthcare Commission found.

During six months in 2007, patients detained under the Mental Health Act went missing on 2,745 occasions for a total of 8,870 nights

The Independent writes specifically about how unsafe such wards are.

Despite increased spending of £1.2bn in real terms on adult mental health services since 2002, one in four of England’s 10,000 mental hospital beds is in a trust rated as “weak”, which, says the commission, “does not meet the minimum requirements and reasonable expectations of patients and public”.

There was previous discussion (originally discussing the waste of the ology and iatry in mental health) which developed into suggesting we should be focusing more on the actual wards rather than splitting neurons; but I’m not about to revive it.

So, back to the report. What does it tell us? Apparently it reports that:

Overall, eight trusts were rated as “excellent” (accounting for 843 beds – 9%), 20 as “good” (2,808 beds – 28%), 30 as “fair” (3,985 beds – 40%) and 11 as “weak” (2,249 beds – 23%).

Hmm… so what does this mean?

On the four key criteria against which we assessed performance, our findings were:

• No trust was scored excellent on all four of the key criteria, suggesting there is room for improvement for all service providers.
• Almost two-fifths (39%) were scored weak on involving service users and carers – this was the area with the highest proportion of weak scores.
• Around one in every nine trusts was scored weak on the criteria for providing individualised whole person care and for ensuring safety.
• No trust was scored excellent for the effectiveness of the acute care pathway, although fewer trusts were scored weak here compared with the other three criteria.

Rather than discuss the results of this report, I’d like to talk about “reports”. How do they do them and what do they really mean?

Well for this one…

[w]e used a combination of national and bespoke data as part of a rigorous assessment process, and our findings fed into our annual health check of trusts’ performance.

So of course, I had to look up ‘bespoke data analysis‘ - and took a look at some of the indicators they used - and decided I’d just leave it there and let someone else write about it instead.

But its come at an interesting time for me because, all the way down here - were going through our Accreditation process.

As an aside, the Aussies were a bit smarter than the UK (sorry to tell you OSB!) and when it came to evaluating services, the Australian Medical Association and Australian Hospitals Association jumped in with self-regulatory policy before federal government did [1]. This means they evaluate what they choose to, how they choose to, in the manner they choose to.

The ACHS proudly states on it’s website:

What is accreditation?

Accreditation is the status obtained by an organisation after a successful third party external evaluation by a recognised body to assess whether an organisation meets applicable pre-determined and published standards

And declares:

Is the ACHS accreditation process similar to an audit?

No. The focus of ACHS accreditation programs is to provide a framework for continuous improvement. ACHS accreditation is not quality assurance, or ticking the boxes and staying at the same level of performance. It is about establishing a structure and processes that allow quality and safety to proliferate

So there you have it; it’s a ticking box audit.

Which explains why everyone at work is frantically ensuring every patient’s file has a sticky label on it and that all errors are crossed through and initialed by the author and the care plans are signed by staff and patient.

It does go further into detail (probably using a bespoke data analysis machine) on some aspects but essentially, as long as it says you did it - it means you did it - and THAT’S A QUALITY IMPROVEMENT!

There’s nothing wrong with standards but two things that always bothered me with benchmarking, audit, evaluation and quality improvement is - why do we benchmark the same things every year if this is a quality improvement thing? And why do we only do it 3 months before we’re (not) being audited and what use is it when, for the most part, it’s faked?

I’d guess a fair bit of this UK HCC report was ‘faked’ too.

Table 21: Bespoke data collection submission & return rates

Questionnaire for acute inpatient leads: 99%

Questionnaire for ward managers
(completed for each relevant ward): 99%

Questionnaire for service user groups
(completed for each relevant ward) : 100%

Audit of care records (completed for a
sample of 50 care records): 100%

I’d also guess it was pretty important, eh? Pity the standards most failed to achieve weren’t as important.

We assessed all of the 69 NHS trusts that provided mental health acute inpatient services during 2006/2007. These trusts registered 554 acute mental health wards within the scope of the review, providing a total of 9,885 beds.

So why the title document, document document?

Well, most benchmarking indicators rely on not seeing what is done or the assessing its quality - but looking through the records to see if it says it was done. In this review, there were several thousand clinical files audited. So if you want to improve your scores - be sure to spend even more time in the office documenting thoroughly and in the right way just exactly what is was you did today; though I believe such audits generally tell us we’re good at administration - rather than mental health care.

By the way - the report actually concludes - inner city care is of a lesser quality than rural.

I wonder how much it cost for that report? And why didn’t anyone listen to the last 20 reports on inadequate inner city care standards?

[1] Palmer, G. & Short, D (2000) Health Care & Public Policy 3rd Ed. Macmillan Pub.