Posted by Sten Westgard, MS
A recent op-ed in the New York Times by Dr. Robert Wachter as well as an interview with Don Berwick in HealthLeaders Media broached a taboo topic: is healthcare measuring too much?
'[T]he measurement fad has spun out of control. There are so many different hospital ratings that more than 1,600 medical centers can now lay claim to being included on a “top 100,” “honor roll,” grade “A” or “best” hospitals list. Burnout rates for doctors top 50 percent, far higher than other professions. A 2013 study found that the electronic health record was a dominant culprit. Another 2013 study found that emergency room doctors clicked a mouse 4,000 times during a 10-hour shift. The computer systems have become the dark force behind quality measures.'
How Measurement Fails Doctors and Teachers, New York Times, January 16, 2016
As the global debate over establishing the best error models and performance specifications rages, laboratories are probably asking themselves the same questions that the doctors are asking: how many metrics are too much?
A possible answer, or at least some more questions, after the jump...
Today's laboratories run their data through a gauntlet of statistics: measurement uncertainty, total error, Sigma-metrics, ABC score, correlation coefficient, comparisons of individual components of error like imprecision and bias to specific goals, and we're just embarking on the new era of "Risk QC", where we may have a multi-component semi-quantitative assessment of test risk. Is there an upper limit on the number of quality metrics for testing?
Dr. Berwick believes that medicine has gone overboard on measurements:
"Over the past 20 years, as evidence grew about defects in care, there was sense of alarm. The reaction was to try to turn the lights on, to increase knowledge about the performance of healthcare in many, many dimensions for many people.
"As a result, we began a festival of measurement, an almost measurement mania, where we began to believe that the solution to performance was transparency and measurement. I'm a complete fan of transparency, but we've overshot.
"Now, the number of metrics exceeds the ability of any reasonable human being to consume usefully. And, there has been insufficient diligence about the alignment and harmonization of measures."
So how do we reign in the measurements, narrow down the metrics to the most useful?
We fall somewhere in the middle of this debate. We don't want to abolish the statistics that have been developed over the last few decades, nor do we believe that we should continue to add more and more statistics without removing older measurements. As quality professionals, we advocate that measurements for basic QC (control data, LJ charts) remain critical, and that some summary measurements are better and more practical (Sigma-metrics). But we also agree that the lab suffers from measurement overkill. You don't need to calculate your Sigma-metric with every new data point. You don't need to calculate critical systematic error, total error and the Sigma-metric for each and every test. You don't need to compare the individual components of imprecision and bias to desirable specifications if you've adopted a total error approach. And realistically you probably don't add value when you report the measurement uncertainty of each analyte in test report you provide to clinicians (You HAVE to calculate measurement uncertainty as part of ISO 15189, but what you do with it after that remains your choice).
Dr. Berwick suggests winnowing out the statistics by cutting the number of hospital measurements by 50%:
"When we are trying to measure something thing four or five different ways—stop and measure it one way. With each of the metrics we're using, subject them to a test. Are the results of the measurement used by anyone? If we are doing measurement and recording data that no one uses, stop it, because there's no action being taken. It can't be useful."
As we have seen in our global MU survey and comments, the MU statistic is one of the most calculated but least used metrics. A high number of the statistics found in EQA and PT reports are probably equally under-utilized. To be fair, however, there is a case to be made that we haven't sufficiently connected Sigma-metrics, or total error, or any of the laboratory quality models to the patient outcomes. Has the adoption of any particular error model changed the laboratory in such a way that better patient care can be measured? Or are all the laboratory quality experts merely engaging in a debate that generates papers and plenaries but no progress?
In the narrow "MU vs. Total Error" debate, we have what economists call a "natural experiment" taking place. US labs almost completely ignore MU in their operations. Other countries have made ISO 15189 and its mandatory MU the backbone of their laboratory regulations. Are there any differences in patient outcomes in the US and other MU-countries that can be attributed to the use of one approach over the other? Are more patients dying because US labs use total error? Are labs choosing different instruments outside the US because of MU? Are there any differences at all between US labs that use total error and European labs that use MU? If there are no differences, we may be arguing over nothing. If a bad instrument can be marketed and sold in both regions of the world, our debate may actually be delaying improvements in instrumentation.
Dr. Berwick has also centered on the larger problem of measurement in healthcare - it is primarily focused on the financial health of the hospital, not the patient:
"My plea is to take the spotlight off finance and profit as the primary responsibility or activity of senior leaders because I believe we will never solve the problem of cost and finance by focusing cost and finance. We're going to have to solve that problem by focusing on the design and redesign of healthcare and the improvement of its quality."
Laboratory leadership is similarly distorted, often focused on the sticker price of an instrument, while being blind to the long-term costs of purchasing a collection of "cheap-but-bad" methods. Laboratories that invest in high quality save money for the entire organization, not just their own departmental budget. Laboratories that go with the lowest bid often hemorrhage their errors into the patient care pathway. Saving pennies on reagent can end up costing thousands in patient care.
Dr. Wachter ends his op-ed by quoting the father of quality measurement, Dr. Donabedian:
"Avedis Donabedian, a professor at the University of Michigan’s School of Public Health, was a towering figure in the field of quality measurement. He developed what is known as Donabedian’s triad, which states that quality can be measured by looking at outcomes (how the subjects fared), processes (what was done) and structures (how the work was organized). In 2000, shortly before he died, he was asked about his view of quality. What this hard-nosed scientist answered is shocking at first, then somehow seems obvious.
"'The secret of quality is love,' he said."
The secret of quality metrics is not to fall in love with the measurements and statistics themselves, but to remember that they are only a means to a more important end. It's all about our patients and our loved ones.
Comments