Posted by Sten Westgard, MS
An interesting abstract was published at the Paris IFCC meeting. It detailed the EQA performance of a set of 12 public laboratories in Catalonia. Can you guess what the failure rate for these labs for biochemistry EQA?
- none of the above
The answer, after the jump...
As you might guess,the answer is actually all of the above:
- 14.5% of labs didn't meet the goals for imprecision for their biochemistry analytes
- 14% of labs didn't meet the goals for bias for their biochemistry analytes
- 3% of labs didn't meet the goals to allowable total error for their biochemistry analytes
The abstract notes that the bias failure rate may actually be low, because most analytes are not compared against a reference method, but instead are compared against their peer group mean. If the study used reference-assigned targets for bias and total allowable error, the failure rates may actually be higher.
Nevertheless, if more than one in seven laboratories is failing at imprecision, this is a big problem. Given the current debate on quality requirements, the evidence from this study seems to indicate that choosing separate bias and CV goals will actually only generate higher failure rates in laboratories. Are we really going to tell one out of every 7 labs to stop running tests? The Total Error approach, which provide a goal that combines the effects of imprecision and bias, provides a more practical grading of laboratory performance. However, even 3% failure is not something to be proud about.
(But don't worry, the Milan Mandate may tighten the biological variation goals to the point where the failure rate of laboratories is much, much higher.)
Let's put this performance in Six Sigma (short-term scale) terms:
- 14.5% of labs failing to meet imprecision goals: between 2.5 and 2.6 Sigma
- 14% of labs filing to meet bias goals: between 2.5 and 2.6 Sigma
- 3% of labs failing to meet allowable total error goals: 3.4 Sigma
Outside of healthcare, the Sigma-metrics for CV and bias would be considered dismal, unacceptable, and dangerous to patients. The Sigma-metric for allowable total error would be considered just above marginal acceptability.
What's even more interesting is another abstract performed by the same group concentrating on the "extra-analytical" quality indicators. That is, pre-analytical and post-analytical quality indicators and failures rates. Here's what that study found:
- 0.8% of serum samples were not received (missing) - a Sigma-metric (short-term) of 4.0
- 0.02% of serum samples were of insufficient volume - a Sigma-metric (short-term) of 5.1
- 0.75% of serum samples were hemolyzed - a Sigma-metric (short-term) of 4.0
What's the biggest source of errors in the total testing process again? I know the common wisdom is that pre-analytical errors are the most predominant. But if we look at the data from these two abstracts, the analytical errors are a much larger problem, particularly when we hold the performance to the standards of database of biologic variation ("Ricos Goals"). Any debate about tightening those goals, switching to more challenging goals, will mean that the analytical performance will become the dominant problem of the laboratory. We should bear in mind these realities and make sure that our goals are chosen carefully.
The two IFCC abstracts are
- Analytical Quality Indicators: Six-year evolution (2008-2013) of the Catalonian Health Institute Working Group, Perich C, Minchinella J, Llopi MA, Alvarez-funes V, Ruiz R, Ibarz M Bioasca C, Simon M, Busquests G, Llovel M, Montesinso M, Blanco-Font A, Martinez-Iribarren A, Serrat N, The Catalonian Health Institute Working Group on Quality Indicators, Spain
- Ten years experience working on preanalytical laboratory quality indicators. Working together for continuous improvement. Perich C, Minchinella J, Llopi MA, Alvarez-funes V, Ruiz R, Ibarz M Bioasca C, Simon M, Busquests G, Llovel M, Montesinso M, Blanco-Font A, Martinez-Iribarren A, Serrat N, The Catalonian Health Institute Working Group on Quality Indicators, Spain