(Or, if only some surveys are based on accuracy, then what are the other surveys based on?)
Posted by Sten Westgard, MS
There's an article that appeared in the October 2010 issue of CAP Today that probably didn't get enough attention. It covers a subject that's been gnawing at us for a while:
Accuracy-based Surveys carve higher QA Profile, by Anne Paxton
For those of you who thought all proficiency testing was "accuracy-based", this article may give you a bit of a shock. In fact, most PT surveys - indeed most EQA programs and even peer-group programs - are not based in accuracy. Instead, those surveys are only based on "consensus."
What's the difference, What does it mean - and how did it come to be this way?
The practical realities of creating a proficiency testing survey (or EQA scheme) often mean that the material or specimen used is one that is not commutable. Thus, the material has been modified in some way that means it no longer acts like a patient specimen. In short, there are matrix issues - and methods that react in one way to this PT specimen may not act in the same way to a real patient specimen.
As the article notes, "... [W]hen these [non-commutable] materials are used for proficiency testing, the laboratory is assessed only in terms of its ability to match the peer group. 'Theoretically, all of the labs' results could be wrong, or all of them could be right; due to the artifacts of these matrix-related biases, there's not enough information to answer the question of accuracy.'[W. Gregory Miller, PhD, director of clinical chemistry at Virginia Commonwealth University, Richmond]"
As a result of non-commutable PT specimens, the PT programs no longer answer the question,
- "Can your laboratory get the right answer?",
and instead focus on another question,
- "Can your laboratory get the same answer as other instruments?"
As you can guess, there's a big difference between those two questions. But because of the CLIA regulations, there is no compliance difference between participating in an Accuracy-Based Survey and a Consensus-Based Survey. As long as you participate in some PT program, you've fulfilled the minimum requirements. Accuracy isn't required, and since consensus is usually cheaper, that's become the dominant form of PT program.
The real problem with consensus-based surveys is that the results are too easy to dismiss. If a laboratory gets PT results back that show they are significantly different than the group mean, they have two rationalizations.
- First, they can dismiss the group mean as insignificant, since that value is not connected to any "true" value.
- Second, they can dismiss their difference as some sort of "matrix effect".
Thus, when problems seem to be indicated by a consensus-based survey results, labs can tell themselves there isn't a problem at all. And manufacturers have been equally slow to respond to any biases reported in PT surveys, since those differences are commonly matrix effects.
CAP's growing list of Accuracy-Based Surveys promises to reveal the "true" biases among test methods, something that may force manufacturers to address performance issues.
Another way of looking at this issue: this is the retail end of Traceability. While there are committees of metrologists talking about the importance of standardization, harmonization and traceability, their discussions can seem a little abstract. Accuracy-Based Surveys offer labs a chance to quantify the biases affecting their own methods - the real-world effects of a lack of standardization. And ultimately, it will be pressure from laboratories - spurred by these uncovered biases found in Accuracy-Based Surveys - that drive manufacturers to make real improvements in their methods.
Comments