Posted by Sten Westgard, MS
I came across an interesting abstract at the AACC/ASCLS conference in July. For reasons that will soon become clear, I don't want to identify the institution or the instrument. But here are some excerpts:
"Results: [EQA agency] precision goals were not met for [analyte 1], [analyte 2] and [analyte 3]. The manufacturer's precision claims were not met for [analyte 4], [analyte 5], and [analyte 6]. Correlations with [another instrument] and [yet another instrument] were acceptable and reference range transference indicated no reference range changes are required."
What do you think the verdict was?
"Despite the imprecision for [analyte 1], [analyte 2], [analyte 3], [analyte 4], [analyte 5], and [analyte 6], the [new instruments] will be the new...analyzers used by the Core Laboratories at [healthcare system]."
My first read of this was a little breath-taking. Recall the purpose of method validation is to assess the performance of methods, and the instruments in question seem to have failed more than a few of the requirements. Nevertheless, the institution decided to accept the methods and instrument. It seems like the decision was made regardless of data and facts. That would make the method validation studies little more than a compliance exercise: the laboratory was required to generate some graphs and crunch some numbers, so they did that (but they didn't actually make any practical use of the results).
My second thought, however, was more measured. Why would this institution accept instruments that had "failed" the validation studies? Perhaps, I wondered, the goals and requirements are at fault, not the performance. If we were holding the instruments to a performance standard that was too hard, we could get the same outcome. If more realistic or appropriate goals had been applied, would the instruments have achieved better (and more acceptable) performance? The first three analytes failed the goals imposed by an EQA agency or regulatory authority. Those goals may not have been the most evidence-based goals. More troubling, however, is when the instrument doesn't achieve the level of performance specified by the manufacturer. Still, if the manufacturer has chosen an inappropriate goal, that may not matter. Still, what matters most is, what is the medically appropriate requirement for quality - and does the new method achieve that performance?
It's a question worth answering: when we go through method validation, are we choosing the right goals?