Why bother with validation?

Posted by Sten Westgard, MS

I came across an interesting abstract at the AACC/ASCLS conference in July. For reasons that will soon become clear, I don't want to identify the institution or the instrument. But here are some excerpts:

"Results: [EQA agency] precision goals were not met for [analyte 1], [analyte 2] and [analyte 3]. The manufacturer's precision claims were not met for [analyte 4], [analyte 5], and [analyte 6]. Correlations with [another instrument] and [yet another instrument] were acceptable and reference range transference indicated no reference range changes are required."

What do you think the verdict was?

Well, they decided to go with it:

"Despite the imprecision for [analyte 1], [analyte 2], [analyte 3], [analyte 4], [analyte 5], and [analyte 6], the [new instruments] will be the new...analyzers used by the Core Laboratories at [healthcare system]."

My first read of this was a little breath-taking. Recall the purpose of method validation is to assess the performance of methods, and the instruments in question seem to have failed more than a few of the requirements. Nevertheless, the institution decided to accept the methods and instrument. It seems like the decision was made regardless of data and facts. That would make the method validation studies little more than a compliance exercise: the laboratory was required to generate some graphs and crunch some numbers, so they did that (but they didn't actually make any practical use of the results).

My second thought, however, was more measured. Why would this institution accept instruments that had "failed" the validation studies? Perhaps, I wondered, the goals and requirements are at fault, not the performance. If we were holding the instruments to a performance standard that was too hard, we could get the same outcome. If more realistic or appropriate goals had been applied, would the instruments have achieved better (and more acceptable) performance? The first three analytes failed the goals imposed by an EQA agency or regulatory authority. Those goals may not have been the most evidence-based goals. More troubling, however, is when the instrument doesn't achieve the level of performance specified by the manufacturer. Still, if the manufacturer has chosen an inappropriate goal, that may not matter. Still, what matters most is, what is the medically appropriate requirement for quality - and does the new method achieve that performance?

It's a question worth answering: when we go through method validation, are we choosing the right goals?

 

Comments

[Kathleen Freeman]

I think this is a great question…..in my opinion, we need to keep going back to the clinical use of the results to determine if the quality requirement/standard is appropriate.

In working with clinicians (and other pathologists) with TEa/quality requirements, I've found that they often have unrealistic goals based on how they'd LIKE to interpret a test….so some education and compromise is needed to let them know what the current state-of-the-art is with regard to instrument performance.

Likewise, manufacturer's need to pay attention to what specifications the user requires and try to get the instrumentation performance in the needed range!

[Hassan Bayat]

Of course choosing “right goals” is a critical issue.
- Any goal is chosen, it must be clear for both sides: laboratories and physicians. Physicians must be aware of the TEa the laboratory is working within, and also be informed about the number of the results that are in the chosen TEa. For example, if a laboratory’s TEa for glucose is 10% and the laboratory’s sigma is 2 (that unfortunately accepted by current rules!), it means that 5% of glucose results are more than 10% away from the correct value. If a physician knows this, then s/he will be able to make better decisions about the patients. But it seems that a lot of physicians aren’t informed of quality goals and sigma number of the laboratories’ performance, and this makes them, as Sten Westgard says, think of the laboratory results as an absolute vale an not a probable value.
- Clinical guidelines must address, and be based on, the required quality goals and needed sigma number. Only the laboratory results that are produced complying the needed TEa and sigma, could be interpreted the way that a guideline says.
- “The 1999 Stockholm consensus hierarchy" addresses the way to choose quality goals. But:
 It is not clear that this statement talks to whom: laboratories or authority organizers? (Of course laboratories have to refer to rules that authority organizations establish.)
 Using “the behavior of the physicians” as a source for establishing quality requirements can result in a loop. The physicians’ behavior is based on the current quality of the assays, at least to some degree. Therefore, the physicians make their mind based on the quality of our performance, and then behave based on that idea, and then we consider the physicians’ behavior as quality requirements for our performance. In short, this means accepting the current performance as the best performance!