Posted by Sten Westgard, MS
Earlier, we posted an article on the website with a darkly humorous take on the passing of the CLSI EP22 guideline, which voted itself out of existence in late 2010. Other websites have also noted its passing.
But it's worthwhile to take a moment to discuss, in all seriousness, where we are with Risk Information, Risk Management, "Equivalent QC", and the CLIA Final Rules. How did we get here? What drove us to this state? Where are we going next?
Recall, if you will, why there was even a need to develop EP22.
Start back in 1988 with the CLIA Final Rule, which lowered the bar on QC practices in the US with a temporary minimum standard of 2 controls, once per run for most types of laboratory tests. I say this was temporary, because in the details of the regulations, it was noted that there was supposed to be a provision for manufacturers QC clearance. This QC clearance was going to be a manufacturer-driven recommendation on the appropriate amount and frequency of QC for each instrument.
However, the development and implementation of this "manufacturer QC clearance" provision got postponed and postponed and postponed and postponed, until, in the "Final" CLIA Final Rule of 2003, the provision was deleted because CMS said it was no longer needed. In the absence of this provision, the minimum compliance standard of 2 controls, once per run (for most tests) took firm root in US laboratories. Soon enough, the minimal QC requirement in the US became the de facto standard for the rest of the world. The temporary provision became permanent practice.
During this time, another problem surfaced as well. Point-of-care devices had been given special exemption from many of the usual QC practices. "Electronic QC", which was not as comprehensive as "traditional" liquid QC practices, was also temporarily allowed. Again, CMS and CLIA were expected to come up with a provision that would reconcile the reduced QC practices of Point-of-Care devices with the more rigorous QC practices expected of the usual laboratory methods and instruments.
The Final CLIA Rule of 2003 provided 3 options for laboratory compliance with QC regulations. The 1st was to implement traditional statistical QC procedures that provided "immediate error detection," another new phrase in the Final Rule. Or, run 2 levels of control per day, the by-now ingrained minimal compliance approach. Or, and here there was a new term introduced, labs could perform "Equivalent Quality Testing".
However, when the State Operator's Manual and its Interpretative Guidelines were published the next year, yet another new term and concept was introduced: "Equivalent QC procedures" or EQC. Somehow "equivalent quality testing" was translated into this new concept of "Equivalent QC procedures." "Equivalent" was never defined. As part of "EQC", CMS identified 3 options and corresponding validation protocols to qualify an analytic system for each option...and more importantly, explained a new way that laboratories were going to be able to justify reducing their QC effort. These were the new EQC Options 1,2, and 3, and they were quickly rejected by not only laboratories, but also by manufacturers. Even CMS finally admitted, "we blew it." But even while acknowledging a complete lack of scientific basis behind "Equivalent QC", they nevertheless kept telling labs it was all right to keep doing these things.
In the face of EQC failures, a new effort was made in 2005 to jumpstart a new "Option 4" through a CLSI project to provide the missing scientific methodology for assessing the proper frequency of QC for different analytic systems. When this proved impossible to do - because of the absence of any established methodology in the scientific literature - CLSI revised the project goal to focus on the use of risk analysis for developing laboratory QC plans. Three different committees in the Evaluation Protocols (EP) area were assigned to provide guidance, first for manufacturer's risk analysis to identify failure modes (EP18), manufacturer's guidance for presentation of risk analysis information to the laboratory (EP22), and laboratory guidance on how to use the risk analysis information to develop a specific, customized QC plan (EP23).
Risk Management was a new approach for medical laboratories, but it has a long history of use in other industries (although the most recent experience of Risk Management in New Orleans, in the financial sector, in the oil spill of 2010, etc., Risk Management has shown itself to have serious flaws).
EP18 and EP23 were successful. EP18 is already an official guideline and EP23 will probably become official this year (I've heard summer of 2011). So why did EP22 stumble?
The problem is that manufacturers were worried that this EP22 Risk Information document, even while voluntary, could become a de facto standard on what they were supposed to disclose to customers. Furthermore, there was concern that
1. It would force manufacturers to list everything that could go wrong with their instruments, which is not something the folks in Marketing like to do. (true)
2. A comprehensive list of failure modes might in some way disclose proprietary advantages and be exploited by competitors. (possibly true).
3. A comprehensive list of all risks might weigh a ton and really no customer is going to read through all of it. (probably true)
4. If the comprehensive list of all risks didn't include a risk that eventually actually happened, this could be used against them in a legal sense. (The FDA actually did confirm this - that in retrospect, they might hold the Risk Information against manufacturers after the fact of an incident.)
Back in 2008, we speculated here that the requirements of EP22 made it unlikely that manufacturers were going to rush out and conform to this voluntary standard:
"In the absence of a legal mandate, I don't see why any manufacturer would willingly create this Risk information and distribute it to their customers. Any corporate lawyer will be advising against it (liability exposure). Any marketer will argue against it (if our competitors aren't saying they have residual risks, why do we need to do so?). Any executive will oppose it (why are we exposing our weaknesses to our competitors and our customers?). And that's the just the internal opposition this will face.
Certainly, I believe disclosure of Risk is the right thing to do, but the competitive and legal environment are not going to be receptive to it. Unless customers overwhelmingly demand this information, manufacturers will have no incentive - and many disincentives - to produce Risk information."
In retrospect, we got this prediction wrong. We were afraid that EP22 would be published as a guideline and then ignored. That is, EP22 would exist but no one would follow it. We didn't realize that manufacturers disliked this standard so much that they weren't going to let it get published at all. Even the existence of a standard of the Presentation of Manufacturers' Risk Information is evidently too much of a threat.
Again, as we noted in the more comical treatment of EP22's demise, this is not meant as a specific criticism of any individuals or members involved with the EP22 committee. Those committee members worked for years and years to try and develop a useful guideline, one that balanced the scientific requirements and the practical realities of the laboratory marketplace. In the end, they weren't able to find something that would make everyone happy.
Sometimes the problem with consensus and committees is, as that iconoclast John Seddon puts it, "in any consensus-building exercise, knowledge has no greater weight than opinion." [Systems thinking in the Public Sector, Triarchy Press, 2008, p178.] There is a belief that disclosing risk is too risky, which may or may not be true. It may be that if all manufacturers' disclosed their risks, the market would be better served and competition would not be any different (indeed, wouldn't it be novel to have instruments compete on the basis of their performance and quality, rather than on their marketing, branding and the sheer number of advertisements?)
So here we are back at the beginning. The manufacturers have played a long game, since 1992 until 2011, to avoid any type of requirement for instruction on the appropriate amount of QC for their methods. What's a customer to do?
As customers, remember, you still have the power to make demands. Until you sign that purchase order, you can make the diagnostic manufacturers dance to your tune.
Our advice: ask for that Risk Information, and ask for it to be presented in the format of EP22. Just because the committee abolished itself doesn't mean you've lost the right to ask for a presentation of risk information.
Ultimately, manufacturers respond to their customers. If enough labs demand this Risk Information, the diagnostic industry will have no choice but to deliver it.
Comments