The Westgard Rules

What's New: September 2013

Posted by Sten Westgard, MS

Total Analytic Error Concepts

Westgard and Westgard contributed an article to the September 2013 issue of Clinical Laboratory News. The article "Total Analytic Error: From Concept to Application" discusses not only the history of this concept but also some of the current debates on how "total" we should be when we talk about "total error."

CLN is a print magazine and had limited space. On the website, we have a bit more room, so we've posted an extended version of the article online.

Should we totally go to the max with Total Analytic Error?

---

Potassium Quality: Can anyone hit the mark?

Continuing with our series examining quality requirements and goals and the performance of methods, we turn to another electrolyte, potassium. We examine the different goals offered for potassium, and evaluate what methods are capable of achieving.

Can any method hit the target for Potassium?

---

Quality of Blood Glucose Meters?

Back in 2010, there was a study of 27 different blood glucose meters that assessed whether or not they were able to achieve the requirements of ISO 15197. More recently, the same authors expanded their study to assess 43 more blood glucose meters, and assess those meters not only against the older ISO 15197 standard, but also the proposed updated version. We review the data to see if newer meters mean better meters.

What's the latest quality assessment of glucose meters?

Avoiding Common Errors in Method Validation

Posted by Sten Westgard, MS

AACC is providing a free program on Method Validation, as part of their Thought Leadership series. This one is being conducted by our good friend and colleague, David Koch, who is a veteran and leader of the Method Evaluation workshops that are run each year at the annual AACC/ASCLS meeting. For those who can't attend the annual meeting, this is a great opportunity to hear David and learn about Method Validation.

David Koch, PhD, DABCC, Professor of Pathology and Laboratory Medicine at Emory University and Director, Clinical Chemistry, Grady Memorial Hospital & Emory University, Atlanta, Georgia

Click here to access the program.

5 Questions we still have about IQCPs

Posted by Sten Westgard, MS

1. Do all labs need to purchase the EP23 guideline, booklet, and worksheet?
   

   The CMS memo contains a curious Acknowledgement:
   
   "In light of advancements in technology, CMS has adopted a risk based approach termed Individualized Quality Control Plan (IQCP). the Clinical and Laboratory Standards Institute (CLSI) document EP23-A - "Laboratory Quality Control Based on Risk Management: Approved Guideline" provides helpful guidance to laboratories on the development of quality control plans for test systems. Portions of the EP23-A document capture the principles of our intended policies. So, as EP23-A is a copyrighted work of CLSI, we obtained CLSI's permission to utilize those portions of the CLSI EP23-A Guideline that captured our intended policies for IQCP under the equivalent quality testing provisioins of the CLIA program."
   
   One might read this and conclude that all the relevant EP23 material will be found directly in the regulations and State Operators Manual, thus it isn't necessary to purchase the EP23 guideline at all. Or perhaps CMS is just cribbing a few of the EP23 notes, and labs should still buy the CLSI documents.

2. Do you need to use any Risk Management tools to comply with EP23 and IQCP?
   Again, a very curious Q&A included in the memo:
   
   "Will laboratories be required to use a process map, fishbone diagrams, formal risk assessment charts and protocols, etc. in their IQCP?
   No, CLIA will not require the use of these tools in the development of an IQCP."
   
   This is strange indeed. CMS is rolling out a risk based approach for QC, but will not be requiring labs to use formal risk tools or protocols? What do you call Risk Assessment if it doesn't include the use of Risk Assessment protocols? Is EP23 a formal risk assessment protocol? If so, is that also unnecessary in the development of an IQCP (see question 1)
   
   The credibility of this new risk based QC approach is built on the good reputation and success that these risk assessment tools and protocols have achieved outside healthcare. The very reason Risk Management was considered a viable approach was the existence, indeed the wealth, of tools and protocols that were available. To adopt a risk approach without any of the tools and protocols is, frankly, a little bizarre .  


3. Will Deemed Accreditation Organizations have to implement IQCPs?
   From the Question and Answer section of the Memo:
   
   "Will accrediting organizations (AO) and exempt states (ES) be required to accept the use of the IQCP option?
   It will be optional for AOs and ESs to incorporate IQCP into their standards; however, any standards they use will need toeither be a regulatory equivalent tothe current CLIA regulations, or an acceptable equivalent tothe IQCP option as laid out in Attachment 1-IQCP."
   
   In other words, all organizations will be forced by CMS to accept IQCPs. An AO has the option to say Yes, or it can lose its deemed status. This is what we saw with EQC. AOs initially resisted, but eventually caved and implemented a version of EQC.


4. Will inspectors be able to assess and even challenge the Risk QC of a laboratory?
   Again, from the Question and Answer section of the Memo:
   
   "What if the surveyor doesn't agree with the way the LD has addressed the risks in the QCP?
   
   The surveyors will be trained to determine if a risk assessment was performed, if the identified risks were evaluated, if the QCP includes any risk(s) that the laboratory director has determined needs to be mitigated, and that quality assessment is occurring and ongoing."
   
   The short answer we're looking for: An inspector is empowered to assess and challenge the laboratory's IQCP. The answer here seems a bit muddled and unclear. If an inspector is not allowed to point out failure modes that the laboratory director may have missed in the risk assessment, or is not allowed challenge the evaluation of those risks (i.e. the laboratory director thinks the risk is acceptable or has been mitigated, but the inspector disagrees), then the IQCP may become a toothless regulation. If inspectors cannot challenge IQCP implementation, then any IQCP implementation will be in compliance (and laboratory directors will not be strongly motivated to identify risks or determine any risks need mitigation).


5. Will labs need to develop "re-testing" plans?
   The language of the regulations on "Quality Assessment" contains some intriguing requirements about what is necessary if a QCP fails:
   
   "All IQCP Quality Assessment monitoring must be part of the laboratory’s overall Quality Assessment plan. The laboratory must establish a review system for the ongoing monitoring of the effectiveness of their IQCP.... "When the laboratory discovers a testing process failure, the laboratory must conduct an investigation to identify the cause of the failure, its impact on patient care, and make appropriate modifications to their QCP, as applicable. The investigation must include documentation of all corrections, corresponding corrective actions for all patients affected by the testing process failure, and evaluation of the effectiveness of the corrective action(s). The laboratory must implement the correction(s) and corresponding corrective action(s) necessary to resolve the failure and reduce the risk of recurrence of the failure in the future. If necessary, the laboratory must update the risk assessment with the new information and modify the QCP, as needed.
   "Probes §493.1256(d).
   "Has the laboratory established a review system for the ongoing monitoring of the QCP...and evaluation of its effectiveness?....
   "In the event of a testing process failure, has the laboratory evaluated all patient test results since the last acceptable quality control?"
   
   Perhaps I'm reading too much into these paragraphs, but this seems like a higher bar than what was required for EQC. The laboratory needs to have a real system in place to monitor and if necessary retest the specimens, should the QCP reveal an out-of-control signal. By default, that might limit the frequency to the capacity of the lab to store specimens.

There are more questions one could raise, and we hope will be raised, as the new IQCP regulations are rolled out. EQC failed because it wasn't introduced properly to the laboratory community. EQC also failed because it was created without a proper scientific foundation. Let's hope the IQCP regulations have identified, addressed, and mitigated those risks.

5 Things we Know Now about IQCPs

Posted by Sten Westgard, MS

On August 16th of this year, CMS issued a new memo about IQCPs. If you recall, IQCP stands for  Individualized Quality Control Plan, and it's part of the new Risk QC being proposed by CMS as a replacement for EQC. The CLSI guideline EP23, which came out about two years ago, laid out the broad outlines of this new policy, but we have been waiting for the government regulations to spell out the specifics of implementation and interpretation.

So here are five key things we know now:

1. It's really coming. The CMS memo lays out a timeline that explains a 2-year process of phasing out EQC and replacing it with IQCP.
   • Week of September 15, 2013: IQCP Training of RO CLIA surveyors
   • Week of November 18, 2013: IQCP training of SA CLIA surveyors
   • January 1st, 2014: IQCP Education and Transition Period Begins
   • January 1st, 2016: IQCP Education and Transition Period Ends [EQC expires]
     
2. We still don't know all the details and it's not really here yet. Obviously, we've got more than two years ahead of us before all labs must switch from EQC to IQCP. The State Operations Manual (SOM), which contained all the devilish details for EQC, has not yet been updated to contain the new devilish details for IQCP. So while the memo lists more details than we've seen to date, the IQCP remained to be fully fleshed out.


3. There's no grandfathering. Unlike the "Final" CLIA Final Rule, no method gets an EQC "pass" [when the Final Rule came into effect, older methods didn't need to go through method validation; only newer methods did]. Everything using EQC will have to be re-established through Risk Assessment for IQCPs. There will not be a simple switch where a group of POC devices that were chugging along with EQC will get to slap an "IQCP" label on top of their current EQC SOP and keep operating the same way. Every device that was using EQC will need to go through a Risk Assessment process, before it can develop an IQCP which may or may not allow reduced QC frequency. Here's the quotation from the memo:
   

   "We do not anticipate grandfathering test systems under the current EQC policies and procedures. However, historical data accumulated during the EQC evaluation protocol and its ongoing usage could (and likely should) be used in the development of the laboratory's IQCP."

4. Manufacturers will not be able to do this for you. Call this the "EP22 - Catch 22." When EP23 and Risk QC was originally proposed, an accompanying guidline EP22 was supposed to be developed, which would allow manufacturers to develop a standard way of presenting Risk information. That would then allow labs to take that information and use it in their Risk Assessments. In other words, EP22 handed off to EP23 and the whole thing was going to be easier. Unfortunately, along the way, EP22 committed suicide by committee, so EP23 was left all alone. Now, CMS appears to be saying that laboratories, and specifically laboratory managers, will not be allowed to simply take a manufacturer's "Risk Assessment" document and sign off on it.Here's the key quote:
   

   "Note: Manufacturer-provided tools and templates, if available, may be helpful for laboratories implementing IQCP; however, laboratories will need to supplement these materials with laboratory-specific information as part of the Risk Assessment. The manufacturer information is not sufficient in and of itself." [In other words, you can't just accept a manufacturer's "Our Device is Low Risk" brochure as gospel. You need to actively make a Risk Assessment of the device when it is operating in your own laboratory.]

5. The Risk of Risk QC is all on the Laboratory Director. As with EQC, the CMS regulations are quick to point out that the practice of reduced QC may be compliant, but it may not be clinically appropriate or even legal:
   

   "The laboratory director must consider the laboratory's clinical and legal responsibility for providing accurate and reliable patient test results (§493.1407 or §493.1445) prior to implementing a QCP that is less stringent than the specified Analytic Systems control regulations..."

It's a strange world we find ourselves in, a bit new (Risk) and a bit familiar (some of the same EQC warnings and weaknesses). That particular provision sounds like a warning: You can implement Risk QC and be in compliance with CMS, but you may not pass a clinical or legal challenge.

We'll know more soon as the regulations slowly dribble out over the rest of the year. This memo answered some questions, but raised others.

Pop Quiz: If labs want to reach quality, do they need MAPS?

Posted by Sten Westgard, MS

In a recent issue of CCLM, an interesting opinion paper reported on a pilot study of the quality of UK laboratories.

Given 5,812 QC data points on 5 different platforms in 9 different laboratories measured over 6 months, and a quality goal of 7.0%  how many of those laboratories do you think achieved 5-Sigma quality?

• 100%
• 75%
• 50%
• 25%
• less than 25%

The answer, after the jump...

Westgard and Westgard at the Lab Quality ConFab

Posted by Sten Westgard, MS

This Tuesday, October 1st, at the Laboratory Quality ConFab, Dr. Westgard will deliver a plenary session:

“Elevating the Analytical Integrity of Your Lab's Test Results: Why Your QA/QC May Be Sub-optimal and How to Find and Fix those Problems ”

Later, in the afternoon, there will be a special session with both Westgards:

"Rethinking Traditional Lab QA/QC: Understanding Weaknesses in Existing Requirements and Mastering Useful Methods and Metrics to Raise Your Lab's Analytical Accuracy."

You can register for the Lab Quality ConFab here

Could your laboratory catch this error? A Coda

Posted by Sten Westgard, MS

Readers may recall an earlier lesson we posted in June titled, Would your laboratory have caught this error?

It generated quite a bit of reader interest, but coincidentally, a recent article published in Clinical Chemistry has reported that the laboratory in question was not alone:

Clin Chem. 2013 Aug;59(8):1187-94. Failure of Current Laboratory Protocols to Detect Lot-to-Lot Reagent Differences: Findings and Possible Solutions. Algeciras-Schimnich A, Bruns DE, Boyd JC, Bryant SC, La Fortune KA, Grebe SK.

As it turns out, the Mayo Clinic and the University of Virginia Medical Center were experiencing similar problems with the same method. Mayo Clinic estimated that if 1,000 patients were being tested per month, they were possibly generating 50 false-positive results, causing extra testing, care, delay, confusion, and anxiety. In an era where health systems are seeking and try to eliminate waste, having a test that causes such problems is a cardinal sin. It appears this method error persisted for multiple years. That's a lot of false positives and a lot of financial cost to a health system, even if it's not easy to tally up on a balance sheet.

What's more concerning and interesting about the recent study is that the usual lot-to-lot reagent tests were not able to catch the growing problem. The typical approach, to test a few patient samples on both old and new reagent lots, was not able to detect the problem - the study authors estimate they would have needed hundreds of patient specimens in a crossover study in order to be able to detect the problem. Which is not practical, even for large volume laboratories.

Ultimately, what we see is that the problems get much harder to handle once they reach the customer (the lab) - it would be far easier if the manufacturer was doing a better job in the first place: monitoring the method quality and the reagent quality. And if the manufacturer wasn't monitoring this, then some type of external quality assurance group needs to be able to detect these problems (possibly with accuracy-based samples).

I guess the obstacle to getting manufacturers to monitor their quality more closely is that it becomes a self-selecting problem: the manufacturers wih the worst quality are the manufacturers least likely to want to monitor their own quality or talk about quality to their customers. Somehow, the manufacturers need a nudge in the correct direction.

 

Total Analytic Error discussed in Clin Lab News

Posted by Sten Westgard, MS

The September issue of Clinical Laboratory News has an article by Westgard and Westgard on Total Analytic Error: From Concept to Application. 

We also have a longer version of the article on the website.

There has been a debate over the scope of error that we should be trying to encompass and estimate. "Total Error" is usually a shorthand for "Total Analytic Error" in the laboratory. For those trying to understand the different kinds of "Total Error" that could be estimated, this article is a good review.