The Westgard Rules

What Labs Really do..., Part Two

Posted by Sten Westgard, MS

Earlier this month, we posted about an interesting survey of laboratory quality control practices. The study was so interesting, it's worth revisiting the topic.

Just to review, here's the citation:

Audit of internal quality control practice and processes in the south-east of England and suggested regional standards. David Housley, Edward Kearney, Emma English, Natalie Smith, Teresa Teal, Janina Mazurkiewicz and Danielle B Freedman. Ann Clin Biochem 2008; 45: 135–139. DOI: 10.1258/acb.2007.007028 (subscription required)

What we'd like to point out in this post is what happened after the survey was completed. Based on the results, the authors of the study proposed a set of standards for quality control for the laboratories in their region:

1. IQC policies should cover 24 hour period.
2. Target values and ranges for IQC material should be assigned locally for each instrument. Manufacturer target levels and ranges should not be used.
3. Labs should use IQC material from a third party source.
4. Use EP5-A2 or equivalent to determine IQC values within each individual laboratory.
5. Encourage method specific rules (single and multi) based on the required quality of the assay, and known method bias and coefficient of variation.
6. Discourage inappropriate use of single rules, but when used ensure they have required degree of error detection.
7. Review grades of staff accepting and rejecting IQC
8. If IQC policies are robust, and the correct IQC with appropriate error detection is used, failed IQC should not be accepted (e.g. to continue processing patients' samples).
9. For analysers that have multiple modules, each module should be assessed individually with IQC on each assessment.

For frequent visitors and readers of Westgard Web, few of these proposed standards should come as a surprise.

Just to find out a bit more, we contacted the authors of this study to ask about the reaction and impact of their findings. Dr. David Housley was gracious enough to send this reply:

"The standards we propose are our audit group's official guideline and have been adopted as such for the region covered by our audit group.  Unfortunately, we have not done a re-audit yet, so we do not know how widely, if at all they have been adopted, or if they have influenced practice.  Anecdotally I think things such as changes to policy to cover 24 hours and changes to ensure all modules are tested have taken place.  One thing the audit / paper did achieve however, was an increased awareness of IQC and the need to review practice.  I am not aware that the ACB or our National Audit Group have proposed them as National standards.  If the other regions that have contacted me with an aim to undertaking their own audit complete their studies, I would imagine these standards (or similar) will progressively spread.

"In general people seem to fully appreciate the important concept of IQC, but fail to understand it sufficiently to make it work in practice.  I think the biggest problem in IQC is the belief that repeating the test is an appropriate course of action, and if the repeat result is within limits, it is safe to proceed and excludes an analytical performance issue.  I suspect that the single 2SD rule is largely the decision tool used for judging a repeat (or often multiple repeats) to be OK, and of course promotes the concept that the IQC is out for statistical reasons and not because of true performance issues."
[personal communication, 3/5/2009]

It's heartening to see such recommendations coming from an Audit and Quality Assurance Group. Somewhere in the world QC is still being taken seriously. If a similar survey of US laboratories were conducted and found similar results, I'm afraid our US regulators would generate the opposite recommendations: that IQC is too hard and thus, we need to dismantle IQC and replace it with something that the laboratories can more easily achieve (i.e. Equivocal QC).

What's New: March 2009

Posted by Sten Westgard, MS

Here's what's been posted on Westgard Web this month:

First, Do No Harm:
A Safety Net to Catch Analytic Errors
Risk Management standards are on the way. Some of the old QC techniques are going to be replaced by these new methods. That's all well and good, but in the end, you still need a way to catch those errors that slip through. Dr. Westgard explains how to balance the best of the old with the best of the new.

High Reliability Testing: Sensitivity to Operations
In our next lesson on High Reliability, we discuss the need for laboratories to treat "routine" tests as anything but routine. How does your laboratory handle the crush of data and detail in its daily operation - and still be sensitive to possible signs of failure?

Westgard Sigma Analysis: 4 Glucose methods
Tight Glycemic Control is one of the latest trends in laboratory medicine - one that requires fast, frequent, and precise testing. A recent paper studied four different glucose methods, one POC, 2 blood gas, and one central laboratory hexokinase method. Can you guess which methods performed the best, and what methods were good enough to support TGC?

We're also running a special on our updated online short course: "Westgard Rules" and Levey-Jennings charts. Order it through the online store http://www.westgardqc.com and use the coupon "RULESANDCHARTS" and you'll save $20 off the registration (our usual fine print: one per customer, per order, cannot be combined, online store only, ends 3/19/09) It takes about one business day to process your order and send you an email with your login and password.

If you need urgent access to the course, or just want to find out more and read the description, go to http://www.westgard.org/index.pl?op=show;isa=Program;iid=23868

Q&A: When will one rule do?

Posted by Sten Westgard, MS

A recent email question came in, asking about when it's best to use a single control rule and when it's better to use multirules:

"Our accreditation guidelines.... state the following:

VII.6.1

Internal QC rules shall be designed to detect random and systematic error.

VII.6.2

Multiple QC rules shall be used to decrease the probability of false

rejection....

"We have been "dinged" ... on a recent accreditation. The main reason being that when a result goes outside the [single rule] limit, we can't tell whether it is a random or systemic error without looking at the QC graphs.  I am not sure how any rule like the 1-3.5s for CK mentioned in the Feb 2009 article (Sigma metrics of Four Glucose Methods) on your website would be able to do this?"

Let's start by identifying two issues and addressing them separately. In the first place, from a compliance point of view, if the accreditation guidelines have specific written instructions to use multirules, you probably need to use multirules. (Obviously, these are not the rules affecting US laboratories, where single rules are the norm.) Arguing the scientific merits of a procedure with an inspector can have unexpected results.

Next, we need to address how the QC procedures are chosen for a laboratory. If a laboratory chooses a single or multirule approach for QC procedures based on tradition, or because they simply apply one set of rules for all tests, regardless of performance, that is one thing. On the other hand, if a laboratory assesses the performance and quality requirements of its methods, choosing QC procedures via a QC Design process, that's another thing entirely.

In the first scenario, where performance of test methods is unknown, multirules might be a better approach. The laboratory isn't adapting its QC procedures to fit the needs of the testing methods, so the additional information supplied by multirules may be valuable. It may also be overkill.

In the second scenario, QC Design has been used to determine the optimal QC procedure based on test performance and the quality required by the test. Thus, if QC Design has determined that the laboratory need only a single rule to QC a particular test, the additional information and detection power of a multirule is not necessary.

The email is correct that when a single-rule QC procedure is violated, for example, the 1_3s control rule, it is difficult to determine immediately whether or not that violation was a random or systematic error. A multirule would be able to help.

But in this example, where QC Design has determined a 1_3s rule is sufficient, that does not preclude you from applying multirules once an out-of-control rule has occurred. That is, you can use a 1_3s rule as your rejection rule, then apply multirules during trouble-shooting to determine the type of error. In some ways this is like the old 2s "warning rule" of the classic "Westgard Rules." The difference here is that the violation is not a warning rule, it's a rejection rule, and the additional rules are being used to determine what kind of error has occurred. (In the classic rules, the additional rules were used to determine if the run should be rejected, not why.)

Take a look at an example with a 5.0 Sigma method (10% TE_a, 1.8% imprecision, 1.0% inaccuracy). The first graph will show you the operating point of the method in comparison with the 1_3s rule:



Now, let's highlight a multirule approach:



Can you spot the difference? It doesn't look like a big difference. The operating point is below both operating lines of the QC procedures. There's  a bit more room under the multirule highlighted in the second chart.

Here's where a Sigma-metric / Critical-Error graph is more informative:

 

In this case, the multirule has approximately 3% false rejection and 100% error detection (it will detect the error within the first run that the error occurs), while the 1_3s has 1% false rejection and 99% error detection. Given that the rejection and detection numbers are determined experimentally and have a margin of +/- a percent or two, these procedures are almost the same. But the use of the multirule will generate some additional false rejections.

Bringing all of this discussion back to the concrete scenario of the question, I would show these graphs to the inspector (obviously, the graphs would have to illustrate the real tests), which should demonstrate the equivalence of a single rule and multirule QC procedure. I would further explain that a a multirule approach is used in trouble-shooting. That combination agrees with the spirit of the law, if perhaps not the letter.

What Labs Really do...

In a recent article in Annals of Clinical Biochemistry, a group over in the UK did an unusual thing - they audited their IQC (internal quality control) practices. The North Thames Audit and Quality Assurance Group use a questionnaire to assess the IQC practices in 54 laboratories in part of the United Kingdom. Here's the specific citation:

Audit of internal quality control practice and processes in the south-east of England and suggested regional standards. David Housley, Edward Kearney, Emma English, Natalie Smith, Teresa Teal, Janina Mazurkiewicz and Danielle B Freedman. Ann Clin Biochem 2008; 45: 135–139. DOI: 10.1258/acb.2007.007028 (subscription required)

The results are eye-opening. The questionnaire itself was comprehensive and lengthy, so we'll just highlight the most interesting parts.

1. Use of "Westgard Rules." When asked what rules they used for accepting / rejecting IQC (29 responses),  68.9% of the labs used "Westgard" rules of one form or another, while 20.6% used "2s" rules or even tighter limits.

This is an encouraging finding. The use of "2s" limits in laboratories is believed to be more prevalent, certainly in US laboratories.

But there is an important caveat to this survey finding. In a separate question, 22 labs responded that they did use "2s" rules for monitoring.  So it is not quite clear if the laboratories that use "2s" monitoring rules are using "Westgard Rules" for acceptance and rejection decisions.

2. Blanket application of one QC rule to all tests.  76% of the laboratories responding used the same IQC rules on all tests, while only one laboratory tailored their IQC efforts based on the needs of individual tests. This is somewhat disappointing. It means that the savings possible from QC optimization and customization (what we term QC Design) are being left on the table for many labs. But this finding is also not terribly surprising, either; laboratories usually lack the skills, tools, and time to commit themselves to QC Design.

3. Is it really out? Finally, the survey seems to have coined an unfortunate term, "non-ideal QC" - which is what occurs when "QC data seemingly showed that an assay had 'failed' to be appropriately controlled.... [ but the run is accepted anyway] .... in order to ensure work is completed." In other words, the control is "out" but the laboratory decides that the run is still "in." 62% of the laboratories said that non-ideal QC is accepted either rarely or never. The other 38% indicated, that it happened occasionally, once per quarter, once per week, 2-3 times a week, and one laboratory indicated that it happened daily.

It would be interesting to know if the same roughly 60% of laboratories using multirule QC are the same roughly 60% that either never or rarely accept non-ideal QC.

Even though this survey is from a specific region in the UK, the findings are relatively close to a private survey conducted by Bio-Rad and Westgard QC a few years ago. In that survey, a slightly higher proportion of laboratories used the "2s" rules alone, and slightly more of them responded to "non-ideal QC" by just repeating the control or taking other action to force the run back "in."

The big takeaway from this survey is that many laboratories remain stuck in the past when it comes to IQC practices. A lot of bad QC is resulting in even worse practices.  I'll have more in a following post on what these survey findings triggered in the UK.