The Westgard Rules

New Paper: Risk-Based QC Strategies with Six Sigma Nomograms

Posted by Sten Westgard, MS

Proud to announce the publication of a new article in the latest issue (Volume 2, Issue 2) of the Journal of Applied Laboratory Medicine:

Planning Risk-Based Statistical Quality Control Strategies: Graphical Tools to Support the New Clinical and Laboratory Standards Institute C24-Ed4 Guidance

Hassan Bayat, Sten A. Westgard, James O. Westgard

DOI: 10.1373/jalm.2017.023192 Published August 2017

 

There's a new tool in town: the Six Sigma QC Frequency Nomogram.

 

Based on modeling of Curt Parvin, and extension of the work of Yago and Alcover, these Nomograms will allow laboratories to finally develop fully-quantified QC frequencies (or runs). As more of these papers are published, we will make more of these Nomograms available through the website.

 

Sigma-metrics now allows you to assess quality, choose the right number of controls, choose the right number of rules, AND now, choose the right frequency of running those controls.

 

That's a low of power packed into a useful metric.

Multisite Analysis of AST performance and goals

Posted by Sten Westgard, MS

At the AACC/ASCLS conference in San Diego, many of the Sigma VP laboratories contributed to a poster on evaluating the applicability of various analytical performance specifications for AST:

Multisite study of AST analytical performance reveals disparities in global performance specifications

The full poster can be downloaded from our westgard.com download section here. (free membership required)

But what was most rewarding was meeting with many of the co-authors who came to the meeting.

Ms Ginny Foo, Mr. Rajkumar Rengaraju, Co-authors from Innovative Diagnostics, Singapore

more pictures after the jump

Are there new (Erna Lenters-)"Westra Rules" for HbA1c?

Posted by Sten Westgard, MS

Erna Lenters-Westra and Emma English have a new paper in CCLM:

Evaluating new HbA1c methods for adoption by the IFCC and NGSP reference networks using international quality targets, Erna Lenters-Westra, Emma English, Clin Chem Lab Med. 2017 Aug 28;55(9):1426-1434. doi: 10.1515/cclm-2017-0109

They continue the standard set by the IFCC Task Force on Implementation of HbA1c standardization (TF-HbA1c) that advocates the Sigma-metric approach as the model for setting and evaluating quality targets for HbA1c. "The TF-HbA1c has set default risk levels of 2 [Sigma] for routine laboratories and 4 [Sigma] for laboratories performing clinical trials."

Further, the paper established its own allowable total error for evaluation:

"The total allowable error (TAE) for HBA1c has been set by the TF-HbA1c as a default of 5 mmol/mol (0.45% DCCT) at an HbA1c level of 50 mmol/mol (6.7% DCCT), which corresponds with a relative TAE of 10%... in SI units [6.9% DCCT units...]"

In conversation with Dr. Lenters-Westra, she has outlined a fleshed-out set of analytical performance acceptability specifications:

• CV in CLSI EP-5 should be at 48 mmol/mol and 75 mmol/mol < or equal to 3% in SI units or < or equal to 2% in DCCT units
• Bias in CLSI EP-9 at 48 mmol/mol and 75 mmol/mol compared with the mean of at least 3 certified secondary reference measurement procedures should < or equal to 2 mmol/mol or < or equal to 0.2% DCCT units.

Perhaps we should refer to these as (Lenters-)"Westra Rules" for HbA1c?

Want to hear more about this? We encourage you to attend the Quality in the Spotlight Conference in 2018, where Dr. Lenters-Westra will lecture extensively on her work (as well as receive the Westgard Award).

Sigma-metrics in Korea PT

Posted by Sten Westgard, MS

A recent paper in Annals of Laboratory Medicine noted that the New-Generation Proficiency Testing program of the Korean Association of Quality Assurance for Clinical Laboratory:

Sigma-Metrics of Electrolyte Tests From a Recently Launched New-Generation Proficiency Testing Program of the Korean Association of Quality Assurance for Clinical Laboratory, Hee Jun Huh, Yun Mi Park, Seungok Lee, Chunhwa Ihm, Soyeon Seo, Sang Gon Lee, Joonseok Par, Hae-il Park,  Ann Lab Med 2017;37:339-342.

 Here they are using a two-sided Six Sigma Method Decision Chart.

Q&A: 10 Questions on QC from the first webinar

Posted by Sten Westgard, MS

In our first webinar, we had a lot of questions come up - thanks to all the participants who submitted them.

I'm going to address them in the order in which they were submitted, so I apologize for the random order.

From Jakarta: "How do we draw OPSpecs charts? By Excel?"

OPSpecs charts can be downloaded directly from the Westgard Website. It requires fairly sophisticated knowledge to build them, so we usually don't see them done by Excel. There are a few programs that build dynamic OPSpecs charts, including the Bio-Rad Westgard Advisor, and our own EZ Rules 3.

From Hyderabad: "What is the exact IQC run frequency in major hospital laboratories?"

This is an interesting question. We can answer it in two ways. There is the QC frequency that current laboratories are performing, and that's usually once a day. This frequency is based more on compliance and regulatory minimums, and has little relationship to the needs of the patient. But the more interesting question is what QC frequency should major laboratories be performing? That answer is based on volume and performance of the assays. It could be that an assay with high quality can still run only once a day even in high volume labortories, but the forthcoming QC frequency tools will soon reveal that a low quality assay may need to run controls every 100 patients or even every 10 patients!

From Tamluk: "Please comment on the 10x rule"

This is a very common question - often because laboratories are seeing a 10:x violation and wonder if they truly need to reject their run(s). The short answer is, if the Sigma-metric is 4 or higher, the 10:x rule is not a necessary rejection rule. Only when performance falls below 4 Sigma do the 10:x (or 6:x. 8:x, 9:x, 10:x, or 12:x rules) become required for proper error detection.
Another tip on using the 10:x rule - make sure there aren't some rounding issues causing the values to fall above or below the mean. If you have a "mean" of 10.5 on a test where your system does not report any decimal places, to give a simplistic example, then any value 10 or 11 is going to be above or below the mean. If you suspect the rounding of your system is causing too many values to fall above or below the mean, try to increase the number of decimal places reported in your system. Internally, you can always use an extra decimal place, although you won't report that extra decimal place in your results.

From Kolkata: "Are "Westgard Rules" only applicable for the diagnostics field?"

We rarely see "Westgard Rules" used outside the diagnostic laboratory medicine field, partly because in other fields, it's pretty cheap to run additional controls. One of the primary reasons we use multirule QC procedures is the expense of controls - we can't afford to run many of them, so we need to get as many rule interpretations out of each measurement as possible. But we have anecdotally heard that rocket engine testing and some anthropological testing (both cases where the number of measurements is small), have implemented "Westgard Rules" in their QC practices.

From Kolkata: "What is the frequency for running 2 levels of control if a lab is processing more than 400 samples a day?"

Like the question above, this has two answers. For compliance purposes, most countries will be satisfied with QC run once a day (of course for hematology and coag and some other fields, a higher frequency, perhaps 3 times per day, is required). But the scientific answer to this question will require more information about the quality required by the test (TEa) and the performance observed (CV, bias) of the method. If we know this is a Six Sigma test, for example, we only need to run QC once a day and we could use just about any QC rule. But a 3 Sigma method will require more frequent QC.

From Indonesia: "What is the best way to troubleshoot when we are out of control?"

There are a lot of things you should NOT do when you're out-of-control, such as repeating controls without any investigation, or running new controls without any trouble-shooting. There are times when you do want to repeat the control (to confirm you are back in-control, for instance), but you should be trouble-shooting and seeking the root cause of the type of error (random or systematic). There are a lot of different trouble-shooting flowcharts you can consult, including some from our Basic QC Practices manual.

From India: "What is your suggestion when there is a narrow or very narrow SD which leads to false rejection?"

This is a situation that occurs all too frequently. We call it "anorexic QC", where the continuous use of tight 2 SD limits, combined with continuous exclusion of 2 SD outliers, can progressively narrow the limits.  The answer is to move away from the 2 SD limits toward a wider QC design (assuming the Sigma-metric justifies it). It could be that over a longer period of time, the use of wider limits will result in a healthier SD (it will lose its anorexia) and then the QC design will be adjusted again to something less wide.  Every few months you should look at performance, and it if has significantly changed (for example, shifting a Sigma-metric from 4 to 5, etc.), you need to re-design your QC procedures.
I usually see laboratories in anorexic QC situations, but rarely do I see them "fatten" up. It would be interesting if a laboratory wanted to share this journey with us - I think it would actually be something that could be published.

From Kolkata: "Can replicate testing in a laboratory check the IQC of the test results?"

I'm having a little bit of a problem figuring out what this question is really trying to ask. Replicate testing of the patient samples is a last resort method of improving performance. If you have a method with a Sigma-metric below 3, it's something you may need to consider. By testing twice, and using that to generate just one results, you reduce the variation in the answer. But it's the equivalent of doubling your test volume, a very expensive step. Now, if the laboratory is actually suggesting making replicate measurements of the control, that is the equivalent of doubling your control measurements. So if you run two controls, but you measure them twice, that gives you effectively 4 control measurements to use in your QC charts. So this is a way to increase your QC without spending more on controls.

From Tamluk: "Is it mandatory to run every level of controls of each test? Like there are up to 4 levels of control for HIV? Is there a requirement to run every level every day?"

The answer to this will depend on the regulations that apply in your country, as well as the manufacturer recommendations. If the regulations only require QC once a day but only a positive and negative, you may be free to run fewer controls than are available. But if the manufacturer instructions recommend that you run 4 levels of control, you really should run those 4 levels. In the US, if you violate the manufacturer recommendations, you have effectively modified the test into a high complexity method and there are a huge number of new method validation studies you must perform. So the short answer is to do what the manufacturer and regulator recommend. Then and only then, can you take into account the actual performance of the method. If your regulations and manufacturer don't specify the number of controls to run, and you have a Six Sigma method, you may consider going to just two controls per run (one low positive and one negative).

From Hyderabad: "Is 1:2s acceptable in hematology?"

Here I must use the dreaded response, "It depends."  If you are performing your QC manually, using your own brain power, and charts drawn on paper, then a 1:2s warning rule may be useful. If you have any kind of software performing your QC, we recommend using only rejection rules and skipping the 1:2s rule as a warning rule. Note this is true for any type of testing, not just hematology or chemistry. The 1:2s rule generates a lot of false rejections, and you don't want to spend your precious time chasing those ghosts.

There were more than 50 questions submitted during the first webinar. We will get to more questions in another post.

Budgets aren't the only things getting squeezed in the lab...

Posted by Sten Westgard, MS

We've just seen probably the most important publication since the Milan Consensus was issued in 2014-2015.

It's a paper that significantly restates the goals for imprecision, bias, and allowable total error based on a new biological variation study:

Clin Chem. 2017 Jun;63(6):1141-1150. doi: 10.1373/clinchem.2016.269811. Epub 2017 Apr 20.

Biological Variation Estimates Obtained from 91 Healthy Study Participants for 9 Enzymes in Serum.

Carobene A^1,2, Røraas T³, Sølvik UØ⁴, Sylte MS⁵, Sandberg S^3,4,5,2, Guerra E⁶, Marino I⁶, Jonker N^7,2, Barla G⁷, Bartlett WA^8,2, Fernandez-Calle P^9,2, Díaz-Garzón J⁹, Tosato F¹⁰, Plebani M¹⁰, Coşkun A^11,2, Serteser M¹¹, Unsal I¹¹, Ceriotti F⁶; European Biological Variation Study of the EFLM Working Group on Biological Variation

 

If we just focus on the desirable allowable total error (TEa), we can see some significant tightening in the performance specifications:

 

Test	TEa - REVISED	TEa - 2014	Reduction
ALT	14.4	27.48	47.6%
AST	13.4	16.69	19.7%
GGT	15.7	22.11	29.0%
ALP	10.7	12.04	11.1%
LDH	7.7	11.4	32.5%
CK	20.4	30.3	32.7%
AMY	13.4	14.6	8.2%
LIP	12.6	37.88	66.7%

 

If we think for a moment what this means for manufacturers, it's a massive change. If you essentially cut the target in half, your current methods probably have to be re-engineered to meet this new level of performance.

 

Put a different way, these new goals may be more accurate, but far less applicable. More ideal, but less practical.

 

For years, there have been complaints that the Ricos goals were too large, that they overestimated the amount of allowable error. Indeed, even after these revisions, there will still be those who claim that we need to be even more demanding about assay performance than how the Ricos allowable total error goals are structured and implemented.

 

But if we shrink our goals down to the point where no method on the market can hit the target, what have we actually accomplished? We will have a perfect set of goals that are impossible to achieve. That may be helpful to research and design departments at the manufacturers when they contemplate their next generation of instruments, but for laboratories that are operating today, with today's methods, they face the prospect of either having to massively increase the amount of QC they run and effort they devote to QC, or they will need to switch to goals that are more achievable and practical for today's clinical use.

 

The phrase that comes to mind here is, Don't let the perfect be enemy of the good. An obsession with constructing the perfect performance specification may make for great papers and publications and conference lectures, but it may also have the practical effect of leaving laboratories struggling with goals they can't achieve and performance they can't change.

 

These revisions may represent the goals of the perfected future. But we may need a set of goals for our imperfect today.

 

 

Setting specifications for setting specifications for...

Posted by Sten Westgard, MS

One of the outcomes of the Milan 2014 meeting was a set of "Task-Finishing" groups that were assigned various aspects of the Milan Mandate (more commonly known as the 1st EFLM Strategic Conference Consensus Statement, or EFLM-SCCS, which can be pronounced, 'ee-effle-em-sucks').

EFLM-SCCS established that there were now 3 models for setting quality specifications, rather than the 5 levels of the Stockholm Consensus Hierarchy that had been established in 1999. This was hardly radical, as it mostly involved merging 3 levels into the lowest form of model, "state of the art".

But what was a true advancement for the field was the recognition that not all tests were going to be able to use the highest, best, or most ideal model for setting performance specifications. That is, some methods like HbA1c can have their performance specifications defined by the clinical use (model 1), but other methods aren't so clearly understood or clinically applied. So we face a future, finally realistic, where some tests will have biologically-based (model 2) goals, and other tests will have clinical-use-based (model 1) goals, and all the rest will have state of the art based goals (model 3). It's going to be a heterogeneous world. Between 1999 and 2014, there had been increasing pressure to use the biologic-based goals (also known as the "Ricos goals"), but evidence had been mounting that the Ricos goals were sometimes too wide, sometimes too narrow, for the methods on the marketplace. Indeed,the EFLM is in the process of revamping the entire database with much more statistical rigor (which is why there hasn't been an update since 2014).

So we have accepted that there will be different goals, and different goal models, for different analytes.

But who gets to choose the goal, the model, for each analyte?

As it happens, the EFLM has designated a group to do just that, and their preliminary findings have been published:

Criteria for assigning laboratory measurands to models for analytical performance specifications defined in the 1st EFLM Strategic Conference, Ceriotti F, Fernandez-Calle P, Klee GG et al, CCLM 2017;55(2):189-194.

This publication doesn't give the assignment of the analytes to the appropriate models, but it does describe the criteria which will be used to decide how to assign each analyte to a model.

In other words, it's not the beginning of the end, but it's the end of the beginning.

The paper does provide some example goal model assignments:

• whole blood HbA1c is the Model 1: outcome-based
• plasma Uric acid is Model 2: biologic variation-based
• urine Sodium is Model 3: state of the art

After three years of effort, it's good to see the progress of the task finishing group. One only hopes it won't be another three years before we see the final list.

A final point to be made is that the criteria note that they are not final, but temporary. Since biologic variation data, clinical use, and even the state of the art are not stationary concepts, these model assignments will evolve over time. What may only be assigned a state of the art model today may eventually graduate to a higher model assignment in the future.

We'll just need to develop some specifications on how often we need to revisit the specifications for assigning specifications...

[P.S. I have a bad attitude about this. I'm not trying to be disrespectful to the EFLM or the committee task group, they are all extremely smart individuals and far more qualified than me to do this work. The paper is a thorough, well-written argument and definitely worth your attention. But it just seems like this is taking a long time. We may be generating lots of material for papers and conferences and abstracts, etc. but we look like we're moving at glacier speed.]

 

Coming to Croatia: Zagreb's EFLM course

Posted by Sten Westgard, MS

I'm a bit behind in placing a notice about this, but nevertheless I am thrilled to be a part of the 15th EFLM postgraduate course, this year's theme being "How to assess the Quality of your Method?"

Over the course of two days, October 24th and 25th, a very strong roster of experts and scientists will cover all the method validation and verification topics necessary for assessing the acceptability of a method.

I will be speaking near the end, with a lecture first on Six Sigma, and lastly on Biological Variation.

You can find more details here about the course.

New Publication: Useful measures and models for quality

Posted by Sten Westgard, MS

There's a new article available online-first in Clinical Chemistry and Laboratory Medicine. This article is of considerable importance to the debate on the "Milan Mandates" and the future implementation of error models for QC management. So while this is only available to subscribers at the moment

Useful measures and models for analytical quality management in medical laboratories, by James O. Westgard, PhD

Useful measures and models for analytical quality management in medical laboratories, Westgard JO, Clin Chem Lab Med 2015, ahead of print.

Since there was no time set aside to discuss the utility and practicality of the Total Error model at the Milan conference, this opinion article is an overdue counterweight to the Milan Mandates. It is expected that there will be counter-editorials also published in the same issue with this editorial.

The debate on error, bias, imprecision, trueness, inaccuracy, total error, measurement uncertainty, etc, etc, continues...

Pop Quiz: what's the EQA failure rate of laboratories?

Posted by Sten Westgard, MS

An interesting abstract was published at the Paris IFCC meeting. It detailed the EQA performance of a set of 12 public laboratories in Catalonia. Can you guess what the failure rate for these labs for biochemistry EQA?

• 14.5%
• 14%
• 3%
• none of the above

The answer, after the jump...