The Westgard Rules

Sigma metrics in veterinary testing

By Sten Westgard, MS

I just wanted to draw your attention to a seminal paper that came out recently:

Quality control validation, application of sigma metrics, and performance comparison between two biochemistry analyzers in a commercial veterinary laboratory, Alison J Farr and Kathleen P Freeman, J Vet Diagn Invest. September 2008;20(5):536-44. (subscription required)

In this paper, Kathy Freeman, who has previously written a guest essay for Westgard Web, and her colleague Alison Farr apply Sigma metrics and QC design to common veterinary diagnostic tests. QC in veterinary testing is not a subject that is well covered in the scientific literature.

What's most interesting from our point of view is that in order to calculate Sigma metrics and perform QC Design on these tests, Freeman and Farr needed key information: quality requirements for veterinary tests for the major non-human species (cat, dog, horse). For the most part, the requirements don't exist, or at least a set of goals is not in widespread circulation. So Freeman and Farr did a survey review and made some assessments of test interpretation in their own laboratory. The result is the largest list of quality requirements for veterinary testing that we've ever seen. We excerpted the table, with Freeman's permission, here. 

It's a comprehensive paper, covering the method principles, decision levels, quality requirements, Sigma metrics, and QC Design with rule implementation. Highly recommended.

Glucose vs. HbA1c

By Sten Westgard, MS

Over on the main website, Dr. Westgard has a new essay discussing estimated average Glucose and the new push to use HbA1c methods for screening and diagnosis of diabetes. There have been two front page articles in Clin Lab News on these topics.

Two additional points worth making about HbA1c. First, the National Glycohemoglobin Standardization Program (NGSP) has set a goal of reducing the imprecision of HbA1c methods. So CAP proficiency testing criteria, for example, are going to be tightened from 12% down to 6% over the next four years.

Let's repeat that. The quality requirement is going to be reduced from 12% to 6% in the next four years.That's cutting the quality requirement in half. Wow.

That's real leadership and courage right there. That is using the power of regulation to force improvements in instrumentation. I know my experience of the laboratory world is limited, but I haven't seen that kind of agressive goal-setting before. Have you?

It will be very interesting to see how the diagnostic manufacturers respond to tightening quality requierments. Simply put, I don't think many of the methods on the market now can make the cut. They will not be good enough to hit a 6% target. ( Also, will the other PT providers follow suit, or will they keep their targets wide to try and scoop up customers? Here's where competition might work against quality. )

Second point. The article doesn't talk about a big issue in HbA1c methods: differences and bias between methods. For all the work on traceability, the methods in the field remain considerably different. There's a recent paper out in the Journal of Clinical Pathology, where Holmes et al seriously question the use of HbA1c testing:

“Our results also question the validity of using HbA1c test results as a measure of the quality of diabetes care.  Given the present state of the art for HbA1c testing, the proposed pay-for-performance use of these results could lead to an ironic scenario in which analytic methods that are biased low relative to the NGSP reference method are considered to reflect a high quality of care and secure a better level of reimbursement for the provider while at the same time promoting clinical inertia for the intensification of diabetes treatment and underestimating the risk of diabetic complications.”

“The gap between the level of analytic quality that experts consider necessary to obtain the maximum clinical usefulness from an HbA1c test and the level of performance presently provided by the methods used in the United States suggests that there is, indeed, cause for concern.  The shortcomings of contemporary HbA1c testing need to be more widely recognized by the end users of the results.  In addition, individuals and organizations involved in the design, manufacturer, performance, accreditation, and regulation of HbA1c testing need to devise and implement plans for improving the analytic quality of this important test.”

Holmes EW, Ersahin C, Augustine GJ, Charnogursky GA, Gryzbac M, Muttell JV, McKenna KM, Habhan F, Kahn SE. Analytic bias amoung certified methods for the measurement of hemoglobin A1c. Am J Clin Pathol 2008;129;540-547.

When you add pay for performance on top of the method bias problems, you've got a scary scenario. Just by switching to a new, lower-reporting method, clinicians and institutions could increase their compensation - without ever changing the quality of healthcare delivered to patients. That's not pay for performance. That's pay for method bias. 

What's New: November 2008

New Download: Sigma-Metrics Tool (and audioconference presentation)

On September 4th, 2008, Dr. Westgard spoke as part of the AACC audioconference on "New Directions in Laboratory QC" -  which was subsequently quoted in the Clin Lab News article on Risk and CLIA.  We are pleased to make available Dr. Westgard's complete presentation, as well as two Sigma-metrics Charts and instructions on how to use them in QC Design.

http://www.westgard.com/audiodownload.html

Westgard Sigma Analysis: A new direct HbA1c Method

At the 2008 AACC/ASCLS convention, a poster was presented for a new direct enzymatic assay for %HbA1c. How does it stack up against HPLC and immunoassay methods? How do you judge a method when you've got multiple comparison methods and multiple quality requirements?

http://www.westgard.com/qcapp53.htm

Interview: Dr. R. Neill Carey (A brief introduction to EP 15)

We were fortunate to get R. Neill Carey, PhD, the chair of the CLSI EP 15 committee, to present and 

explain that new standard at the Chicago Method Validation workshop. He also contributed a chapter to the new Basic Method Validation manual on the same topic. But for those who have never heard of EP 15 before, we conducted a short interview with Dr. Carey. This short introduction to EP 15 may pique your interest in this new guideline.

http://www.westgard.com/interview9.htm

Thinking about Three Sigma: 2 thoughts on troublesome performance

In a previous lesson, we discussed some possible actions to take when the Sigma-metric for a method is higher than Six. But what about those methods with low Sigma-metrics? What do you do when Sigma analysis delivers bad news?

http://www.westgard.com/lesson87.htm

CLIA at 20

posted by Sten Westgard, MS

This week, AACC is going to host an audioconference on the latest update of CLIA. We're celebrating 20 years of CLIA in 2008.

Over at the online store, you can celebrate by saving $20 off the price of the CLIA Final Rules manual.
Use the coupon code clia20 during the checkout and you'll save $20.

This offer will run from now (October 6th) until Halloween - that's more than 20 days of savings.

c8000 Sigma Analysis

Posted by Sten Westgard

One of the highlights for me of the AACC convention in Washington, DC, was my inclusion in a poster that analyzed the method performance of the Abbott Architect c8000. I'm pictured here with fellow authors (left to right) Gene Osikowicz, Charles Wilson, and John Baker (lead author). They deserve most of the credit for the work of collecting the data.

The poster can be viewed here and the QC application on Westgard Web can be viewed here.

What's new on Westgard Web: September 2008

What's New this month at Westgard Web:

• Fall Workshops last chance for early registration
• High Reliability Testing: First Principle
• Guest Essay: Time to engage in Measurement Uncertainty
• Westgard Sigma Analysis: An automated POC hematology instrument

FALL WORKSHOPS: Last chance for Early Registration

WORKSHOP: Method Validation and Verification
1-day intensive training based on the new edition of Basic Method Validation
Two locations to choose from

October 10th, Abbott Park, Chicago, Illinois
  [supported by an educational grant from Abbott Diagnostics]

October 17th, Olmsted Clinic, Rochester, Minnesota
  [supported by an educational grant from Olmsted Clinic]

Find out more and get a coupon for early registration discounts at
 http://www.westgard.com/workshops.html#bmv

WORKSHOP: Six Sigma Quality Design and Control

Dr. Westgard's most popular and most requested workshop is back.
1-day introduction to Six Sigma, with new applications
and innovative tools.

November 12th, Spectra Laboratories, Rockleigh, New Jersey
  [Supported by an educational grant from Spectra Laboratories]

Find out more and save $100 off the regular registration rate at
 http://www.westgard.com/workshops.html#sigma

HIGH RELIABILITY TESTING: First Principle - Preoccupation with Failure

One of the key principles in High Reliability Organizations is a
Preoccupation with Failure. No, this doesn't mean a lab that needs Prozac.
While Preoccupation with Failure may sound bad, in HROs, this focus on error
is a key factor in safety and success.

http://www.westgard.com/lesson85.htm

GUEST ESSAY: Time to Engage in Measurement Uncertainty

In the ongoing "War of Words" in the Lab, it's time to hear another voice.
Dr. Dietmar Stockl, an expert from across the Atlantic,
provides us with a detailed essay explaining how measurement uncertainty
 can be useful to the laboratory - and even co-exist with Total Error.

http://www.westgard.com/guest41.htm

WESTGARD SIGMA ANALYSIS: Automated POC hematology instrument

A recent paper in a prominent journal evaluated a new POC hematology instrument.
They concluded that the performance was "within acceptable limits."
When the bar is that low, what's the actual Sigma performance?

http://www.westgard.com/qcapp51.htm

Another voice on Measurement Uncertainty

By Sten Westgard, MS

While Dr. Westgard has spoken about the "War of Words" between Uncertainty and Total Error in several essays (here, here and here most recently), we thought readers might want an additional perspective on the issue.

Dietmar Stockl, an expert in statistics and laboratory quality control from across the Atlantic, graces us with an guest essay on the calculation, use, benefits and limitations of measurement uncertainty: Time to Engage in Measurement Uncertainty. Dr. Stockl provides an in-depth look at uncertainty concepts and calculations, as well as a moderate viewpoint on the use of the term. He views both Total Error and Measurement Uncertainty are useful concepts and believes there is room in the world for both of them. It's not necessary for one term to eliminate the other.

Not so coincidentally, a colleague of Dr. Stockl's, Linda Thienpont, has a letter in Clinical Chemistry talking about the attempt to incorporate bias into the calculation of measurement uncertainty. If bias can be thrown into the uncertainty calculations, a case could be made that Total Error is no longer necessary. Dr. Thienpont points out that slipping bias into the uncertainty calculations is not a good idea and can lead to distorted results. She concludes that bias must remain separate from uncertainty calculations, which means another approach like Total Error is required to account for it.

See more at Thienpont LM. Calculation of measurement uncertainty-Why bias should be treated separately. Clin Chem 2008;54:1587 (subscription required)