The Westgard Rules

Comments on "HbA1c for screening and diagnosis of diabetes?"

By Sten Westgard, MS

Dr. Westgard's recent essay on HbA1c for screening and diagnosis of diabetes and an accompanying post generated an interesting reply. Offered here in its entirety:

Jim, Sten,

 

I would like to comment on your article on HbA1c for screening and Diagnosis.  First, the comment by Sten that the tighter CAP criteria are not addressing the issue of bias.  It does indeed address both bias and imprecision because the CAP is using an accuracy base - the NGSP assigned values.  So if a method is either biased from NGSP or shows variability within or between labs, this will decrease the lab's chances of passing.  The CAP no longer uses peer group grading for HbA1c. 

 

I'm not sure what you mean by "scientific marketing" and if this should be taken as a negative comment.  Sure manufacturers would like to see HbA1c used for screening since this would increase use of the test.  But use of HbA1c for screening has been discussed for a long time and the discussion is based on very sound arguments.

 

The main reason to use HbA1c rather than fasting glucose is not for "convenience of any particular group".  It, in fact, will get people screened and diagnosed that may not otherwise be diagnosed.  There is still a large group of people who have undiagnosed diabetes.  This is why so many people (25% of those diagnosed) have complications at the time of diagnosis!  Using HbA1c will allow screening and/or diagnosis more readily e.g. at regular doctors' appointments where patients usually don't come fasting.  Furthermore, many physicians already use HbA1c but there are no specified cutoffs so it's hard to know exactly how it is being used and it may not be used appropriately. 

 

Another drawback of fasting glucose is that it has a large (much larger than HbA1c) biological variability.  And measurement of glucose isn't perfect either.

 

The screening cuoff for HbA1c of 6% is equivalent to 125 mg/dL AVERAGE GLUCOSE, not fasting glucose.  The equation (eAG =28.7*HbA1c-46.7) gives you a mean blood glucose, not fasting.   You can't easily equate a patients mean glucose with his or her fasting.

 

And I disagree that discussions of using HbA1c for diagnosis/screening are guided by market forces and not quality of care.  The main goal is to get more people with undiagnosed diabetes diagnosed.  It is not for the pockets of manufacturers or scientists. 

 

Randie

 

Randie R. Little, Ph.D.

NGSP Network Coordinator

Co-Director Diabetes Diagnostic Laboratory

Depts. Of Pathology & Anatomical Sciences and Child Health
University of Missouri School of Medicine
Columbia Missouri

Just to be clear, we meant no disrespect of the NGSP efforts in either the essay or the post. Indeed, they are doing far more to improve quality than most other organizations of a similar nature.

As they say, it's important to be able to disagree without being disagreeable. We welcome feedback, comments, and different points of view.

Glucose vs. HbA1c

By Sten Westgard, MS

Over on the main website, Dr. Westgard has a new essay discussing estimated average Glucose and the new push to use HbA1c methods for screening and diagnosis of diabetes. There have been two front page articles in Clin Lab News on these topics.

Two additional points worth making about HbA1c. First, the National Glycohemoglobin Standardization Program (NGSP) has set a goal of reducing the imprecision of HbA1c methods. So CAP proficiency testing criteria, for example, are going to be tightened from 12% down to 6% over the next four years.

Let's repeat that. The quality requirement is going to be reduced from 12% to 6% in the next four years.That's cutting the quality requirement in half. Wow.

That's real leadership and courage right there. That is using the power of regulation to force improvements in instrumentation. I know my experience of the laboratory world is limited, but I haven't seen that kind of agressive goal-setting before. Have you?

It will be very interesting to see how the diagnostic manufacturers respond to tightening quality requierments. Simply put, I don't think many of the methods on the market now can make the cut. They will not be good enough to hit a 6% target. ( Also, will the other PT providers follow suit, or will they keep their targets wide to try and scoop up customers? Here's where competition might work against quality. )

Second point. The article doesn't talk about a big issue in HbA1c methods: differences and bias between methods. For all the work on traceability, the methods in the field remain considerably different. There's a recent paper out in the Journal of Clinical Pathology, where Holmes et al seriously question the use of HbA1c testing:

“Our results also question the validity of using HbA1c test results as a measure of the quality of diabetes care.  Given the present state of the art for HbA1c testing, the proposed pay-for-performance use of these results could lead to an ironic scenario in which analytic methods that are biased low relative to the NGSP reference method are considered to reflect a high quality of care and secure a better level of reimbursement for the provider while at the same time promoting clinical inertia for the intensification of diabetes treatment and underestimating the risk of diabetic complications.”

“The gap between the level of analytic quality that experts consider necessary to obtain the maximum clinical usefulness from an HbA1c test and the level of performance presently provided by the methods used in the United States suggests that there is, indeed, cause for concern.  The shortcomings of contemporary HbA1c testing need to be more widely recognized by the end users of the results.  In addition, individuals and organizations involved in the design, manufacturer, performance, accreditation, and regulation of HbA1c testing need to devise and implement plans for improving the analytic quality of this important test.”

Holmes EW, Ersahin C, Augustine GJ, Charnogursky GA, Gryzbac M, Muttell JV, McKenna KM, Habhan F, Kahn SE. Analytic bias amoung certified methods for the measurement of hemoglobin A1c. Am J Clin Pathol 2008;129;540-547.

When you add pay for performance on top of the method bias problems, you've got a scary scenario. Just by switching to a new, lower-reporting method, clinicians and institutions could increase their compensation - without ever changing the quality of healthcare delivered to patients. That's not pay for performance. That's pay for method bias. 

Failure is an option?

By Sten Westgard

On November 4th, the Joint Commission issued an interesting press release, titled "Lab Decisions Will No Longer Affect Hospital Decisions."

The specific language of the press release stated:

"Beginning January 1, 2009, under new Joint Commission policy, laboratory accreditation decisions will no longer immediately impact hospital accreditation decisions."

I have subsequently seen comments on a listserve wondering if it's now acceptable for JC-accredited hospitals to have laboratories that fail inspections. The simplistic interpretation of this rule is that laboratory problems no longer impact the hospital. Hospitals can keep running regardless of the state of their laboratory.

But that's not really the case.

I contacted Megan Sawchuk, Associate Director of the Standards Interpretation of the Joint Commission. She elaborated on the new policy and cleared up any ambiguity:

"The December 2008 Perspectives announcement regarding laboratory accreditation decisions has two important elements. One, the Accreditation Committee voted to eliminate the automatic, direct weight of an adverse decision in the laboratory on the hospital. And two, an adverse laboratory decision from The Joint Commission, CAP or COLA will be added to the hospital's Priority Focus Process (PFP) data. PFP data is presently used by The Joint Commission to monitor the hospital's overall performance and prioritize the timing of their unannounced survey in the 18-39 month window. Thus, an adverse decision in the laboratory will significantly increase the likelihood of an earlier hospital survey to assess compliance at the organizational level.

"By using this method, the hospital decision is based on their actual overall performance with consideration of that of the laboratory. This is an improvement over the current process of automatically applying an adverse laboratory decision to the hospital, which assumes an overly simple relationship between the two integrated but separate entities. Noncompliance in the laboratory is often associated with poor performance in the overall organization, but not always. This method also maintains the integrity of the the laboratory as an essential service in the hospital's accreditation decision process."

To be clear: a failing laboratory will still take down a hospital with it. The downward spiral to revocation of accreditation may not be as fast as it used to be. But the usual regulatory process takes time in any case. Inspections generates citations, which require responses, which may then generate additional inspections, additional responses, etc. Immediate action happens very rarely. The Joint Commission retains all the policies and tools they need to come down hard on a lab and hospital. This new policy just gives them a little more latitude.

One last thing: this is a clear admission that many laboratories in America have significant problems. If laboratories were operating perfectly (or even just in compliance) and there weren't any worries about them, we would have no need to decouple their accreditation decisions from the hospitals.    

What's New: November 2008

New Download: Sigma-Metrics Tool (and audioconference presentation)

On September 4th, 2008, Dr. Westgard spoke as part of the AACC audioconference on "New Directions in Laboratory QC" -  which was subsequently quoted in the Clin Lab News article on Risk and CLIA.  We are pleased to make available Dr. Westgard's complete presentation, as well as two Sigma-metrics Charts and instructions on how to use them in QC Design.

http://www.westgard.com/audiodownload.html

Westgard Sigma Analysis: A new direct HbA1c Method

At the 2008 AACC/ASCLS convention, a poster was presented for a new direct enzymatic assay for %HbA1c. How does it stack up against HPLC and immunoassay methods? How do you judge a method when you've got multiple comparison methods and multiple quality requirements?

http://www.westgard.com/qcapp53.htm

Interview: Dr. R. Neill Carey (A brief introduction to EP 15)

We were fortunate to get R. Neill Carey, PhD, the chair of the CLSI EP 15 committee, to present and 

explain that new standard at the Chicago Method Validation workshop. He also contributed a chapter to the new Basic Method Validation manual on the same topic. But for those who have never heard of EP 15 before, we conducted a short interview with Dr. Carey. This short introduction to EP 15 may pique your interest in this new guideline.

http://www.westgard.com/interview9.htm

Thinking about Three Sigma: 2 thoughts on troublesome performance

In a previous lesson, we discussed some possible actions to take when the Sigma-metric for a method is higher than Six. But what about those methods with low Sigma-metrics? What do you do when Sigma analysis delivers bad news?

http://www.westgard.com/lesson87.htm

Talking about CLIA and Risk in Clin Lab News

by Sten Westgard, MS
Updated 11/7/08

Dr. Westgard is quoted in the November 2008 issue of Clinical Laboratory News

in their report on "Risk Management for Clinical Labs" [subtitle: As CLIA Turns 20, A New QC Paradigm Lies Ahead for Labs] . Here's the most interesting quote:

One of the most outspoken critics of the EQC guidelines has been James Westgard, PhD, FACB, emeritus professor in the department of pathology and laboratory medicine at the University of Wisconsin Medical School and president of Westgard QC, Inc. “So what are equivalent QC procedures? Equivalent in performance? Equivalent in detecting medically important errors? Equivalent in assuring the necessary desired quality is achieved?” asked Westgard at a September 4 AACC audioconference called “New Directions in Laboratory QC: EQC, Alternate QC and Risk Assessment.” “No, it’s none of the above. It’s just a name. It has nothing to do with any practical meaning in terms of what we think of equivalents. It’s just a name,” he said.

Back a few years ago, there were more critics of Equivocal QC, for example CAP, but then their deeming status came up for review. Now all the deemed providers have implemented equivalent QC options. CMS flexed its muscle, and now there are fewer people to be outspoken.

The whole article(pdf) is worth a read. Again, one of the most interesting details is how the FDA might handle Manufacturer Risk Information - and use it against manufacturers in the event of a problem:

Steven Gutman, MD, director of FDA’s Office of In Vitro Diagnostic Device Evaluation and Safety indicates that FDA will not directly review a manufacturer’s risk assessment material. “Because they’re providing straight-forward information on risks and risk management, we would view that as an acceptable practice, and there might be various formats for them to do that,” Gutman explained. “It might be technical bulletins, it might be on their websites, but it wouldn’t be a part of FDA-sanctioned labeling unless they violate FDA regulations, in which case our compliance program would become concerned.” Gutman said the FDA would look to see if companies were going overboard with explicit or implicit claims about their products, reviewing the information only if the agency thought there was a problem.

As we noted before in What are the Risks of Communicating Risks?, this is a tough spot for a manufacturer.

Update: You can now download Dr. Westgard's complete presentation from the September 4th, 2008 audioconference. The pdf download includes his slides, two Sigma-metrics QC Design charts, and instructions on how to use them.

DNV Healthcare and Labs - a correction

By Sten Westgard, MS

In an earlier post about DNV Healthcare and Laboratory accreditation, I inadvertently implied that only CAP or COLA accreditation would be accepted by DNV. I should have been more precise in my listing of possible accreditation agencies for laboratories; that was a mistake on my part. I didn't intend to imply that only those two accreditations would be accepted by DNV. My apologies for my imprecise statement.

The important point is that DNV currently will not accredit labs. If you move to DNV Healthcare to accredit your hospital, you will need to have another agency accredit your laboratory.

Again, I am sorry to have given the wrong impression. Thanks to Margaret Peck of Joint Commission for alerting me to the problem. The original post has been corrected as well.

Sten

CLIA at 20

posted by Sten Westgard, MS

This week, AACC is going to host an audioconference on the latest update of CLIA. We're celebrating 20 years of CLIA in 2008.

Over at the online store, you can celebrate by saving $20 off the price of the CLIA Final Rules manual.
Use the coupon code clia20 during the checkout and you'll save $20.

This offer will run from now (October 6th) until Halloween - that's more than 20 days of savings.

DNV Healthcare - Hospitals now, Labs later?

After some follow-up, we need to note that DNV Healthcare has only been deemed by CMS for accrediting hospitals. They did not get deemed status for laboratory accreditation.

In other words, they cannot inspect or accredit laboratories. Any hospital that switches to DNV Healthcare will need still need to to accredit their lab by some other agency. [In an earlier version of this post, I inadvertently implied that only CAP or COLA accreditation would be accepted by DNV. That was a mistake on my part. I didn't intend to imply that only those two accreditations would be accepted by DNV. My apologies for my imprecise statement. The important point is that DNV currently will not accredit labs. ]

Whether deemed status for laboratory accreditation is in the future for DNV Healthcare is anyone's guess. It took years of battling for DNV to achieve the hospital deeming authority. Getting to laboratory deeming authority may require years more.

Still, this reality may make it less attractive for JC accredited hospitals to switch. If you currently use JC for both your laboratory and hospital accreditation, switching to DNV will require you to switch to COLA or CAP as well. Two switches for the price of one?

CAP may be a better fit with DNV's hospital accreditation, since CAP offers an ISO 15189 approach and DNV is going to stress ISO 9001 as part of the accreditation. But COLA has also been using a lot of ISO terminology and quality systems approach in their accreditation.

The New Kid on the Block

Sten Westgard, MS

The game of accreditation agencies hasn't changed much over the last 40 years: Joint Commission, CAP, COLA. Laboratories didn't have many other choices.

Now there's a new kid on the block.

CMS recently approved DNV Healthcare as a new hospital accreditation organization. DNV's hospital accreditation program has met all the CMS requirements to deem hospitals in compliance with the Medicare Conditions of Participation. The DNV program is called NIAHO^SM (National Integrated Accreditation for Healthcare Organizations). [ DNV stands for Det Norske Veritas which is a Norwegian-based company that provides accreditation, certification, risk management and other services to many industries. What is it about Scandinavians, quality, and regulations?]

What's more interesting than just the entry of a new player into the accreditation market is their approach. NIAHO is not another compliance-oriented program - participation in this accreditation program requires the hospital to seek and achieve ISO 9001 certification. So hospitals will have to be accredited by NIAHO and certified in ISO 9001.

Here's the schedule DNV proposes for accreditation and certification:

• Year One: NIAHO Accreditation Survey and ISO 9001 Pre-assessment Survey
• Year Two: NIAHO Accreditation Survey and ISO 9001 compliance or Certification Survey
• Year Three: NIAHO Accreditation Survey and ISO 9001 Periodic Survey
• Year Four: NIAHO Accreditation Survey and ISO 9001 Periodic Survey
• Year Five: NIAHO Accreditation Survey and ISO 9001 compliance or Re-Certification Survey
• Year 6 through Year 8 and Beyond: Continue to repeat Year 3 through Year 5.

DNV will conduct annual unannounced surveys on hospitals. That's a significant change right there.

DNV's NIAHO is different than CAP's nascent ISO 15189 program. CAP is offering an ISO certification on top of the usual certification. That is, you have to do the usual CLIA-based certification, but you can add ISO 15189 on top of it. If you choose DNV Healthcare, you'll have to seek ISO 9001 certification as part of the process. Compliance alone is not a DNV option.

We have yet to see what kind of specific laboratory rules DNV Healthcare will provide. As with a lot of the ISO standards, specifics are often hard to find. Many ISO standards provide broad goals without technical specifics, leaving it up to the managers to adapt and apply the rules. Will there be Checklists? Tracers? Something else? So far, we don't know.

Obviously, whatever DNV Healthcare applies will have to be in compliance with CLIA regulations. But how will ISO 9001 and CLIA minimums mix? Will DNV require more from hospitals and laboratories than JC or CAP?

The even bigger question is - will DNV Healthcare compete on cost, quality or another feature? The cynic in us wonders if more competition will drive down prices and possibly sacrifice quality. The optimist in us thinks it would be interesting to see an accreditation body make excellence, instead of compliance, its competitive strategy.

Stay tuned.

Has Quality gone Sub-prime?

Posted by Sten Westgard, MS

As our Congress ponders a $700 billion dollar bail-out of Wall Street, and the rest of us on Main Street ponder our financial futures, it seems like as good a time as any to draw some parallels between the larger US economy and the laboratory economy.

For those of you who aren't watching the news, this Wall Street crisis got its start in Sub-Prime mortgages. (A nifty picture here illustrates how and why this happened and a handy little timeline on the most recent events in the crisis can be found here) Subprime mortgages used to be something that only people with poor credit history could get, perhaps only 15% of the mortgages. They used to be considered a risky proposition that most investors avoided. Yet they become very popular with both borrowers and investors alike, until nearly 25% of all mortgages were subprime.

What happened?

The Start: The Government is here to help

The seeds of this financial disaster were planted, ironically, in the ashes of the last crisis, the Internet bubble. At that time, the Federal Reserve lowered interest rates to just 1% in order to spur the economy. These low rates made it easier to borrow money, making home ownership possible for previously marginal home owners.

In the aftermath of laboratory scandals, the government passed the CLIA legistation. However, since they were unable to work out the procedure for manufacturer's QC clearance, they delayed that provision. In the interim, they gave some devices and QC methods temporary waivers.

The New Rules

Eager to attract grow their margins, the mortgage industry created a new raft of "financial instruments" to extend loans to riskier and riskier customers. These loans included the following types:

• Teaser Rates, aka Adjustable Rate Loans. This offered a low interest rate at the beginning of the loan, followed later by a dramatic rise in the rate. Save and buy now, Pay a lot later.
• No Doc or "Liar Loan." This loan allowed the applicant to "state" their income but not provide any proof of this income. It was perhaps first intended for people who work in jobs where the wages are primarily earned in cash, perhaps day laborers and waitresses.
• NINA Loan: No Income. No Asset Verification. For this loan, the applicant did not have to state an income, nor did they have to verify their assets. In theory, these were loans for people who wanted their privacy and were willing to pay through the nose to keep it.
• NINJA Lona: No Income. No Job. No Assets. This was a loan given to people who could fog a mirror. This isn't just tortured logic. This is logic that has been murdered.

At the height of the bubble, when housing prices were rocketing up, these loans seemed to make sense. Why not take out a subprime mortgage, if only a few years from now, the value of the house is going to double. If housing prices rise forever, you can sell the house to the next person for far more than your mortgage, no matter what the terms.

What excerbated this problem even more was the unscrupulous nature of some of the loan officers. Since they were paid commissions up front for getting new subprime loans, some officers would falsify mortgage applications for their unknowing clients.

After repeated delays, CMS and the FDA cannot find an agreeable way to implement manufacturer QC clearance. So it's back to the drawing boards. CMS writes a new clause in the law - "Equivalent Quality Testing" - and invents a new set of "Equivalent QC Procedures" which are added to the Interpretative Guidelines of the State Operators Manual: Option 1: Run controls for 10 days, then run QC only once a month Option 2: Run controls for 30 days, then run QC only once a week Option 3: Run controls for 60 days, then run QC only once a week "Option 4" or "Alternative QC" has been the subject of numerous CLSI committee, with years of work put into developing so-called Risk Management approaches The absurdity of these options have been exhaustively documented and discussed on Westgard Web. Yet despite the near-universal condemnation of the the options, despite the admission from CMS that "We blew it", it appears that these options are here to stay.

The Collaborators: Pervasive Leadership Failure

It wasn't inevitable that subprime mortgages got out of hand. It wasn't inevitable that Wall Street would become so addicted to these dubious financial instruments. If the heads of these institutions had demonstrated leadership (and ethics), instead of relentless greed, we wouldn't be in this mess.

Part of the problem was the dispersion of risk and responsibility for these subprime loans. Since no single institution owned an entire subprime loan, it didn't feel the need nor the obligation to ensure that it was a good loan. Back in the old world of mortgages, if a single bank made too many subprime loans, it would become obvious that there was too much risk and the market would force the bank to stop its practices. However, banks and mortgage lenders didn't hold onto their own loans anymore. Instead, they packaged them up into "collateralized debt obligations (CDOs)," a type of "mortgage-backed security," which they then sold off to the broader market. Thus, the banks and mortgage brokers got paid up front, outsourced the responsibility to someone else, and had little incentive to be responsible or think for the long term.

Furthermore, competitive pressures meant that choosing to be ethical was a money-losing proposition. "Everyone was doing it." If one bank refused to get into the subprime business, they suffered and another bank got that business.

Worse still, the rating agencies, which could have put a stop to the mortgage-backed securities by giving them bad (risky) ratings, instead felt pressure (and reaped rewards) to give high ratings. Thus, a security made up of many risky subprime mortgages was given a AAA rating by agencies like Moody's and Standard & Poor's. Since mortgage securities were outside federal regulations, the government did not act to rein in the risky practices, either.

Individual criticism of Equivocal QC wasn't that hard to find. But it was strange to see that most manufacturers, agencies, and professional organizations were silent on the issue, particularly at first. Once the popular verdict on EQC started to come in, more institutions were willing to join them. But quality seems to be a subject like politics in the US; everyone has an opinion on it, but usually they don't want to talk about it in public. Manufacturers face the market pressure to match whatever "cost saving" features their competitors offer. So if EQC means reduced QC costs, it's difficult for a single company to hold fast and insist that their customers need to spend more on QC. Since the professional organizations draw their support from manufacturer sponsorships, advertisements and other support, they have little incentive to discuss a topic that might offend them. The accrediting agencies, like JC, CAP, and COLA, were probably in the best position to halt the downward slide of quality. But they faced competitive pressures, too. If CAP was too strict on POC devices and EQC, labs could and would switch to the laxer standards of JC. Finally, when CMS decided to push EQC through, it simply flexed its deeming authority powers and forced the agencies to adopt some form of EQC or lose its accreditation status. Responsibility for quality has also been spread out until everyone and no one is responsible. Customers talk about quality but make decisions based on cost. Manufacturers struggle to maintain a balance of the two, but cost is the overwhelming driver of their business. The EQC protocols insist that the laboratory director must weigh the "clinical and legal responsibility" against the EQC options, a tacit admission that EQC probably is neither clinically nor legally appropriate. Everyone wishes quality was better, but acts in ways that reduces it.

The Bubble Pops

It's hard to summarize the factors that brought the end to speculation. But as interest rates rose, it became harder to take out loans. Housing prices began to drop, which meant you could no longer sell your house to pay off the mortgage. As housing values began to drop further, the value of some houses dropped below the mortgages (known as negative equity). People began to miss, even default entirely, on their mortgage payments, as teaser rates reset, as impossibly large payments came due on people who had no means to pay. As defaults rose, more houses came back onto the market, depressing the pricing. It turned into a negative feedback loop - more colorfully put, a death spiral.

Wall Street made it worse. Banks and hedge funds were highly leveraged in mortgage-backed securities (i.e. they had made all these investments using lots of borrowed money). Now their loans were coming due and their investments increasingly looked like they weren't worth the paper they were printed on. As banks and firms "wrote down" the value of these investments (admitted the money they had invested was gone), they began to run out of money. Some went bankrupt, as we've seen. Others have been taken over by the government. As fear gripped the financial community, there wasn't enough money left for the usual, normal and necessary loans - even those that aren't as risky as subprime mortgages. And here we are, looking at hundreds of billions, probably trillions of money, that will be needed to fix the mistakes.

Will the quality bubble ever pop for the laboratory? If we continue to debase the rules and loosen the regulations, and if our institutions acquiesce in this drift, it's probable that we will fall outside the margins of safety. If we reach the point where testing QC once a month becomes an accepted practice, it's probable we will see bad test results impacting large numbers of patients. The Maryland General scandal, which was the case of an individual laboratory with debased quality practices, is merely an hors d'oeuvre compared to the systemic problems we could face. I believe healthcare is more resilient than Wall Street. We believe more in mission than mere profit. Our professionals have more scruples and commitment to good practices. But all of that means that the errors can grow larger, the bubble can get bigger, before it overwhelms the system. A quality crisis is not inevitable. All we need is leadership. And even then, leadership isn't necessary if enough voices are raised. Your voice. The voices of laboratory professionals. If the chorus is loud enough, the leaders and the market and maybe even the rules will follow.

Offered in the spirit of a previous essay, Enron and Quality