The Westgard Rules

Dark Laboratories

By Sten Westgard, MS

This is the last thing I have to say about the CLSI and CMS update at the AACC/ASCLS conference in DC. There was one striking point by Judy Yost (Director, Division of Laboratory Services, Centers for Medicare & Medicaid Services) during her presentation (PDF link) on frequent laboratory deficiencies: Nearly 80% of laboratories in the US do not have any routine oversight

The slide above breaks down the most recent CMS information (December 2007) on the types of laboratories in the US. As you can see, there are 119,938 waived labs (labs that do only waived testing), and another 38,903 labs perform only Provider-Performed Microscopy testing. Since neither of these types of testing requires inspection, there is no routine oversight for nearly 160,000 laboratories.

In contrast, CLIA inspects 19,827 laboratories (called Compliance labs), and the Joint Commission, CAP, and other "deemed" providers inspect another 16,098 laboratories (the Accredited labs). Finally, since New York and Washington state are exempt from the CLIA rules, those states inspect the remaining 6,000 laboratories. All told, there are about 41,925 laboratories that do get some form of routine oversight.

When I heard this, I was reminded of the physics concept called Dark Matter.

Dark Matter is term coined by astronomers. Its meaning is pretty simple: it is matter and it is dark. Specifically, dark matter is matter "that does not clump together in stars and hence does not give off light and that thus exerts a gravitational pull without revealing itself visibly."¹ Basically, we, here on earth, cannot observe Dark Matter in the universe. We can only infer that it exists because we see its effects: without Dark Matter, the universe would fly apart. There is so much Dark Matter, it helps hold the universe together. Less than 5% of the universe is visible matter (stars, planets, gases, and other things we can see with various kinds of telescopes and observation technologies), while Dark Matter is close to 25% of the universe.² So there is five times more stuff we can't see than stuff we can.

It seems that we have reached the same situation in healthcare laboratories. While we argue vociferously about regulations and complexity and validation and quality control in those nearly 42,000 laboratories that are regulated, but there are four times as many laboratories that have almost no supervision and we don't talk about them even though they are majority of testing facilities. The increased scrutiny and requirements of regulated labs may have driven more and more laboratories to the "dark side", since waived testing allows them to escape tough requirements for testing. This would be a good thing, if we knew that waived laboratories were operating correctly.

But we don't really know what waived laboratories are doing. Without any routine oversight, we have no data to look at, so we can really only guess at their performance.

The few studies of waived laboratories have not been encouraging. The GAO report on laboratory testing found that between 5 and 10% of laboratories did not follow manufacturer's instructions, and about 5% did not run two levels of QC.³

A 2002-2004 survey of waived laboratories found that 12% did not have manufacturer's instructions, 21% did not perform QC, 45% did not document the name, lot number and expiration date for all tests performed, and 35% did not maintain a quality control log at all.⁴

The CMS presentation listed the top 10 deficiencies of laboratories as of 2007. Here they quote the specific rule that is being violated:

1. 493.1236--Alternative assessment for test not in PT (5.9%)
2. 493.1252--Lab must select test system (5.7%)
3. 493.1252—Lab defines storage conditions (5.6%)
4. 493.1289—Analytic systems QA (5.6%)
5. 493.1251—procedure manual content (5.0%)
6. 493.1239—General lab systems QA (4.2%)
7. 493.1407—Lab director establishes QA plan (4.2%)
8. 493.1291—Lab report content (4.0%)
9. 493.1252—No expired reagents (3.8%)
10. 493.1403—Unqualified lab director (3.5%)

The most recent numbers appear to represent improvements in waived laboratory performance. Nevertheless, even these lowered rates are not low enough. The problems we see in waived laboratories are larger than what we see in more professional laboratories. Even if only 5.6% of the labs aren't doing their QA and QC, that could mean that about 7,000 labs in the country are running without proper quality control in place.

When the CDC issued its report on Laboratory Medicine: A National Status Report, it also issued a report on the Quality of Proficiency Testing. In that second report, the authors made the extraordinary recommendation that all waived labs should have to perform proficiency testing - a major change which would require an modification to existing law. Those authors clearly saw that waived testing needs to be brought back into the regulatory fold, because too many errors are happening in places where we are blind.

It is unlikely that the regulatory, industry, or political will exists to make such a change to the US laws. And until then, unfortunately, a lot of laboratories will continue to operate in the dark...

Further articles on this topic:

• The CLSI description of the meeting
  
• Update on EP22, EP23, and "Alternative QC"
• The Quality of Proficiency Testing

References

1. Brian Greene, The Fabric of the Cosmos, 2005, Vintage Books (New York, NY), p.295
2. You probably note that 5+25 does not equal 100. Evidently, even the existence of Dark Matter doesn't explain the make up of the universe. The other 70% of the universe is made up of Dark Energy. I don't know what that is, and it's even less relevant to this discussion, so let's not go there. Wiki Dark Matter if you're curious.
3. Memorandum from Mark McClellan, CMS Administrator, to Leslie Aronovitz, GAO, May 17, 2006. Baltimore, MD; Centers for Medicare and Medicaid Services, 2006. Quoted on page 284 in Laboratory Medicine: A National Status Report, a report prepared for the Division of Laboratory Systems, National Center for Preparedness, Detection, and Control of Infectious Diseases, of the Centers for Disease Control and Prevention.
4. Howerton D, Anderson N, Bosse D, et al. Good Laboratory practices for waived testing sites: survey findings from testing sites holding a certification of waiver under CLIA 88 and recommendations for promoting quality testing. MMWR Recommendations and Reports 2005;54(RR-13):1-25. As quoted on page 278 in Laboratory Medicine: A National Status Report, a report prepared for the Division of Laboratory Systems, National Center for Preparedness, Detection, and Control of Infectious Diseases, of the Centers for Disease Control and Prevention.

EP22: What are the Risks of Communicating Risks?

Posted by Sten Westgard

While Dr. Westgard has given a thorough update on the status of EP22 and EP23, as presented at the AACC/ASCLS conference in Washington, DC, and Dr. Jan Krouwer has done the same on his own blog, I thought it worth beating the dead horse on this.

Dr. Jan Krouwer was there at the presentation and asked perhaps the most important question about EP22 since it was created. Is the FDA going to review the Risk information created by manufacturers about their instruments?

I think the answer was somewhat nuanced and subject to interpretation. According to Dr. Krouwer, he heard that the FDA would review EP22 Risk information, but only superficially, and "only egregious problems would be flagged by the FDA."

Basically, I heard the same thing, with a slightly different interpretation: the FDA isn't really going to review the Risk information when the device is first submitted to them. But retrospectively, it might review them very thoroughly. Since the FDA has authority over labeling and branding of medical devices, if it turns out that some of the risk information from a manufacturer is incorrect, inaccurate, or misleading, than the FDA retains the power to sanction the manufacturer.

In other words, manufacturers have the right to remain silent (i.e. don't produce any Risk information, since EP22 is voluntary), and anything they say (i.e. the EP22 Risk information) can be used against them in a court of law.

Call it the Manufacturer's Miranda Risk Rights.

In the absence of a legal mandate, I don't see why any manufacturer would willingly create this Risk information and distribute it to their customers. Any corporate lawyer will be advising against it (liability exposure). Any marketer will argue against it (if our competitors aren't saying they have residual risks, why do we need to do so?). Any executive will oppose it (why are we exposing our weaknesses to our competitors and our customers?). And that's the just the internal opposition this will face.

Certainly, I believe disclosure of Risk is the right thing to do, but the competitive and legal environment are not going to be receptive to it. Unless customers overwhelmingly demand this information, manufacturers will have no incentive - and many disincentives - to produce Risk information.

As Dr. Krouwer laments, there is a lot of good work in the EP22 and EP23 guidelines, but because of the toxic environment, these standards are likely to be stillborn.

Yet another sorry milepost in the story of EQC, AQC and Option 4.

The 2008 "Westgard Wear" Raffle winners are...

posted by Sten Westgard

Every year at the AACC/ASCLS conference, we hold a "Westgard Wear" raffle, a chance for someone to win the rare, obscure, and occasionally treasured hat, shirt or messenger bag with the Westgard logo on it. (Below, you can see the staff modeling some of the items)

This year, our winners were...

• Christopher Dukes of BayHealth, Delaware
• Elizabeth Madden of the University of Washington
• Janet Means of Boston Medical Center, Masachusetts

Congratulations, winners! You now can wear something that will make "normal" people look at you and scratch their heads. Of course, we hope other people in the laboratory might think it's not so bad.

What's that on the right side of this page?

posted by Sten Westgard

As you can no doubt see, there's more than one new feature of the Westgard Blog. In addition to the new look, new features, and new organization, there are some Google AdSense text ads along the right column.

That's right, the world has ended. There is now some advertising on Westgard Web.

This doesn't mean that visitors to Westgard Web will be deluged with pop-ups, pop-unders, and other attacking ads. But it does mean that Westgard QC is continuing to look at different ways to fund the website. As we commented many years ago in the essay Quality and Commerce, it's not possible for us to give everything away. We have to find ways to generate revenue to sustain the website, to allow us to continue to add to the website's offerings.

For now, ads will be confined to the blog. We will watch and see if limited advertising in limited sections of Westgard Web enables us to sustain the website. Given the omnipresent marketing in all the other publications and events in the laboratory marketplace, we think you will probably survive the exposure to a few more ads. We continue to believe that the best policy is to allow as many people as possible to have access to the important lessons in Quality Control.

If you see an ad that is inappropriate, or most importantly, offensive, do let us know. We have some ability to block what ads Google decides to place on the blog.

As always, we welcome your comments on this. If you think we've made a major error, or broken faith with you, do let us know.

The Blog is Back

posted by Sten Westgard

Yes, the Westgard blog is back.

Regular visitors will remember that back in 2006, Westgard QC dipped its toe in the blog water, but after a dozen or so postings, the blog went silent.

It turns out, the technology we were using to run the blog was not that robust or user-friendly. And it was particularly vulnerable to SPAM, with hundreds, if not thousands, of scripts attempting to comment on the blog with links to frightening websites. That forced us to be very strict on comments so we could be certain only genuine comments could go through. The combination of these problems meant that the blog was really not usable for our purpose. We ceased posting but kept it barely alive. The revamping of the blog went on the To-Do list. Unfortunately, that list was so long it took a few years to get done.

Now, we're using a better blog platform, with more capabilities and features than before. This should allow better posting, better commenting, less SPAM, and easier navigation. (Also note, the previous posts from the old blog have been transferred over, so you still have access to the content.)

The blog is always going to me an adjunct to the main westgard website, www.westgard.com. But we'll use this to point to new articles and events, as well as make shorter comments and points about topics of interest to the laboratory community. There are a lot of issues and news stories that warrant a short note, but aren't really worth an entire essay.

In short, we hope that we'll be able to post more stuff and do it faster.

Thanks for your continued readership of the website and the blog. We hope we can continue to improve its offerings.

Don't forget to make your thoughts known in the comments.

Thanks for visiting Westgard at AACC/ASCLS 2008!

posted by Sten Westgard

We were happy to see the hundreds of people who visited the booth or attended the lectures given by Dr. Westgard and Sten Westgard. If you weren't able to attend the convention, you can find out quite a bit by reading our reviews and follow-ups from the show.

For those of us who have been attending conventions for many years, the convention is almost like a family reunion. You get to see colleagues that you won't see any other time during the year. Ironically, we even saw people who live close enough to be our neighbors in Wisconsin, but nevertheless we both traveled all the way to DC and met there instead.

One of the most important updates was on the CLSI guidelines  EP22 and EP23. These committee were charged, several years ago, with the formulation of an acceptable path to "alternate QC" also known as "Option 4." It's been a long process. Dr. Westgard discusses the latest developments here.

Molecular Diagnostic QC?

Originally posted on October 3rd, 2006.

We don't often talk about molecular diagnostics and quality control. Getting a handle on the "usual" tests in chemistry, hematology, etc., has been a tough enough job. But the field of molecular diagnostics is expanding rapidly. And quality for those tests is going to be just as important, if not more important in the future.

 
Here's a simple description of a quality issue in molecular diagnostics testing: picture a tube or a well, or in the case of multiplex testing, several tubes or an array of wells. Each entity contains several different reactions to determine the presence or absence of specific genetic sequences. The current practice is to test just one or two of the reactions for quality control. And even if the QC reaction has different detection probes or primers than all the others reactions, if the value for the control is "in", it is assumed that all of the other results are valid.

 
This has been the state of the art for a limited number of assays and it seems risky at best. But these array tests are growing by orders of magnitude. Where now there may be under 100 tests in an array, new ones that will have several hundred, maybe a thousand, are under development. The validity of the current state of the art for QC, that is, to rely on testing just a small sampling of the different reactions, needs to be examined.

 
Dr. Clark Rundell, from Maine Molecular Diagnostics, is at the forefront of the QC field for molecular diagnostics. He recently wrote an article for MLO on the "QC challenges for molecular testing." It's worth a look - this is going to be one of the key early articles for this important area.

http://www.mlo-online.com/articles/0506/0506special_feature.pdf

Permission given by Medical Laboratory Observer, May 2006
Copyright (symbol) 2006 by Nelson Publishing Inc *www.mlo-online.com

Thank you...

Originally posted September 26, 2006

There are not enough thanks for my colleagues, friends and family who gathered to celebrate (and to a certain extent, roast) my retirement from the University of Wisconsin.

Last week, September 21st, I was lucky enough to be surrounded by my children and grandchildren, old friends, and valued colleagues. As those of you with Scandinavian background know, we are not used to compliments and in fact we grow uncomfortable when we hear our own name praised. Listening to stories of my early career, or tales of my parenting, I can only reflect on the blessings of my life and be very, very thankful.

Happier still was a reunion of the “Wisconsin Mafia” ­ a collection of brilliant scientists with whom I’ve had the pleasure of working over the years. Ron Laessig, Merle Evenson, Ian Carlson, all now retired; Neill Carey, now of Peninsula Regional Hospital, Carl Garber, now of Quest Diagnostics; David Koch, now of Emory University; and Sharon Ehrmeyer, Don Wiebe and Terri Darcy, still at UW. Ron, Merle, and Ian were collegues and collaborators in my early scientific work; Neill, Carl, and Dave worked with me in Clinical Chemistry and are the fellows who continue the labor of the annual “Method Validation” workshop at the AACC conference. Sharon and Terri have been especially supportive of my teaching and educational efforts. Also present were a number of Clinical Lab Scientists who were influential in putting some of my ideas and theories into practice, particularly Trish Barry and Elsa Quam at UW Hospital. Don Wiebe emcee'd the event - he's always provided humor that makes life and work enjoyable and he kept us laughing that night.

It is increasingly rare that one can work for a single institution for your entire career. But I was fortunate to find a place in the University of Wisconsin, a great public institution that took its job seriously. The UW takes care of its students, its educators, and its patients. Even under withering budget pressure from less charitable legislators, it endures in its mission. One of the reasons why Wisconsin is so great is because we give our citizens a great education. It has produced an amazing set of professionals who have made a huge impact on the practice of laboratory medicine. I am proud to be in their number.

Jack Levine, an old colleague from Bayer Diagnostics, gave a speech entitled “The Wisconsin Conundrum” in which described how he met both me and Dr. Laessig early in our careers. We all worked together on some very early and important papers, which frankly laid the foundations of how we use analytical instrumentation in the laboratory. Over the years, Jack and Ron and I managed to do pretty well for ourselves. But Jack outlasted us. He is still working at Bayer (which is now going to be Siemens) with more than 40 years under his belt.

There are more names, more colleagues, more memories than I can describe right now. The frenetic pace of life gives us few moments to stop and take stock of our lives, family, and friends. For me that night was one of those golden moments. So let me again say thank you.

Just one final note: I still find myself going into work. My office space hasn’t been reoccupied yet, so there is still a space where I can go in and tackle some issues.

You’ll hear more from me soon…

            

Labs are Vital

Originally posted on August 8th, 2006

One last report on the AACC/ASCLS conference.

What do you get when you combine techno music, pulsing neon light, flashing LED badges, Lance Armstrong-style rubber wristbands, a free buffet, and five open bars?

Labs Are Vital. The Abbott launch of a truly important initiative.

Jeffrey R. Binder, President of Abbott Diagnostics and Senior Vice President of Operations for Abbott, announced a new program to address the biggest problem facing the healthcare laboratory: the coming shortage of qualified laboratory workers.

In the last 25 years, more than 600 schools and university programs for medical technology have closed. In 2012 we will need 138,000 laboratory scientists, but at best there will be 42,000 available. (source) Why? Low salaries for workers mean less interest in the profession. Worse still, the schools and programs face high expenses. Providing instruments and other devices necessary to properly train technologists is not cheap.

Here's where Labs are Vital comes in. At this event, Abbott announced a new $1 million dollar donation program - schools and programs can apply for free instruments, reagents, and service. AACC past president, Stephen Kahn, PhD and Bernie Bekken, President of ASCLS, were present to welcome this new effort.

It's good to see that Abbott recognizes a critical reality: While Abbott may be comfortably profitable, their future profits are in jeopardy if there aren't enough workers to run their instruments. Not only do the schools need to be supported, but the profession itself needs to be supported. Labs and lab workers need some better public relations - and they need to get out of the basement. In the increasingly cost-stressed healthcare system, the anonymous role of the laboratory worker means out of sight, out of compensation. Labs Are Vital hopes to raise the profile of the laboratory worker and make more of them.

Abbott noted that the Labs are Vital program would be non-branded and invited participation by other diagnostic companies. This part is key. For any real initiative to succeed, it can't simply be a marketing effort by a single company. It's an easy PR move for a company to make a donation. But if the initiative is strongly identified with just one company, there is less incentive for other companies to participate.

How will the other companies react? Will they join the effort? Will they create their own? We'll see.