The Westgard Rules

30 Years of Workshops!

Originally posted August 7th, 2006

One of the more significant milestones passed at the AACC/ASCLS conference was the 30th year of the "Concepts and Practices in the Evaluation of Laboratory Methods" workshop. This workshop, taught by Carl Garber, PhD, R. Neill Carey, PhD, and David Koch, PhD, is now the longest continuously taught workshop at the conference.

An article celebrating this anniversary was in the Monday "Convention Daily" of Clinical Laboratory News (Title: Workshops Offer Participants New Lab Tools and Skills: Instructors of 'Evaluation of Laboratory Methods' Mark 30th Year'). Unfortunately, the article is not available online (yet), so I will quote a short passage where Neill Carey and Carl Garber talk about the 'early years' of the workshop:

"'At the time, what we were suggesting was a fairly new concept,' Carey recalled. 'Before that point, people hadn't talked much about making decisions on the acceptability of methods based on measuring errors.' Garber explains that their workshop has always been intended to provide practical advice for laboratorians on the use - and the misuse - of statistics. 'Our focus has consistently been on clinical significance - where the rubber meets the road,' he said 'Through the years we've tried to help our colleagues better understand what the clinician requires in terms of allowable error and variation on a test result.'"

I worked with Carl and Neill and David when they worked at the University of Wisconsin Hospital and Clinics in Madison.  They are a part of that "Wisconsin mafia" that has helped keep quality and statistics at the center of laboratory testing. I must also admit I had a hand in developing this workshop, too. We published a series of papers  in the early 1970s on the "proper use of statistics in evaluating methods" and presented them in a workshop in 1976 at the American Society of Medical Technology. The next year, 1977, I was heading over to Uppsala, Sweden, for a sabbatical (where my interest in QC would begin and where the "Westgard Rules" would be developed). I recommended that Carl and Neill take the workshop to the AACC conference (David Koch would join them in 1982.). And they have taught that workshop ever since.

Over the years, they have applied continuous improvement to their Method Evaluation workshop, modifying, updating, and adding material. The workshop, now part of the new AACC University, is a tightly packed four hours that includes not only the basic statistical studies for method validation, but also Six Sigma, CLIA regulations and CLSI guidelines. There's probably no other workshop that is so stuffed with information. If you ever get a chance to come to the AACC conference, I highly recommend attending the workshop. You'll learn a lot - either something new or something you've forgotten.

What they've accomplished is the long distance marathon of conference programming. Few people would have the stamina, diligence, and determination to make a fresh presentation every year on this important topic. But Carl, Neill, and David have done it and I hope they continue to do it for decades to come.

So congratulations again.

Rumors of my retirement are somewhat exaggerated...

Originally posted August 2nd, 2006

Here is probably one of the best pictures of our time in Chicago - maybe one of the best pictures of my career. On the left, you see one of my first students at the University of Wisconsin, back in the 1970s. On the right, you see another one of my students, but from this year, 2006. Both were my students. Both are now working in the laboratory field here in Wisconsin. But can you guess another way these two are related?

That's right. They literally are related. Emily and Julie Kepner have both been my students of mine. It's a humbling experience, to have taught both mother and daughter. I hope it says something about what and how we teach at the University of Wisconsin. I hope it shows the strong values of work and ethics here in Wisconsin. But I must admit, it chiefly says something about my age.

At the AACC/ASCLS convention, there was a rumor going around that I had retired. Well, it's true. In mid-July, I officially retired from the University of Wisconsin-Madison. I am now an emeritus professor.

However, I have not retired from Westgard QC, which has been my "moonlighting" job for nearly fifteen years. In fact, it's a lucky thing that I retired from the University, because the business of Westgard QC has continued to grow so much - my traveling and consulting in particular - that my "day job" was starting to get in the way. Now that I'm "only" working one job, I hope to have more time to devote to Westgard QC and its books, courses, workshops and other services. So you may actually here more from me, not less. (For some, this will come as a disappointment.)

(I should also admit that I really downsized from three jobs to two, not one. I'm an employee of my wife Joan's antique business, Pieces of Time. She buys the antiques and I fix, refinish and transport them to our booth at the local antique mall. If you ever travel to Madison, we'd be happy to give you a tour. We expect to be adding new inventory soon.)

There's a lot I want to say about my time at the University of Wisconsin-Madison. The state university system in Wisconsin has been very good to me. But for now, I just want to say I'm thankful I could be a part of the public education system for so long. I'll write more on this in a future essay.

In the meantime, stay tuned. And if you want to contribute an anecdote to my upcoming retirement party, please feel free to contact us, or add a comment below.

Thanks for Visiting

Originally posted August 1st, 2006

We want to thank the hundreds of people who came to visit the Westgard QC booth at the AACC/ASCLS convention in Chicago last week (July 25-27).

This convention is always a rewarding experience for us. It helps us put faces to the people who visit, read, and embrace the website. For our visitors, it's a chance to look at our books in hardcopy (instead of reading free excerpts online), watch a live demonstration of the software, or just ask questions.

You would be amazed at the questions we get, some very simple and others quite complex. One visitor asked about rule interpretations of the 10:x rule for certain specific situations in their laboratory. Another person asked, "Is Dr. Westgard still alive?" - that's a question we get every year, actually. And we're always happy to dispel the rumors.

What encourages us is the fact that the energy and interest in quality is never lacking. In other fields, people turn their backs on quality, from the highest CEO to the lowest entry worker. But in the laboratory industry, people know they have to get their job done right the first time.

If you weren't at the conference, but have wanted to contact us about quality, now is the time. Email us at "westgard at westgard.com", call us at 608-833-0640, or just add a comment.

We want to hear from you.

GAO: The Top 14 States

Originally July 14, 2006.

One of the Appendices included in the GAO Report provies a list of the number of labs in each state that are inspected by state agencies, as well as the number of labs in that state that were sanctioned during the years 1998 to 2004. This state by state breakdown reveals some interesting details. For instance, it's clear that some states are either (a) blessed with perfect laboratories; or (b) aren't very tough when it comes to enforcing laboratory regulations.

1. Alaska (50 labs)
2. Connecticut (246 labs)
   
3. Delaware (46 labs)
4. Idaho (203 labs)
5. Kansas (279 labs)
6. Kentucky (386 labs)
7. Maine (89 labs)
8. Massachusetts (409 labs)
9. Mississippi (441 labs)
10. North Dakota (74 labs)
11. Oregon (270 labs)
12. South Carolina (315 labs)
13. Tennessee (705 labs)
14. Vermont (46 labs)

There you have it, fourteen states that have had no recent sanctions in any of their 3,559 laboratories. If you're lucky enough to reside in one of those states, congratulations. You're living in a place with world class healthcare.

Of course, that's just one of the conclusions you can draw about this data. The other conclusion - the one that the GAO seems to have chosen - is that some state agencies don't impose sanctions very often, and that the enforcement of regulations is not performed consistently across the country.

Which conclusion do you prefer?

GAO Recommendations

Originally posted July 12, 2006

The GAO developed thirteen recommendations based on their investigation of CMS, CAP, JCAHO, COLA and the other state agencies. The recommendations are aimed at CMS and what it should do and how it should change. Here's a quick list in plain English:

1. Standardize survey findings
2. Ensure consistent advance notice by state survey agencies
3. Ensure identifying deficiences comes before education
4. Impose sanctions on labs with consecutive failures
5. Require all survey organizations to require whistle-blower posters
6. Require quarterly proficiency testing
7. Ensure timely evaluation of survey organizations
8. Ensure that new survey rules are reviewed in a timely manner
9. Hire more staff
10. Perform more validation inspections of state survey organizations
11. Require more independent validation of survey organizations
12. Collect and review more survey data
13. Establish an enforcement database

Some of these recommendations are common sense. Some of them are bizarre. Most of them will do nothing to address the real problem with the quality of laboratory testing.

To read in detail - and see the recommendations in the original bureaucratese - read the essay we've posted.

Breaking: GAO Report on CMS is out

Originally posted June 30, 2008

Remember the Maryland General laboratory scandal of two years ago? Part of the outcome of the media circus and the Congressional Hearings on Maryland General was a request by Congress for an investigation of CMS by the GAO (Government Accountability Office). The GAO was asked to examine (1) the quality of lab testing; (2) the effectiveness of surveys, complaint investigations, and enforcement actions in detecting and addressing lab problems; and (3) the adequacy of CMS's CLIA oversight.

Well, that report is finally out - and it's not kind to CMS.

The titles say it all:

• Testimony to Congress: CMS and Survey Organization Oversight Is Not Sufficient to Ensure Lab Quality
• Full Report: CMS and Survey Organization Oversight Should Be Strengthened

Westgard Web will provide some detailed analysis of the reports (both what is in the reports and what got left out) in the coming days, but the first thing we want to do is encourage you to read the reports directly. Go straight to the source and see what they found, and see if their conclusions match your own. In July, we'll tell you what we think of the reports - and you can judge us as well.

Let me pull a few highlights just from the one page summary:

• "Because of limited comparable data from CMS and survey organizations, too little is known about the quality of lab testing."
• "....GAO’s analysis of an indicator that measures a lab's ability to consistently produce accurate test results suggests that lab quality may not have improved at hospital labs in recent years."
• "....real and potential lab quality problems are masked by survey, complaint, and enforcement weaknesses."
• "....other factors suggest that surveys and complaints do not present a realistic picture of lab quality."

In other words, we still don't know what we don't know. You haven't heard that before, have you?

Update on QC Design Tools

Originally posted June 20, 2008

Just a short note to point out a new

essay

on QC Design tools that we've posted.

As I tell people who attend the workshops, you can't just listen to the theory of QC Design. You've got to go back to your laboratories and try it yourself.

Over the years, I've been a part of many teams that developed QC Design tools. Every new tool seems to get better, more sophisticated, and yet easier to use. The ultimate goal is to give you a powerful tool that you can use with minimal effort.

Read the essay and come back here to make comments.

    

Indicating ourselves to death?

Originally posted June 16, 2008

You've probably noticed that new JCAHO National Patient Safety Goals are out. Some new goals have been added this year. These goals are some of the most prominent new measures, metrics, and indcators that are being introduced to healthcare. At the Westgard Workshops, Teresa Darcy, MD, MMM, gave a presentation on this recent proliferation of Quality Indicators.

Do you know how many Indicators are out there? Here's just a short list of what Dr. Darcy discussed:

• IQLM Indicators Workgroup: 12 core indicators
• CAP Lab General: 11 key indicators
• JCAHO National Patient Safety Goals: 15 goals so far, with 23 As, Bs, and Cs
• AHRQuality Program: 4 modules, 91 conditions, measures, and indicators
• NCQA Health Plan Employer and Data Set (HEDIS®) specifications: 7 volumes
• NQF Compendium 2000-2005: over 200 consensus standards
• International Quality Indicators Project: 5 care sections, 39 indicators, numerous subparts
• Quality Indicators Project: 5 sections, 42 indicators, numerous subparts

These committees, commissions, taskforces, and groups are also forming at the state level. As part of the Patient Safety movement, states are beginning to collect and publish the measurements of key events. Sometimes these are adverse events, other times they are performance measurements. In Wisconsin, for instance, there is now a Collaborative for Healthcare Quality, with 6 Aims, 9 Clinical Topics, 44 Measures, and 3 "Exclusive" Measures. All of these data points are used to rank the hospitals against their in-state competitors.

Do you see anything wrong with this picture? I didn't realize how many committees are working on quality indicators, so I found  Dr. Darcy's presentation eye-opening. As new committees are formed, I guess the old ones keep going. And more indicators and measures are created, more reports are compiled, and we hope that this somehow changes what's being done. Should we create a committee for meauring the impact of these committees?

There's no question these groups are all created with good intentions. Everyone is trying to improve medical care. They are all finding useful ways to determine the quality of care. But the law of diminishing returns kicks in at some point.

At some point, we may create an indicator death spiral: a hospital will spend more time measuring itself than serving the patient. You can see the punchline: "The operation was a success. The hospital is in compliance. Our indicators are good. And the patient died."

The lure of committees is strong. Creating a NEW COMMITTEE looks like you're taking action, when in reality, committees are slow-moving beasts, usually incremental in approach, and rarely have the power to change any uncomfortable realities.

One last point - out of all these indicators, can you guess how many directly discuss laboratory analytical quality?

Always the bridesmaid...

Westgard Workshops 2006, Part III: What's actually being done

Originally posted June 14, 2006.

I'm sitting in Copenhagen enjoying a beer right now. Those who attend the Westgard Workshops know that beer is an important part of the workshop experience, as well as the Wisconsin experience. I'm attending the Scandinavian Clinical Chemistry meeting right now. It's nice to be a participant of a meeting after running a meeting of our own. There's more time to relax and think..

One of the best parts of the Westgard Workshops this year was seeing the application and implementation of Six Sigma concepts in both industry and laboratories. In many cases, these were participants who attended the workshops several years ago, went back to their companies and implemented the concepts. This year, they came back to the workshops to present their results.

Dr. Gordon Kapke of Covance Central Laboratory Services discussed his efforts to measure pre-analytical errors. Covance runs a global operation with 8 laboratories on 6 continents. They have over 1 million reportables every month - so defects per million is not an abstract concept for them. Dr. Kapke showed how design and automation of their processes have reduced their defects. While Covance primarily serves the pharmaceutical development market, their laboratory processes offer many lessons to the clinical laboratory. Because of their size, Covance is not only introducing approximately 1 new method a day, they are also being inspected on average once a day (when it's not the regulatory inspectors, it's the clients checking up on projects). This constant change and scrutiny demands robust processes - Six Sigma techniques have proven their worth.

Tony Orzechowski of Abbott Laboratories discussed how Six Sigma design can be applied from the very beginning of instrument and assay design. Abbott is now injecting quality requirements into their earliest product development cycles. Not only do they assign error budgets to the assays, they also determine how much error their calibrators and control materials are allowed to have. The ideal goal is to deliver a method or a control lot with a Sigma metric attached to it. Tony uses OPSpecs charts to develop an operating "envelope" for Abbott methods. They can use those charts and "envelopes" to match up with customer experiences of performance, and possibly even predict when and what kinds of technical issues may occur. Truly at the cutting edge.

John Yundt-Pacheco of Bio-Rad also demonstrated how his company has embraced Sigma metrics in their products. Bio-Rad will soon be rolling out a service called Westgard Advisor. This program will complement their online peer group offerings and allow customers to easily (very easily) benchmark their performance, have Sigma metrics calculated for their methods, and receive instant QC recommendations. [Full disclosure: this was developed as part of a collaboration between Westgard QC and Bio-Rad. We have a vested interest in this product, if the name didn't tip you off.] John showed the results of Sigma medians for chemistry analytes from this peer data. The results are quite encouraging (he will have a poster at the AACC conference which will elaborate on this data) - many labs are getting high Sigmas on many of the automated chemistry tests. For many analytes, laboratories are in fact OVERcontrolling their performance. With the help of data-interfaced software, the opportunities for reduced QC effort (and savings) is made clear.

Finally, Dr. David Parry of St. Boniface Hospital in Winnipeg, Manitoba, Canada, talked about his experiences implementing Sigma metrics in his own hospital. The key lesson that he taught was that there is a significant impact on the bench techs when you rationally design your QC processes. The importance of reduced false alarms cannot be minimized.

I'm not doing justice to their presentations here. And there was much more discussed at the workshops. But you get the idea.

From industry to laboratory, Sigma metrics are being implemented with real results. These aren't the giant "Master Black Belt" programs that cost millions and focus on TAT and billing cycles - these are small projects that involve performance in the lab being done by "belt-less" individuals. You don't need a massive program, a huge training period, or huge resources (although they all help) - QC Design with Six Sigma can be done even if all you have is the will to do something. The power is yours.

Westgard Workshops 2006, Part II: What was unsaid

originally posted June 13, 2006

The first day of the Westgard Workshops covered Standards and Practices, Indicators and Guidelines.

We were lucky enough to have presentations by both CAP and JCAHO officials, detailing the latest changes in the checklists, guidelines, patient safety goals, etc. I can't do justice to all the details here. There are simply too many changes year to year to cover in a short note. What's frightening is that ever since the "final" CLIA rule in 2003, every year is bringing a huge set of changes. You would think that a laboratory would need to have a FTE devoted just to tracking the changes in the regulations.

Obviously, unannounced inspections were a hot topic. But even more interesting was what wasn't being said.

With the CLSI committees working on several new guidelines for both manufacturers and laboratories (EP22 and EP23), with CLIA still formulating what "Option 4" is going to be, this year is mainly about anticipation. Cross your fingers, because we are really hoping that the people who gave us the disastrous EQC options 1 through 3 and going to hit a home run this time.

An even more intriguing silence was the  impending GAO report. As you recall, in the wake of the Maryland General Hospital problems, several congressmen asked the GAO to evaluate how CMS, CAP and JCAHO were doing their jobs. This has been a multi-year evaluation.

Well evidently, the GAO has finished the report and it was supposed to be released to the public in early June. Our JCAHO presenter, Kathy Steffens mentioned that a draft report had been delivered in April. CAP has already responded. JCAHO has also responded. But CMS asked for an extended period of time to respond.

Overall, there is an expectation that this GAO report is going to be a disappointment. It will ask for changes, but many of those changes will not be useful. It may reach some strange conclusions. And the odds that analytical quality is going to be improved by this report are very slim.