We don’t often talk about molecular diagnostics and quality
control. Getting a handle on the “usual” tests in chemistry,
hematology, etc., has been a tough enough job. But the field of molecular
diagnostics is expanding rapidly. And quality for those tests is going to be
just as important, if not more important in the future.
Here’s a simple description of a quality issue in molecular
diagnostics testing: picture a tube or a well, or in the case of multiplex
testing, several tubes or an array of wells. Each entity contains several
different reactions to determine the presence or absence of specific genetic
sequences. The current practice is to test just one or two of the reactions for
quality control. And even if the QC reaction has different detection probes or
primers than all the others reactions, if the value for the control is
“in”, it is assumed that all of the other results are valid.
This has been the state of the art for a limited number of
assays and it seems risky at best. But these array tests are growing by orders
of magnitude. Where now there may be under 100 tests in an array, new ones that
will have several hundred, maybe a thousand, are under development. The
validity of the current state of the art for QC, that is, to rely on testing
just a small sampling of the different reactions, needs to be examined.
Dr. Clark Rundell, from Maine Molecular Diagnostics, is at
the forefront of the QC field for molecular diagnostics. He recently wrote an
article for MLO on the “QC challenges for molecular testing.” It’s
worth a look - this is going to be one of the key early articles for this
important area.
http://www.mlo-online.com/articles/0506/0506special_feature.pdf
Permission given by Medical
Laboratory Observer, May 2006
Copyright (symbol) 2006 by Nelson
Publishing Inc *www.mlo-online.com
There
are not enough thanks for my colleagues, friends and family who
gathered to celebrate (and to a certain extent, roast) my retirement
from the University of Wisconsin.
Last week,
September 21st, I was lucky enough to be surrounded by my children and
grandchildren, old friends, and valued colleagues. As those of you with
Scandinavian background know, we are not used to compliments and in
fact we grow uncomfortable when we hear our own name praised. Listening
to stories of my early career, or tales of my parenting, I can only
reflect on the blessings of my life and be very, very thankful.
Happier
still was a reunion of the “Wisconsin Mafia” a collection of
brilliant scientists with whom I’ve had the pleasure of working over
the years. Ron Laessig, Merle Evenson, Ian Carlson, all now retired;
Neill Carey, now of Peninsula Regional Hospital, Carl Garber, now of
Quest Diagnostics; David Koch, now of Emory University; and Sharon
Ehrmeyer, Don Wiebe and Terri Darcy, still at UW. Ron, Merle, and Ian
were
collegues and collaborators in my early scientific work; Neill, Carl,
and Dave worked with me in Clinical Chemistry and are the fellows who
continue the labor of the annual “Method Validation” workshop at the
AACC conference. Sharon and Terri have been especially supportive of my
teaching and educational efforts. Also present were a number of
Clinical Lab Scientists who were influential in putting some of my
ideas and theories into practice, particularly Trish Barry and Elsa
Quam at UW Hospital. Don Wiebe emcee’d the event - he’s always provided
humor that makes life and work enjoyable and he kept us laughing that
night.
It is increasingly rare that
one can work for a single institution for your entire career. But I was
fortunate to find a place in the University of Wisconsin, a great
public institution that took its job seriously. The UW takes care of
its students, its educators, and its patients. Even under withering
budget pressure from less charitable legislators, it endures in its
mission. One of the reasons why Wisconsin is so great is because we
give our citizens a great education. It has produced an amazing set of
professionals who have made a huge impact on the practice of laboratory
medicine. I am proud to be in their number.
Jack
Levine, an old colleague from Bayer Diagnostics, gave a speech entitled
“The Wisconsin Conundrum” in which described how he met both me and Dr.
Laessig early in our careers. We all worked together on some very early
and important papers, which frankly laid the foundations of how we use
analytical instrumentation in the laboratory. Over the years, Jack and
Ron and I managed to do pretty well for ourselves. But Jack outlasted
us. He is still working at Bayer (which is now going to be Siemens)
with more than 40 years under his belt.
There are
more names, more colleagues, more memories than I can describe right
now. The frenetic pace of life gives us few moments to stop and take
stock of our lives, family, and friends. For me that night was one of
those golden moments. So let me again say thank you.
Just
one final note: I still find myself going into work. My office space
hasn’t been reoccupied yet, so there is still a space where I can go in
and tackle some issues.
You’ll hear more from me soon…
One last report on the AACC/ASCLS conference.
What
do you get when you combine techno music, pulsing neon light, flashing
LED badges, Lance Armstrong-style rubber wristbands, a free buffet, and
five open bars?
Labs Are Vital. The Abbott launch of a truly important initiative.
Jeffrey R. Binder, President of Abbott Diagnostics and Senior Vice
President of Operations for Abbott, announced a new program to address
the biggest problem facing the healthcare laboratory: the coming
shortage of qualified laboratory workers.
In the last 25 years, more than 600 schools and university programs for
medical technology have closed. In 2012 we will need 138,000 laboratory
scientists, but at best there will be 42,000 available. (source)
Why? Low salaries for workers mean less interest in the profession.
Worse still, the schools and programs face high expenses. Providing
instruments and other devices necessary to properly train technologists
is not cheap.
Here’s where Labs are Vital comes in. At this event, Abbott announced a
new $1 million dollar donation program - schools and programs can apply
for free instruments, reagents, and service. AACC past president,
Stephen Kahn, PhD and Bernie Bekken, President of ASCLS, were present
to welcome this new effort.
It’s good to see that Abbott recognizes a critical reality: While
Abbott may be comfortably profitable, their future profits are in
jeopardy if there aren’t enough workers to run their instruments. Not
only do the schools need to be supported, but the profession itself
needs to be supported. Labs and lab workers need some better public
relations - and they need to get out of the basement. In the
increasingly cost-stressed healthcare system, the anonymous role of the
laboratory worker means out of sight, out of compensation. Labs Are
Vital hopes to raise the profile of the laboratory worker and make more
of them.
Abbott noted that the Labs are Vital program would be non-branded and
invited participation by other diagnostic companies. This part is key.
For any real initiative to succeed, it can’t simply be a marketing
effort by a single company. It’s an easy PR move for a company to make
a donation. But if the initiative is strongly identified with just one
company, there is less incentive for other companies to participate.
How will the other companies react? Will they join the effort? Will they create their own? We’ll see.
One of the more significant milestones passed at the AACC/ASCLS
conference was the 30th year of the “Concepts and Practices in the
Evaluation of Laboratory Methods” workshop. This workshop, taught by
Carl Garber, PhD, R. Neill Carey, PhD, and David Koch, PhD, is now the
longest continuously taught workshop at the conference.
An article celebrating this anniversary was in the Monday “Convention Daily” of Clinical Laboratory News
(Title: Workshops Offer Participants New Lab Tools and Skills:
Instructors of ‘Evaluation of Laboratory Methods’ Mark 30th Year’).
Unfortunately, the article is not available online (yet), so I will
quote a short passage where Neill Carey and Carl Garber talk about the
‘early years’ of the workshop:
recalled. ‘Before that point, people hadn’t talked much about making
decisions on the acceptability of methods based on measuring errors.’
Garber explains that their workshop has always been intended to provide
practical advice for laboratorians on the use - and the misuse - of
statistics. ‘Our focus has consistently been on clinical significance -
where the rubber meets the road,’ he said ‘Through the years we’ve
tried to help our colleagues better understand what the clinician
requires in terms of allowable error and variation on a test result.’”
I worked with Carl and Neill and David when they worked at the
University of Wisconsin Hospital and Clinics in Madison. They are
a part of that “Wisconsin mafia” that has helped keep quality and
statistics at the center of laboratory testing. I must also admit I had
a hand in developing this workshop, too. We published a series of
papers in the early 1970s on the “proper use of statistics in
evaluating methods” and presented them in a workshop in 1976 at the
American Society of Medical Technology. The next year, 1977, I was
heading over to Uppsala, Sweden, for a sabbatical (where my interest in
QC would begin and where the “Westgard Rules” would be developed). I
recommended that Carl and Neill take the workshop to the AACC
conference (David Koch would join them in 1982.). And they have taught
that workshop ever since.
Over the years, they have applied continuous improvement to their
Method Evaluation workshop, modifying, updating, and adding material.
The workshop, now part of the new AACC University, is a tightly packed
four hours that includes not only the basic statistical studies for
method validation, but also Six Sigma, CLIA regulations and CLSI
guidelines. There’s probably no other workshop that is so stuffed with
information. If you ever get a chance to come to the AACC conference, I
highly recommend attending the workshop. You’ll learn a lot - either
something new or something you’ve forgotten.
What they’ve accomplished is the long distance marathon of conference
programming. Few people would have the stamina, diligence, and
determination to make a fresh presentation every year on this important
topic. But Carl, Neill, and David have done it and I hope they continue
to do it for decades to come.
So congratulations again.
Here
is probably one of the best pictures of our time in Chicago - maybe one
of the best pictures of my career. On the left, you see one of my first
students at the University of Wisconsin, back in the 1970s. On the
right, you see another one of my students, but from this year, 2006.
Both were my students. Both are now working in the laboratory field
here in Wisconsin. But can you guess another way these two are related?
That’s right. They literally are related.
Emily and Julie Kepner have both been my students of mine. It’s a
humbling experience, to have taught both mother and daughter. I hope it
says something about what and how we teach at the University of
Wisconsin. I hope it shows the strong values of work and ethics here in
Wisconsin. But I must admit, it chiefly says something about my age.
At the AACC/ASCLS convention, there was a rumor going around that I had retired. Well, it’s true. In mid-July, I officially retired from the University of Wisconsin-Madison. I am now an emeritus professor.
However, I have not retired
from Westgard QC, which has been my “moonlighting” job for nearly
fifteen years. In fact, it’s a lucky thing that I retired from the
University, because the business of Westgard QC has continued to grow
so much - my traveling and consulting in particular - that my “day job”
was starting to get in the way. Now that I’m “only” working one job, I
hope to have more time to devote to Westgard QC and its books, courses,
workshops and other services. So you may actually here more from me, not less. (For some, this will come as a disappointment.)
(I should also admit that I really downsized from three jobs to two,
not one. I’m an employee of my wife Joan’s antique business, Pieces of Time.
She buys the antiques and I fix, refinish and transport them to our
booth at the local antique mall. If you ever travel to Madison, we’d be
happy to give you a tour. We expect to be adding new inventory soon.)
There’s a lot I want to say about my time at the University of
Wisconsin-Madison. The state university system in Wisconsin has been
very good to me. But for now, I just want to say I’m thankful I could
be a part of the public education system for so long. I’ll write more
on this in a future essay.
In the meantime, stay tuned. And if you want to contribute an anecdote
to my upcoming retirement party, please feel free to contact us, or add
a comment below.
We
want to thank the hundreds of people who came to visit the Westgard QC
booth at the AACC/ASCLS convention in Chicago last week (July 25-27).
This convention is always a rewarding experience for us. It helps us
put faces to the people who visit, read, and embrace the website. For
our visitors, it’s a chance to look at our books in hardcopy (instead
of reading free excerpts online), watch a live demonstration of the
software, or just ask questions.
You would be amazed at the questions we get, some very simple and
others quite complex. One visitor asked about rule interpretations of
the 10:x rule for certain specific situations in their laboratory.
Another person asked, “Is Dr. Westgard still alive?” - that’s a
question we get every year, actually. And we’re always happy to dispel
the rumors.
What encourages us is the fact that the energy and interest in quality
is never lacking. In other fields, people turn their backs on quality,
from the highest CEO to the lowest entry worker. But in the laboratory
industry, people know they have to get their job done right the first
time.
If you weren’t at the conference, but have wanted to contact us about
quality, now is the time. Email us at “westgard at westgard.com”, call
us at 608-833-0640, or just add a comment.
We want to hear from you.
One of the Appendices included in the GAO Report provies a list of the
number of labs in each state that are inspected by state agencies, as
well as the number of labs in that state that were sanctioned during
the years 1998 to 2004. This state by state breakdown reveals some
interesting details. For instance, it’s clear that some states are
either (a) blessed with perfect laboratories; or (b) aren’t very tough
when it comes to enforcing laboratory regulations.
- Alaska (50 labs)
- Connecticut (246 labs)
- Delaware (46 labs)
- Idaho (203 labs)
- Kansas (279 labs)
- Kentucky (386 labs)
- Maine (89 labs)
- Massachusetts (409 labs)
- Mississippi (441 labs)
- North Dakota (74 labs)
- Oregon (270 labs)
- South Carolina (315 labs)
- Tennessee (705 labs)
- Vermont (46 labs)
There you have it, fourteen states that have had no recent sanctions in
any of their 3,559 laboratories. If you’re lucky enough to reside in
one of those states, congratulations. You’re living in a place with
world class healthcare.
Of course, that’s just one of the conclusions you can draw about this
data. The other conclusion - the one that the GAO seems to have chosen
- is that some state agencies don’t impose sanctions very often, and
that the enforcement of regulations is not performed consistently
across the country.
Which conclusion do you prefer?
The GAO developed thirteen recommendations based on their investigation
of CMS, CAP, JCAHO, COLA and the other state agencies. The
recommendations are aimed at CMS and what it should do and how it
should change. Here’s a quick list in plain English:
- Standardize survey findings
- Ensure consistent advance notice by state survey agencies
- Ensure identifying deficiences comes before education
- Impose sanctions on labs with consecutive failures
- Require all survey organizations to require whistle-blower posters
- Require quarterly proficiency testing
- Ensure timely evaluation of survey organizations
- Ensure that new survey rules are reviewed in a timely manner
- Hire more staff
- Perform more validation inspections of state survey organizations
- Require more independent validation of survey organizations
- Collect and review more survey data
- Establish an enforcement database
Some of these recommendations are common sense. Some of them are
bizarre. Most of them will do nothing to address the real problem with
the quality of laboratory testing.
Remember the Maryland General laboratory scandal
of two years ago? Part of the outcome of the media circus and the
Congressional Hearings on Maryland General was a request by Congress
for an investigation of CMS by the GAO (Government Accountability
Office). The GAO was asked to examine (1) the quality of lab testing;
(2) the effectiveness of surveys, complaint investigations, and
enforcement actions in detecting and addressing lab problems; and (3)
the adequacy of CMS’s CLIA oversight.
Well, that report is finally out - and it’s not kind to CMS.
The titles say it all:
Testimony to Congress: CMS and Survey Organization Oversight Is Not Sufficient to Ensure Lab Quality
Full Report: CMS and Survey Organization Oversight Should Be Strengthened
Westgard Web will provide some detailed analysis of the reports (both
what is in the reports and what got left out) in the coming days, but
the first thing we want to do is encourage you to read the reports directly.
Go straight to the source and see what they found, and see if their
conclusions match your own. In July, we’ll tell you what we think of
the reports - and you can judge us as well.
Let me pull a few highlights just from the one page summary:
- “Because of limited comparable data from CMS and survey organizations, too little is known about the quality of lab testing.”
- “….GAO’s analysis of an indicator that measures a lab’s
ability to consistently produce accurate test results suggests that lab
quality may not have improved at hospital labs in recent years.” - “….real and potential lab quality problems are masked by survey, complaint, and enforcement weaknesses.”
- “….other factors suggest that surveys and complaints do not present a realistic picture of lab quality.”
You haven’t heard that before, have you?
Just a short note to point out a new
As I tell people who attend the workshops, you can’t just listen to the theory of QC Design.
You’ve got to go back to your laboratories and try it yourself.
Over the years, I’ve been a part of many teams that developed QC
Design tools. Every new tool seems to get better, more sophisticated,
and yet easier to use. The ultimate goal is to give you a powerful tool
that you can use with minimal effort.
You’ve probably noticed that new JCAHO National Patient Safety
Goals
are out. Some new goals have been added this year. These goals are some
of the most prominent new measures, metrics, and indcators that are
being introduced to healthcare. At the Westgard Workshops, Teresa
Darcy, MD, MMM, gave a
presentation on this recent proliferation of Quality Indicators.
Do you know how many Indicators are out there? Here’s just a short list
of what Dr. Darcy discussed:
- IQLM Indicators Workgroup: 12 core indicators
- CAP Lab General: 11 key indicators
- JCAHO National Patient Safety Goals: 15 goals so far, with 23 As, Bs, and Cs
- AHRQuality Program: 4 modules, 91 conditions, measures, and indicators
- NCQA Health Plan Employer and Data Set (HEDIS®) specifications: 7 volumes
- NQF Compendium 2000-2005: over 200 consensus standards
- International Quality Indicators Project: 5 care sections, 39 indicators, numerous subparts
- Quality Indicators Project: 5 sections, 42 indicators, numerous subparts
These committees, commissions, taskforces, and groups are also forming
at the state level. As part of the Patient Safety movement, states are
beginning to collect and publish the measurements of key events.
Sometimes these are adverse events, other times they are performance
measurements. In Wisconsin, for instance, there is now a Collaborative
for Healthcare Quality, with 6 Aims, 9 Clinical Topics, 44 Measures,
and 3 “Exclusive” Measures. All of these data points are used to rank
the hospitals against their in-state competitors.
Do you see anything wrong with this picture? I didn’t realize
how many
committees are working on quality indicators, so I found Dr.
Darcy’s presentation eye-opening. As new committees are formed, I guess
the old ones
keep going. And more indicators and measures are created, more reports
are compiled, and we hope that this somehow changes what’s being done.
Should we create a committee for meauring the impact of these
committees?
There’s no question these groups are all created with good intentions.
Everyone is trying to improve medical care. They are all finding useful
ways to determine the quality of care. But the law of diminishing
returns kicks in at some point.
At some point, we may create an indicator death spiral: a hospital will spend more time measuring
itself than serving the patient. You can see the punchline: “The
operation was a success. The hospital is in compliance. Our indicators
are good. And the patient died.”
The lure of committees is strong. Creating a NEW COMMITTEE looks like
you’re taking action, when in reality, committees are slow-moving
beasts, usually incremental in approach, and rarely have the power to
change any uncomfortable realities.
One last point - out of all these indicators, can you guess how many directly discuss laboratory
analytical quality?
Always the bridesmaid…
I’m
sitting in Copenhagen enjoying a beer right now. Those who attend the
Westgard Workshops know that beer is an important part of the workshop
experience, as well as the Wisconsin experience. I’m attending the
Scandinavian Clinical Chemistry meeting right now. It’s nice to be a
participant of a meeting after running a meeting of our own. There’s
more time to relax and think..
One of the best parts of the Westgard Workshops this year was
seeing the application and implementation of Six Sigma concepts in both
industry and laboratories. In many cases, these were participants who
attended the workshops several years ago, went back to their companies
and implemented the concepts. This year, they came back to the
workshops to present their results.
Dr. Gordon Kapke of Covance Central Laboratory Services
discussed his efforts to measure pre-analytical errors.
Covance runs a global operation with 8 laboratories on 6 continents.
They have over 1 million reportables every month - so defects per
million is not an abstract concept for them. Dr. Kapke showed how
design and automation of their processes have reduced their defects.
While Covance primarily serves the pharmaceutical development market,
their laboratory processes offer many lessons to the clinical
laboratory. Because of their size, Covance is not only introducing
approximately 1 new method a day, they are also being inspected on
average once a day (when it’s not the regulatory inspectors, it’s the
clients checking up on projects). This constant change and scrutiny
demands robust processes - Six Sigma techniques have proven their
worth.
Tony Orzechowski of Abbott Laboratories discussed how Six Sigma
design can be applied from the very beginning of instrument and assay
design. Abbott is now injecting quality requirements into their
earliest product development cycles. Not only do they assign error
budgets to the assays, they also determine how much error their
calibrators and control materials are allowed to have. The ideal goal
is to deliver a method or a control lot with a Sigma metric attached to
it. Tony uses OPSpecs charts to develop an operating “envelope” for
Abbott methods. They can use those charts and “envelopes” to match up
with customer experiences of performance, and possibly even predict
when and what kinds of technical issues may occur. Truly at the cutting
edge.
John Yundt-Pacheco of Bio-Rad also demonstrated how his company
has embraced Sigma metrics in their products. Bio-Rad will soon be
rolling out a service called Westgard Advisor. This program will
complement their online peer group offerings and allow customers to
easily (very easily) benchmark
their performance, have Sigma metrics calculated for their methods, and
receive instant QC recommendations. [Full disclosure: this was
developed as part of a collaboration between Westgard QC and Bio-Rad.
We have a vested interest in this product, if the name didn’t tip you
off.] John showed the results of Sigma medians for chemistry analytes
from this peer data. The results are quite encouraging (he will have a
poster at the AACC conference which will elaborate on this data) - many
labs are getting high Sigmas on many of the automated chemistry tests.
For many analytes, laboratories are in fact OVERcontrolling their
performance. With the help of data-interfaced software, the
opportunities for reduced QC effort (and savings) is made clear.
Finally, Dr. David Parry of St. Boniface Hospital in Winnipeg,
Manitoba, Canada, talked about his experiences implementing Sigma
metrics in his own hospital. The key lesson that he taught was that
there is a significant impact on the bench techs when you rationally
design your QC processes. The importance of reduced false alarms cannot
be minimized.
I’m not doing justice to their presentations here. And there was much more discussed at the workshops. But you get the idea.
From industry to laboratory, Sigma metrics are being implemented with
real results. These aren’t the giant “Master Black Belt” programs that
cost millions and focus on TAT and billing cycles - these are small
projects that involve performance in the lab being done by “belt-less”
individuals. You don’t need a massive program, a huge training period,
or huge resources (although they all help) - QC Design with Six Sigma
can be done even if all you have is the will to do something. The power
is yours.
The first day of the Westgard Workshops covered Standards and Practices, Indicators and Guidelines.
We were lucky enough to have presentations by both CAP and JCAHO
officials, detailing the latest changes in the checklists, guidelines,
patient safety goals, etc. I can’t do justice to all the details here.
There are simply too many changes year to year to cover in a short
note. What’s frightening is that ever since the “final” CLIA rule in
2003, every year is bringing a huge set of changes. You would think
that a laboratory would need to have a FTE devoted just to tracking the
changes in the regulations.
Obviously, unannounced inspections were a hot topic. But even more interesting was what wasn’t being said.
With the CLSI committees working on several new guidelines for both
manufacturers and laboratories (EP22 and EP23), with CLIA still formulating what
“Option 4″ is going to be, this year is mainly about anticipation.
Cross your fingers, because we are really hoping that the people who
gave us the disastrous EQC options 1 through 3 and going to hit a home
run this time.
An even more intriguing silence was the impending GAO report. As
you recall, in the wake of the Maryland General Hospital problems,
several congressmen asked the GAO to evaluate how CMS, CAP and JCAHO
were doing their jobs. This has been a multi-year evaluation.
Well evidently, the GAO has finished the report and it was supposed to
be released to the public in early June. Our JCAHO presenter, Kathy
Steffens mentioned that a draft report had been delivered in April. CAP
has already responded. JCAHO has also responded. But CMS asked for an
extended period of time to respond.
Overall, there is an expectation that this GAO report is going to be a
disappointment. It will ask for changes, but many of those changes will
not be useful. It may reach some strange conclusions. And the odds that
analytical quality is going to be improved by this report are very slim.
First of all, we want to thank all of our participants who came
from all directions to learn and share. The most rewarding part of this
continuing series is that some of our speakers this year were the
participants only a few years ago. Clearly, we have seen people absorb
the material, put it into action, and go even further with it.
Lookingat the “class of 2006″, it’s going to be very interesting to see
who emerges as the new leaders and teachers.
Here are some quick pictures of some of the class:
speakers. They, too, came from afar to present and enrich the Westgard Workshops. A special thank you goes to Callum Fraser,
who came from Scotland to discuss the role of Biologic Variation in
Quality Design and Planning. Probably the most illuminating discussion
on the topic this year. We also thank Bio-Rad, Inc. for supporting Dr.
Fraser’s travel and accomodation.
Dr. Fraser’s work on Biologic Variation contains some serious
statistics, but the overarching message is simple: the data on Biologic
Variation can quickly determine the needed performance of laboratory
methods. With the extensive databank compiled by Ricos et al., and
calculator tools easily available, all you have to do is plug a few
numbers in and click a button. It’s that easy to figure out the
clinical usefulness of current methods, as well as the performance
needed for the next generation of methods.
I’ll add more comments on the workshops throughout this week. Stay tuned.
I’ve been intrigued by some of the reasoning that is guiding laboratory priorities and directions, particularly the emphasis on pre-analytic and post-analytic processes along with a disregard for the quality of analytic processes. You can see this emphasis in the IQLM recommendations for laboratory quality indicators.
As we discussed earlier, there are 6 indicators for pre-analytic processes, 5 indicators for post-analytic processes, and only 1 indicator for analytic processes. That analytic indicator is for accuracy in Point-of-Care testing, thus there is no analytical indicator for a healthcare laboratory. Obviously, that means analytical quality is no longer an issue in laboratories today! CMS confirms this conclusion by recommending that laboratories today can reduce the amount of QC performed from the previous minimum of 2 levels of control per day to 2 levels per week (EQC option 2 or 3) or 2 levels per month (EQC option 1).
Why do people go to a gas station?
Today I noticed that my favorite gas station was out of gas. Not a problem if I go there only to get coffee and doughnuts, but a serious issue if I go there only when gas is available. If the station is out-of-gas, then I’ll go down the street to a coffee shop. The core competency I expect from a gas station is to be able to provide gasoline for my car. I don’t go there just to get coffee or use the restroom. Those services are nice, but they are valuable mainly as additions to the core capability to provide gasoline.
Why do people go to a hospital?
It’s obvious that the core competency of a hospital is billing, right? Billing is most important to support the operations of the hospital and allow customers to have access to the facilities and services. If not billing, then it must be parking! Parking makes it easy to access the hospital, so it is very important to the patients. [I’ve been wondering if it might be a good moneymaker to expand these services to provide gasoline and car washes.] BUT, don’t you think that the patients really expect the hospital to be competent to deliver the healthcare services that they need! That would be their number 1 concern and the core competency that is needed. Parking is a pre-analytic process that’s important, but not essential if the hospital can’t deliver the needed medical services – no reason for the customer to try to access those services. Billing is a post-analytic process and becomes irrelevant if the hospital can’t deliver services that are billable.
Why do people go to a laboratory?
Obviously it’s to have blood drawn or to deliver other specimens for analysis! Therefore the pre-analytic processes are very important if those specimens are to lead to reliable test results. BUT, the real competency that is expected from the laboratory is that it can produce reliable test results, otherwise, people wouldn’t bother to submit specimens and it wouldn’t be necessary to produce and deliver reports. Getting the right answer is still the fundamental and core competency of a laboratory! Calling critical values is important to facilitate the use of test results for treatment, but the assumption is that the test results being produced are correct! If a laboratory can’t get the correct result, then it doesn’t need to collect specimens and report results.
Why do we have this issue today?
People argue that getting the correct test result isn’t important if you have mis-identified the patient, collected the wrong specimen, or ordered the wrong test. True, but that doesn’t change the core competency expected of the laboratory. You shouldn’t be ordering tests and collecting specimens from patients if the laboratory can’t get the correct answer.
The real issue today is that many people assume the analytical quality of laboratory tests is better than needed for medical diagnosis and treatment, therefore it is appropriate to pay attention to other pre-analytic and post-analytic factors that affect the reliability of the testing process. BUT, there’s no data to prove that analytical quality is better than needed for medical applications! This is a myth – a mistaken yearning, theory, or hypothesis.
What is the quality needed for medical applications? If you can’t answer this question, what makes you think we are somehow achieving that unknown quality!
What QC is needed to guarantee achievement of the intended quality? Obviously, this is a difficult question in the absence of knowledge of the quality needed for medical applications. The right QC depends additionally on the precision and accuracy observed for the analytical method in the particular laboratory. The right QC will vary from test to test because there are different quality requirements for each test and different performance characteristics for each method.
Here’s some data on analytical quality today!
In an attempt to debunk this myth that analytical quality is better than needed for medical care today, we recently published a paper titled “The Quality of Laboratory Testing Today: An assessment of sigma-metrics for analytic quality using performance data from proficiency testing surveys and the CLIA criteria for acceptable performance” [Westgard JO, Westgard SA. Am J Clin Pathol 2006;125:343-354]. Here’s the abstract:
“To assess the analytic quality of laboratory testing in the United States, we obtained proficiency testing survey results from several national programs that comply with the Clinical Laboratory Improvement Amendments (CLIA regulations). We studied regulated test (cholesterol, glucose, calcium, fibrinogen, and prothrombin time) and nonregulated tests (internal normalized ratio [INR], glycohemoglobin, and prostate-specific antigen [PSA]). Quality was assess on the sigma-scale with a benchmark for minimum performance of 3-sigma and a goal for world-class quality of 6-sigma. Based on the CLIA criteria for acceptable performance in proficiency testing (allowable total errors [TEa]), the national quality of cholesterol testing (TEa=10%) is estimated as 2.9 to 3.0 sigma; glucose (TEa=10%) 2.9 to 3.3 sigma; calcium (TEa=1.0 mg/dL) 2.8 to 3.0 sigma; prothrombin time (TEa=15%) 2.8 to 3.0 sigma; INR (TEa=20%) 2.4 to 3.5 sigma; fibrinogen (TEa=20%) 1.8 to 3.2 sigma; glycohemoglobin (TEa=10%) 1.9 to 2.6 sigma; PSA (TEa=10%) 1.2 to 1.8 sigma.”
Here’s the conclusion!
The analytic quality of laboratory tests requires improvement in measurement performance and more intensive quality control monitoring than the CLIA minimum of 2 levels per day.
To:
Those who will influence the future quality of laboratory testing
CLSI Area Committee on Evaluation Protocols
CLSI Subcommittee on Establishment of Manufacturer’s QC Recommendations
CLSI Subcommitee on Laboratory Quality Control Procedures
I want to call your attention to an article on “The Quality of Laboratory Testing Today” that was published in the American Journal of Clinical Pathology in March 2006 (v125:pp343-354). This article was prepared for the “QC for the Future” workshop, which was held in March 2005 in Baltimore. Because you attended this conference and/or are involved in the ongoing project work to develop QC for the Future, I hope you will have some interest in the results of this study.
I take the unusual step of calling your attention to this study to ensure that the CLSI committees charged with the responsibility for future QC practices are challenged to consider the current analytical quality of laboratory tests in quantitative and measurable terms. Emerging laboratory quality indicators, such as being proposed by IQLM, focus mainly on pre-analytic and post-analytic variables and pay little attention to analytical quality. The results of this study indicate that the analytical quality of laboratory tests is still a serious problem that requires ongoing improvements in analytical methodology and laboratory QC.
In considering QC for the Future, CLSI seems to be driven by the ISO guideline for manufacturers (ISO 15198: Validation of manufacturer’s recommended procedures for quality control) which recommends a risk analysis approach to identify sources of variability that ideally should be eliminated by careful design of the analytic system. In Section 4 on “Quality control recommendations,” the following guidance is given to manufacturers:
-
Section 4.1: “Residual risks should be minimized by the manufacturers’ recommended quality control procedures. The quality control procedures shall include a method of detection (e.g., quality control material, electronic monitoring system, or on-board chemical control) and acceptability criteria that will determine when a critical failure occurs or a means to determine the acceptability criteria. Limitations of the quality control procedure shall be identified and described in the instructions for use.”
While that guidance is fine and good for manufacturers, I want to remind you of ISO 15189 document (Medical laboratories – Particular requirements for quality and competence), which provides guidance to laboratories. In discussing examination procedures (i.e., measurement procedures), laboratories are given the following specific guidance:
-
Section 5.5.4: “Performance specifications for each procedure used in an examination shall relate to the intended use of that procedure.”
-
Section 5.6.1: “The laboratory shall design internal quality control systems that verify the attainment of the intended quality of results.”
Note first the need to define the “intended use” or “intended quality” in order to implement examination procedures with appropriate precisian and accuracy, as well as the appropriate QC design.
Note first the need to define the “intended use” or “intended quality” in order to implement examination procedures with appropriate precision and accuracy, as well as the appropriate QC design. Second, the guidance for internal QC design doesn’t say mitigate or reduce or minimize the risk of not attaining the intended quality of results - it says to verify the attainment of the intended quality of results. The laboratory is responsible for verifying that the intended quality is achieved, not just minimizing the risk of poor quality.
This objective of verifying the attainment of intended quality of results can be achieved by designing a statistical QC procedure to account for the quality required for the test and the precision and accuracy observed for the measurement procedure, as described in CLSI document C24-A3 (Statistical Quality Control for Quantitative Measurement Procedures: Principles and Definitions). Note that the ISO guidance to manufacturers identifies statistical QC as an example “method of detection,” thus statistical QC is an appropriate way to monitor residual risks, as well as to verify the attainment of the intended quality of test results.
As you develop recommendations for QC for the Future, please keep in mind that statistical QC is still a valuable tool to verify the attainment of the intended quality of test results. There seems to be a predisposition today to try to avoid the application of statistical QC in laboratories, or reduce the amount of QC performed, as evidenced by the original CMS proposal for “equivalent QC procedures.” It is important to set the objective as performing the right amount of QC to detect medically important errors, which depend on the quality required for the test and the precision and accuracy observed for the measurement procedure. In any approach you recommend, make sure you can account for intended quality and verify its attainment in routine laboratory operations. Otherwise, we run the risk that the quality of laboratory testing in the future will be worse than it is today.
Reference: Westgard JO, Westgard SA. The Quality of Laboratory Testing Today: An assessment of metrics for analytic quality using performance data from proficiency testing surveys and the CLIA criteria for acceptable performance. Am J Clin Pathol 2006;125:343-354.
We invite your comments on this open letter. (Please note that we will moderate all submissions to ensure that commentary is substantive).
Welcome to the Westgard “blog”! For many of you, this may be the first blog you’ve visited. Don’t worry, a blog is nothing unusal. Simply put, this is just a different format for what we’ve been doing at Westgard Web for nearly ten years. The most interesting wrinkle about a blog is that it can include comments from the readers. That is, not only do you get to read what the author has to say, but you also get to see what the viewers think as well.
We’re going to experiment with the blog format. Comments will be moderated so that we can weed out any statements that are not germane to the issue of the day (believe it or not, there is such a thing as spam comments). For those who are more comfortable with the traditional Westgard Web presentation of articles, you can go back to the
Whether we go “all-blog”, “no-blog” or somewhere in between, will depend on your response.
If you love blogs and want Westgard Web to move to this format, let us know. If you prefer us to stay out of the blogosphere, tell us that, too.
Oh, and thanks for visiting
[Sten Westgard]