« August 2008 | Main | October 2008 »

September 2008

September 25, 2008

Has Quality gone Sub-prime?

Posted by Sten Westgard, MS

As our Congress ponders a $700 billion dollar bail-out of Wall Street, and the rest of us on Main Street ponder our financial futures, it seems like as good a time as any to draw some parallels between the larger US economy and the laboratory economy.

For those of you who aren't watching the news, this Wall Street crisis got its start in Sub-Prime mortgages. (A nifty picture here illustrates how and why this happened and a handy little timeline on the most recent events in the crisis can be found here) Subprime mortgages used to be something that only people with poor credit history could get, perhaps only 15% of the mortgages. They used to be considered a risky proposition that most investors avoided. Yet they become very popular with both borrowers and investors alike, until nearly 25% of all mortgages were subprime.

What happened?

The Start: The Government is here to help

The seeds of this financial disaster were planted, ironically, in the ashes of the last crisis, the Internet bubble. At that time, the Federal Reserve lowered interest rates to just 1% in order to spur the economy. These low rates made it easier to borrow money, making home ownership possible for previously marginal home owners.

In the aftermath of laboratory scandals, the government passed the CLIA legistation. However, since they were unable to work out the procedure for manufacturer's QC clearance, they delayed that provision. In the interim, they gave some devices and QC methods temporary waivers.

The New Rules

Eager to attract grow their margins, the mortgage industry created a new raft of "financial instruments" to extend loans to riskier and riskier customers. These loans included the following types:

  • Teaser Rates, aka Adjustable Rate Loans. This offered a low interest rate at the beginning of the loan, followed later by a dramatic rise in the rate. Save and buy now, Pay a lot later.
  • No Doc or "Liar Loan." This loan allowed the applicant to "state" their income but not provide any proof of this income. It was perhaps first intended for people who work in jobs where the wages are primarily earned in cash, perhaps day laborers and waitresses.
  • NINA Loan: No Income. No Asset Verification. For this loan, the applicant did not have to state an income, nor did they have to verify their assets. In theory, these were loans for people who wanted their privacy and were willing to pay through the nose to keep it.
  • NINJA Lona: No Income. No Job. No Assets. This was a loan given to people who could fog a mirror. This isn't just tortured logic. This is logic that has been murdered.

At the height of the bubble, when housing prices were rocketing up, these loans seemed to make sense. Why not take out a subprime mortgage, if only a few years from now, the value of the house is going to double. If housing prices rise forever, you can sell the house to the next person for far more than your mortgage, no matter what the terms.

What excerbated this problem even more was the unscrupulous nature of some of the loan officers. Since they were paid commissions up front for getting new subprime loans, some officers would falsify mortgage applications for their unknowing clients.

After repeated delays, CMS and the FDA cannot find an agreeable way to implement manufacturer QC clearance. So it's back to the drawing boards. CMS writes a new clause in the law - "Equivalent Quality Testing" - and invents a new set of "Equivalent QC Procedures" which are added to the Interpretative Guidelines of the State Operators Manual:

  • Option 1: Run controls for 10 days, then run QC only once a month
  • Option 2: Run controls for 30 days, then run QC only once a week
  • Option 3: Run controls for 60 days, then run QC only once a week
  • "Option 4" or "Alternative QC" has been the subject of numerous CLSI committee, with years of work put into developing so-called Risk Management approaches

The absurdity of these options have been exhaustively documented and discussed on Westgard Web. Yet despite the near-universal condemnation of the the options, despite the admission from CMS that "We blew it", it appears that these options are here to stay.

The Collaborators: Pervasive Leadership Failure

It wasn't inevitable that subprime mortgages got out of hand. It wasn't inevitable that Wall Street would become so addicted to these dubious financial instruments. If the heads of these institutions had demonstrated leadership (and ethics), instead of relentless greed, we wouldn't be in this mess.

Part of the problem was the dispersion of risk and responsibility for these subprime loans. Since no single institution owned an entire subprime loan, it didn't feel the need nor the obligation to ensure that it was a good loan. Back in the old world of mortgages, if a single bank made too many subprime loans, it would become obvious that there was too much risk and the market would force the bank to stop its practices. However, banks and mortgage lenders didn't hold onto their own loans anymore. Instead, they packaged them up into "collateralized debt obligations (CDOs)," a type of "mortgage-backed security," which they then sold off to the broader market. Thus, the banks and mortgage brokers got paid up front, outsourced the responsibility to someone else, and had little incentive to be responsible or think for the long term.

Furthermore, competitive pressures meant that choosing to be ethical was a money-losing proposition. "Everyone was doing it." If one bank refused to get into the subprime business, they suffered and another bank got that business.

Worse still, the rating agencies, which could have put a stop to the mortgage-backed securities by giving them bad (risky) ratings, instead felt pressure (and reaped rewards) to give high ratings. Thus, a security made up of many risky subprime mortgages was given a AAA rating by agencies like Moody's and Standard & Poor's. Since mortgage securities were outside federal regulations, the government did not act to rein in the risky practices, either.

Individual criticism of Equivocal QC wasn't that hard to find. But it was strange to see that most manufacturers, agencies, and professional organizations were silent on the issue, particularly at first. Once the popular verdict on EQC started to come in, more institutions were willing to join them. But quality seems to be a subject like politics in the US; everyone has an opinion on it, but usually they don't want to talk about it in public.

Manufacturers face the market pressure to match whatever "cost saving" features their competitors offer. So if EQC means reduced QC costs, it's difficult for a single company to hold fast and insist that their customers need to spend more on QC. Since the professional organizations draw their support from manufacturer sponsorships, advertisements and other support, they have little incentive to discuss a topic that might offend them.

The accrediting agencies, like JC, CAP, and COLA, were probably in the best position to halt the downward slide of quality. But they faced competitive pressures, too. If CAP was too strict on POC devices and EQC, labs could and would switch to the laxer standards of JC. Finally, when CMS decided to push EQC through, it simply flexed its deeming authority powers and forced the agencies to adopt some form of EQC or lose its accreditation status.

Responsibility for quality has also been spread out until everyone and no one is responsible. Customers talk about quality but make decisions based on cost. Manufacturers struggle to maintain a balance of the two, but cost is the overwhelming driver of their business. The EQC protocols insist that the laboratory director must weigh the "clinical and legal responsibility" against the EQC options, a tacit admission that EQC probably is neither clinically nor legally appropriate. Everyone wishes quality was better, but acts in ways that reduces it.

The Bubble Pops

It's hard to summarize the factors that brought the end to speculation. But as interest rates rose, it became harder to take out loans. Housing prices began to drop, which meant you could no longer sell your house to pay off the mortgage. As housing values began to drop further, the value of some houses dropped below the mortgages (known as negative equity). People began to miss, even default entirely, on their mortgage payments, as teaser rates reset, as impossibly large payments came due on people who had no means to pay. As defaults rose, more houses came back onto the market, depressing the pricing. It turned into a negative feedback loop - more colorfully put, a death spiral.

Wall Street made it worse. Banks and hedge funds were highly leveraged in mortgage-backed securities (i.e. they had made all these investments using lots of borrowed money). Now their loans were coming due and their investments increasingly looked like they weren't worth the paper they were printed on. As banks and firms "wrote down" the value of these investments (admitted the money they had invested was gone), they began to run out of money. Some went bankrupt, as we've seen. Others have been taken over by the government. As fear gripped the financial community, there wasn't enough money left for the usual, normal and necessary loans - even those that aren't as risky as subprime mortgages. And here we are, looking at hundreds of billions, probably trillions of money, that will be needed to fix the mistakes.

Will the quality bubble ever pop for the laboratory? If we continue to debase the rules and loosen the regulations, and if our institutions acquiesce in this drift, it's probable that we will fall outside the margins of safety. If we reach the point where testing QC once a month becomes an accepted practice, it's probable we will see bad test results impacting large numbers of patients.

The Maryland General scandal, which was the case of an individual laboratory with debased quality practices, is merely an hors d'oeuvre compared to the systemic problems we could face.

I believe healthcare is more resilient than Wall Street. We believe more in mission than mere profit. Our professionals have more scruples and commitment to good practices. But all of that means that the errors can grow larger, the bubble can get bigger, before it overwhelms the system.

A quality crisis is not inevitable. All we need is leadership. And even then, leadership isn't necessary if enough voices are raised. Your voice. The voices of laboratory professionals. If the chorus is loud enough, the leaders and the market and maybe even the rules will follow.

Offered in the spirit of a previous essay, Enron and Quality

September 24, 2008

Westgard Interview on the Web, Part Two

Posted by Sten Westgard

COLAsymposium Back at the 2008 AACC conference in Washington, DC, Dr. Westgard sat down for an hour with some friends from COLA. They interviewed him on a number of topics, from his personal story and the journey of his career in laboratory medicine, to the origin and use of the "Westgard Rules."

You can see the second part of this video interview on the Advance website, sponsored by COLA.

[ Here's the link to first part of the interview.]

The release of this interview coincides with the run-up to COLA's Symposium for Clinical Laboratories. Dr. Westgard will give the keynote address for this symposium and he will join experts like Lucia Berte, Greg Cooper, Judy Yost, and many others, to discuss quality and laboratory medicine. This symposium is taking place on September 25th through the 27th, in Scottsdale, Arizona.

These excerpts won't be up for much longer. So if you want to actually see and hear Westgard on the Web, do it ASAP.

September 18, 2008

What's new on Westgard Web: September 2008

What's New this month at Westgard Web:

  • Fall Workshops last chance for early registration
  • High Reliability Testing: First Principle
  • Guest Essay: Time to engage in Measurement Uncertainty
  • Westgard Sigma Analysis: An automated POC hematology instrument



FALL WORKSHOPS: Last chance for Early Registration

WORKSHOP: Method Validation and Verification
1-day intensive training based on the new edition of Basic Method Validation
Two locations to choose from

October 10th, Abbott Park, Chicago, Illinois
  [supported by an educational grant from Abbott Diagnostics]

October 17th, Olmsted Clinic, Rochester, Minnesota
  [supported by an educational grant from Olmsted Clinic]

Find out more and get a coupon for early registration discounts at
 http://www.westgard.com/workshops.html#bmv



WORKSHOP: Six Sigma Quality Design and Control

Dr. Westgard's most popular and most requested workshop is back.
1-day introduction to Six Sigma, with new applications
and innovative tools.

November 12th, Spectra Laboratories, Rockleigh, New Jersey
  [Supported by an educational grant from Spectra Laboratories]

Find out more and save $100 off the regular registration rate at
 http://www.westgard.com/workshops.html#sigma



HIGH RELIABILITY TESTING: First Principle - Preoccupation with Failure

One of the key principles in High Reliability Organizations is a
Preoccupation with Failure. No, this doesn't mean a lab that needs Prozac.
While Preoccupation with Failure may sound bad, in HROs, this focus on error
is a key factor in safety and success.

http://www.westgard.com/lesson85.htm



GUEST ESSAY: Time to Engage in Measurement Uncertainty

In the ongoing "War of Words" in the Lab, it's time to hear another voice.
Dr. Dietmar Stockl, an expert from across the Atlantic,
provides us with a detailed essay explaining how measurement uncertainty
 can be useful to the laboratory - and even co-exist with Total Error.

http://www.westgard.com/guest41.htm



WESTGARD SIGMA ANALYSIS: Automated POC hematology instrument

A recent paper in a prominent journal evaluated a new POC hematology instrument.
They concluded that the performance was "within acceptable limits."
When the bar is that low, what's the actual Sigma performance?

http://www.westgard.com/qcapp51.htm

September 13, 2008

Westgard interview on the web

Posted by Sten Westgard

COLAsymposium Back at the 2008 AACC conference in Washington, DC, Dr. Westgard sat down for an hour with some friends from COLA. They interviewed him on a number of topics, from his personal story and the journey of his career in laboratory medicine, to the origin and use of the "Westgard Rules."

You can see the first part of this video interview on the Advance website, sponsored by COLA.

The release of this interview coincides with the run-up to COLA's Symposium for Clinical Laboratories. Dr. Westgard will give the keynote address for this symposium and he will join experts like Lucia Berte, Greg Cooper, Judy Yost, and many others, to discuss quality and laboratory medicine. This symposium is taking place on September 25th through the 27th, in Scottsdale, Arizona.

I believe more excerpts of this interview will be posted soon, but they may not be up for long. So if you want to actually see Westgard on the Web, do it ASAP.

September 08, 2008

Quality Control vs. Quality Compliance: What's SAFE?

James O. Westgard, PhD

Here are some notes, comments, and links that go along with my presentation at the AACC teleconference on “New Directions in Laboratory QC: EQC, Alternate QC and Risk Assessment,” which was on September 4, 2008.

There is a strong push by CMS and CLSI to introduce new QC guidelines that are based on “risk analysis.” This direction is consistent with ISO’s emphasis on risk analysis for manufacturers. Soon we will see how it influences for laboratory QC. I realize that US laboratories may not yet have much interest in ISO, but we need to recognize that ISO will play an increasing role for guidance in quality management in the future.

QC Options. CLIA rules allow laboratories 4 different options for QC. Here I call these options statistical QC (SQC), deFault QC (FQC), Equivalent QC (EQC), and Alternate QC (AQC). They are discussed in the context of the CLIA regulation as well as ISO 15189 guidance for medical laboratories. The reason for including ISO is that the emphasis on risk analysis originates with ISO, thus we should also understand the context in which ISO describes guidance for QC. Statistical QC. The first option is to do statistical QC (SQC) to monitor the precision and accuracy of the process, detect immediate errors, and monitor test performance over time. This can be done following guidelines from ISO 15189 and CLSI C24-A3, as outlined in the slides. A Sigma-metrics tool will soon be available for download from this website. It will include directions, a Sigma-metrics graph for 2 levels of QC, and a sigma-metrics graph for 3 levels of QC. Note that the C24-A3 document does not include a tool for 3 levels of material. There are many other materials on Sigma-metrics and QC planning on this website. Start with the following if you want more information and background:

  • Some background on “power curves” – those detector response curves that show the probability of rejecting runs have different sizes of errors – can be found in the lesson “Power Function Graphs” at www.westgard.com/lesson4.htm.
  • The application of “Sigma metrics” for selecting the “right QC” procedures is described in the lesson “CLIA Final Rule: Appropriate QC Procedures” at www.westgard.com/cliafinalrule9.htm.
  • There also is a PDF file available for a paper “Internal Quality Control: Planning and Implementation Strategies. www.westgard.com/IQCpaper2003.html

Default QC. There is little to say! This is what most laboratories do today – two levels every 24 hours. However, you should understand that this was never the intention of CLIA that this would become the standard QC practice. It was supposed to be temporary guidance until such time as the FDA provided a QC clearance process that would provide for review and approval of a manufacturer’s QC directions. After five postponements and some 10 plus years of default QC, it was concluded there no longer was a need for an FDA QC clearance process. For some historical background on CLIA and its original recommendation for QC clearance, see the following:

  • For an explanation of the QC clearance provision, see “QC Clearance Postponed Again & Again & Again & Again.” www.westgard.com/essay37.htm
  • For a brief summary of the changes in the final, final, final, final, final version of CLIA, see “The Final CLIA Rule.” www.westgard.com/essay50.htm

Equivalent QC. There is a lot of discussion on this website. Start with the following:

  • See “Equivocal QC: Coming soon to laboratory near you!” www.westgard.com/essay119.htm. At the end of this essay you will find links to the most important earlier discussions. There’s probably more than you want to know, but you may get interested in seeing how this has evolved over the last few years.

Alternate QC. There’s not much specific information available unless you can find drafts of the EP22 and EP23 documents. Those will be hard to come by because they are pretty well protected until the documents are ready to be released. That means the first public drafts will likely become available in 2009 and the final approved drafts probably in 2010. Meanwhile, you can set up a process by which you can take advantage of this information when it becomes available. That’s what I’ve briefly described as a “Total QC Strategy” near the end of this presentation – see slides 34-36 and the following discussions on this website:

Risk Analysis Too. Be sure to read Dr. Kent Dooley’s essay on “Frequency of QC: A Patient Safety Perspective. www.westgard.com/guest35.htm

What’s SAFE?

A few comments about my thinking here.

  • SQC is first on my list because we currently have good design tools for selecting the right SQC rules and numbers of control measurements. The main limitation is that we don’t have an objective methodology for determining how often controls should be run. Here’s where Dr. Parvin’s ideas about “event” and “non-event” QC provide the best available guidance.
  • AQC is lower on the list because we don’t yet have the guidance on how to do this. However, if you consider that SQC will most likely be necessary under most conditions to monitor “residual risks,” then the ability to design SQC sets the stage for what should follow with AQC. It’s possible that AQC will move into the #1 spot if the guidelines turns out to be practical and easy to implement, which will wreck my SAFE acronym. I’m not sure how long that will take, but I’m betting that I’m SAFE for another 2 years and by that time I’ll probably really be retired. Also, those tools in the QC tool box may not be so easy to implement, in which case AQC may never become practical. Patient data QC sounds good and can be very useful, but it’s not trivial to optimize the design of patient data algorithms to assure the attainment of the intended quality of results.
  • FQC gets an F on my grading sheet. The only plus is that you’re still running controls, but you really should implement the right QC design, which would then move this into SQC above.
  • EQC is even worse because you’re not running controls as often. Remember that the laboratory director must take full responsibility for any decision to implement EQC. CLIA allows you to do this, but if you do it, you are responsible and you can’t blame CLIA if you run into problems. By that I mean that CLIA will not be a “get out of jail” card here. You do this at your own risk.

Q&A. Some additional information that may provide more complete answers to some of the questions that were asked:

  • Concerning how to review, assess, and evaluate your QC procedures and process, see “Good Laboratory Practices for Statistical QC. Part I. Designing the Right QC.” www.westgard.com/essay102.htm. It is also very important to implement SQC properly, which is discussed in “Part II. QC Limits and Limitations.” www.westgard.com/essay103.htm
  • Concerning slide 14 and the figures in column Ped in the key. Those figures should be 0.00. I generate the underlying power curves here using a computer program that typically draws a single line for a particular error condition or sigma-value. Here I need a figure without that line, so I have to fake out the program to get the underlying graphic. It looks like I must have set the error condition to be off-scale on the left, therefore ended up with the same figures as for false rejection (Pfr). Someone has pretty sharp eyes because no one has ever caught me at this before! Now I will have to figure out how to get zeros.
  • Concerning the question about instruments that have internal statistical control: Yes, there is at least one BGAS system that has a complete internal SQC procedure, as well as many additional built in instrument checks. And that system still must be qualified for EQC according to CMS, even though the internal SQC is much better than any external SQC in this case because of the sample handling difficulties with BGAS controls.
  • Concerning the use of patient data algorithms, such as “Bull’s algorithm” or the more generic “average of normals” or “average of patients,” see “Six Sigma: Patient Data for Assessing Process Performance and Stability” at www.westgard.com/qcapp17.htm. For some explanation of “multi-stage QC,” i.e., the idea of using multiple QC procedures in a complementary way, see the discussion “Sage Advice about New Approaches to Quality Control” at www.westgard.com/essay30.htm

Compliance vs Excellence.

CAP has introduced a new accreditation program CAP:15189 that is based on the ISO document. CAP says that this doesn’t replace the LAP accreditation which is based on CLIA. And they explain that “although it is not currently a standard in the US, CAP believes that a laboratory that is working to achieve best practices in quality management systems will seek out accreditation to ISO 15189:2007 and its benefits.” It looks to me like we’re heading for a two-tiered system where CLIA will be used for compliance with minimum standards and ISO 15189 will be used for excellence. That’s why we need to start paying attention to ISO 15189 if we want our laboratories to remain competitive in the global marketplace. For additional discussion about CAP and its intentions with 15189, see Sten’s interview with CAP officials at www.westgard.com/interview8.htm.

September 04, 2008

Another voice on Measurement Uncertainty

By Sten Westgard, MS

While Dr. Westgard has spoken about the "War of Words" between Uncertainty and Total Error in several essays (here, here and here most recently), we thought readers might want an additional perspective on the issue.

Dietmar Stockl, an expert in statistics and laboratory quality control from across the Atlantic, graces us with an guest essay on the calculation, use, benefits and limitations of measurement uncertainty: Time to Engage in Measurement Uncertainty. Dr. Stockl provides an in-depth look at uncertainty concepts and calculations, as well as a moderate viewpoint on the use of the term. He views both Total Error and Measurement Uncertainty are useful concepts and believes there is room in the world for both of them. It's not necessary for one term to eliminate the other.

Not so coincidentally, a colleague of Dr. Stockl's, Linda Thienpont, has a letter in Clinical Chemistry talking about the attempt to incorporate bias into the calculation of measurement uncertainty. If bias can be thrown into the uncertainty calculations, a case could be made that Total Error is no longer necessary. Dr. Thienpont points out that slipping bias into the uncertainty calculations is not a good idea and can lead to distorted results. She concludes that bias must remain separate from uncertainty calculations, which means another approach like Total Error is required to account for it.

See more at Thienpont LM. Calculation of measurement uncertainty-Why bias should be treated separately. Clin Chem 2008;54:1587 (subscription required)

My Photo
Blog powered by TypePad

Pages

Paying the Bills