Posted by Sten Westgard, MS
I recently got a smart question from a concerned laboratory scientist. After reviewing one of the Sigma-metric studies on the website, he noted that while a particular method had a bad Sigma-metric, the main reason was due to the bias. His question was essentially (and I am paraphrasing here), "If the bias component comes from a particular difference between the instrument or kit and a reference system, shouldn't it be excluded from the Sigma-metric calculation?"
The reasoning is that the bias problem could be (1) eliminated through recalibration, (2) it may be a bias against a method that is not a reference method, so the difference might not be "real", or (3) if the reference range is adjusted and the method is used in exclusion, bias doesn't matter anyway.
We've had a lot of discussion about bias in our statistics lately. Is this a case where the Sigma-metric is "skewed"? What's your verdict? A discussion after the jump.
Let's take this argument a little more broadly, and apply the discussion to not only bias but to all the variables in the Sigma-metric equation.
Here are some valid reasons by each variable in the Sigma-metric equation may not be useful.
1. Quality goal. There is a wide range of quality requirements available, from CLIA to Rilibak to RCPA to the biologic variation database. Some quality requirements are larger than others. It's possible to choose one that is too large or too small and it may not be appropriate for the actual clinical use of the test results.
2. Imprecision. There are many ways to calculate imprecision: within-run, within-day, between-day, cumulative, etc. The shorter the time period, the more optimistic the estimate of imprecision might be, and thus the higher the Sigma-metric. There may even be an argument that the imprecision of the control materials does not properly reflect the performance on true patient samples (a commutability issue).
3. Bias. As we mentioned earlier, the ideal bias is calculated when we compare the test method against a reference method or material. Often, this is impractical. So laboratories make comparisons against "local methods" or other methods that aren't as traceable.
In other words, the principle of "Garbage In, Garbage Out" applies to Sigma-metrics, just as it applies to any other statistical calculation. When we depart from the ideal in our obtaining of data, we will get less than ideal statistics out of the equation.
Here are a few articles where we cover the different issues in greater detail:
In the specific question above, we certainly believe that laboratories are empowered to discount a bias calculation if they believe it is not relevant in their situation. Indeed, for laboratories that are starting out on their Sigma-metric equations, our common recommendation is that if they can't determine what bias is, start by assuming that bias is zero. In those cases, it's an estimate of the maximum Sigma potential of the method. And if later on, the laboratory can estimate bias, they should bring that back into the equation. Whenever more and better data is available, it's always good practice to use it.
One other note on the calculation of Sigma-metrics. Perhaps one of our problems with the metric is happening because we have so few studies of Sigma performance, and that we see them so infrequently, so that any particular study stands out as a single verdict. We need more data, more data, and more data still. If we had multiple studies of method performance, we would have a better picture of performance. So if one study showed a particular bias, the weight of the other studies might show that that was only a fluke, an outlier. A problem with bias tells us we should do more studies, not fewer, so we can either confirm that this problem truly exists, or discount it because it was an outlier.
Here's one last argument for including bias: in an age of converging health information, where eletronic medical records are expected to allow clinicians to compare different test results for a single patient, bias is increasingly relevant. While we might have been able to dig a moat around our own individual laboratory in the past, and keep all other test results from view, that is not feasible for the future. We will have to take into account the bias between methods, and inform our clinicians about it, or patients will start to get treated on the basis of inter-method differences that are not indicative of any clinical change.
Ultimately, it's better to confront bias and determine how real and relevant it is, rather than to ignore it and pretend it doesn't impact our tests.