Posted by Sten Westgard, MS
So we all know that Risk Analysis is coming to laboratories in the US. (click here if this is news to you). But Risk Analysis, particularly the FMEA technique (Failure Mode and Effects Analysis), is not new to healthcare. Outside the laboratory, plenty of healthcare practitioners have been performing FMEA.
So what do they think about this technique? Try and guess which one of these responses is from a someone in healthcare:
- "The jury's still out on the FMEA process because... has anybody evaluated FMEA as a tool for analysing risk? And it turns out there isn't... well why are we doing this process?... When all it is doing is bringing a few things to the surface, which is no bad thing, but it's not a validated process."
- "...Forget FMEA. It doesn't really work effectively, I don't think, and the scores are a hindrance rather than anything else, year... We wasted a lot of time on FMEA before we realized, this isn't actually working. Yeah, because I think you can get caught up on just the score, that's the thing."
- "The scoring in the FMEA teams need to be the same people, if you change half way through because of the highly subjective interpretation things change dramatically."
Failure Mode and Effects Analysis: view of hospital staff in the UK, Nada Shebl, Bryony Franklin, Nick Barber, Susan Burnett, Anam Parand, J Health Serv Res Policy, 2012 Jan;17(1):37-43.
In the UK, hospitals have been ahead of the US in their use of Risk Management techniques:
"Between 2004 and 2008, FMEA was introduced in 24 British hospitals as a method to improve patient safety as part of the Safer Patients Initiative (SPI)."
After several years of field experience with FMEA, the study authors conducted interviews to find out what healthcare professionals thought about the technique. As it turns out, very few studies of this kind have ever been conducted, particularly studies that also assessed the validity and reliability of FMEA results.
The main findings of the study focus on the FMEA Risk Priority Number (RPN). RPN's, you will remember,as the product of (severity * occurrence * detectability). In the case of this study, participants had used FMEA with this three-factor model, each of which was ranked on a scale of 1 to 10. Thus, the RPN's would be on a scale of 1 to 1000.
There were two problems that the study found:
- The FMEA teams focus too much on the RPN number.
- The reliablity and validity of that RPN number is suspect.
With the first point, it was clear that many teams "described the success or failure of their FMEA according to their 'reduction score' rather than the actual actions and outcomes implemented." That is, if they had a failure mode with an RPN of 700, if they brought it down to 350, they considered that to be a measure of success, regardless of what actually had changed in the process. The authors note that
"the RPN should not serve as an indicator for the success or failure of an FMEA as the teams may become more focused and pressured to reducing a numerical figure to prove their successful risk assessment rather than patient outcomes. In addition, as the RPN is a product of three different scoring scales, any factors besides the teams' recommendations, such as team composition or method of recalculating the scores, may influence the RPN values."
The second problem with FMEA is more well-known: the numbers are not always objective, but may instead be subjective judgments (or guesses) from the team. Quoting the authors again,
"There are several reasons to question the reliability and validity of FMEA. It requires people to estimate events and their consequences and to covert these to an ordinal scale, both of which are likely to be inherently unreliable processes. Additionally, calculating the RPN requires multiplication of ordinal scales, which breaches their mathematical properties."
That isn't to say that all interviewees felt that FMEA was useless. Indeed, many participants liked the fact that it brought people together to discuss and tackle risks in their processes. Overall, the study concludes that the FMEA technique has its benefits, strengths and weaknesses:
"Participants opinions of FMEA demonstrated their diverse understanding of the process. Overall, the majority concluded that it was a subjective but systematic tool that helps identify high risk areas by getting people together to discuss the problems. The main limitations were seen as its time consuming nature, difficulty with the scores and its lack of validity and reliability."
For US laboratories that will soon embrace FMEA as their preferred technique for designing QC Plans, this study gives a "heads-up" notice on problems they might encounter. Seeking out a more robust Risk Analysis FMEA technique may help avoid some of these FMEA flaws.
Comments