I once flagged a bug in Epic, the big EHR system.
The system had somehow mixed up kilograms and pounds.
For example, a normal adult male weight of 150lbs would be ~68kg,
But accidentally save it without converting and get 150kg.
Convert back and it becomes 330lbs.
Suddenly our reasonably slim man becomes grossly obese.
It's not just wrong,
it's extremely dangerous.
In an emergency situations,
where morphine is commonly administered for extreme pain,
the dosage needed to relieve the pain of a 330lb man would kill a 150lb man.
Granted the responder at the patient's side would probably realize something is amiss,
but a pharmacist in another room filling an order wouldn't have the context,
and could make the error.
I wouldn't trust that a nurse or doctor that is bedside to flag that either, though. Hospitals are woefully understaffed, and while they will do there best, we are all just humans.
My wife's grandmother was killed by a second dose of metformin (well kidney failure after a second dose) because the attending that administered the first dose left the room, planning on coming back a moment later, when the next round nurse came in, they noticed the does hadn't been administered (wasn't in the chart), ordered another dose, and injected it.
There were multiple layers that should have prevented that. The prescription shouldn't have been filled for a second time without someone noticing. The first doctor should have filled in the chart before leaving. And the pharmacist should have noticed that it had already been requested.
Too many patients, too few doctors, and with Epic, too many button clicks.
I am very sad to hear that your grandmother died as a result of a medical error, but the details of the story as you remember them aren’t quite plausible.
1) Metformin is not available in an IV formulation
2) Metformin itself is not nephrotoxic
It certainly is believable that a medical error caused kidney failure but it is very unlikely to have been caused by an incorrect second IV dose of Metformin.
I expect GP mistook metformin-associated lactic acidosis in a patient with kidney disease as something metformin-caused. A separate but coincident IV misadministration could be an exacerbating factor. It would have been a rare case, but a plausible explanation for the misunderstanding.
Metformin is relatively kidney safe and not administered by IV. Thank you for clarifying that for anyone that may currently on or considering Metformin. It would be great if medical professionals were infallible communicators and had time to verify understanding, but they are human and we need more doctors and less time-pressure by profit extracting private equity.
Also even when medical professionals communicate flawlessly, they're communicating to non-professionals who will necessarily encode their understanding in an imperfect mental model that will likely degrade over time.
Oh absolutely, especially in an emergency. I mentioned the other scenario because when I've told this story before people have been skeptical that a bedside provider could make that kind of error.
In emergency situations (or even routine ones) where I'm administering morphine, I don't need a computer to help me figure out the dose. There are more complex dose calculations where good tech matters far more. Harold Thimbleby has some very accessible talks on safety in health tech: https://www.youtube.com/watch?v=AobMb3S5OtY&t=1034s
I suppose I'd better change my example to something that is more dependent on accurate weight. I do know that at one point there were weight/dose charts for morphine administration, e.g. https://nursing.duke.edu/sites/default/files/documents/SCD-O...
but that could be an outdated practice.
The calculation charts are valid but you can give it slowly and stop at whatever dose the patient reports good effect.
I do highly recommend Prof Thimbleby's talks in this topic.
I don't know anything about the bug other than my provider who I'd communicated and demonstrated the behavior to came back to me confirming that it was a real bug and was being fixed.
"An external audit by the auditing firm KPMG..." Piling one nonsense upon another. I worked with them on a different health project. After much song and dance we ended up with another bland slide deck.
The audit and consulting practices at KMPG and similar organizations are almost completely separate. And in fairness to KPMG, a bland slide deck is exactly what their customers want. If a hospital wanted groundbreaking disruptive innovation then they wouldn't hire a management consulting firm in the first place.
After some 25 years of consulting/professional services in IT and working probably a dozen or more health care industry businesses or hospitals themselves, I can honestly say they are the worst clients and most I will simply refuse now as a rule. I've never met one that wasn't run poorly run, overly political, low paying, high spending, morale/morally deprived... I could go on, but mostly mismanaged applies, and more so this is universal when it comes to their EMR software systems, as every one needs one.
One need only google "Oracle failure" or visit theregister.com to see a list of atrocities committed yearly by Oracle for ERP and Financial system implementation fiascos, buying Cerner for their EMR that was already bad to use could only make it worse. It's like bad and worse got together to make a baby, but what are your alternatives in the space? I never met an ERP that wasn't a mess, or the staff that tend to manage them. Oracle just keeps expanding, because why be a lesser evil?
If you work in the medical IT field, you have my condolences, and my best advice is don't. The only ones that win are the executives above their meat shields and the investors. If you have to visit one in the US, you also have my condolences, as you'll see first hand the product above commentary.
Vanilla is the best flavor. The decision makers at large health systems often waste fortunes customizing EHR configurations based on the mistaken belief that they know better than the vendor. But often it would be better to stick with the software defaults and retrain the users. Of course this is politically difficult when experienced physicians who bring in a lot of revenue threaten to leave if they don't get their way.
I used to work at Cerner for a couple years. Honestly working at Cerner was a decent experience. Biggest private employer in Kansas City so everyone knew about the company. Huge campuses across Missouri and Kansas. Huge gym in HQ campus.
I was aware of this VA project at the time, although I was no part of it. I knew it was going to take years and we basically had no competition. Shortly before I left the company we got a new CEO from Google (Health?). Now looking back this guy was probably brought specifically for preparing for the Oracle acquisition.
We were a team of 5. After the Oracle acquisition everyone left. That team basically disappeared. Three of my teammates left for another IT company in Kansas. Things must have gotten really tough after the acquisition.
I'll tell you how "dangerous errors" work in practice in a hospital
Let's say we design a "dangerous errors" system. What it's going to consist of is some "are you sure you want to do that" style popups, and some sort of error reporting system.
The error reporting system (human entered) is where it gets interesting. The real nasty errors dutifilly get reported, no choice. The nasty and/or embarrassing errors are not reported. You get some of what they call "virtue signalling" errors where a person may report themself and pat themselves on the back for it - small errors. And than finally, about 1-5% of staff, the sorts that don't belong in medicine, will use it as their personal vendetta machine, ham up, falsify, fictionalize stories written against whoever it was that crossed them that day. Every day.
There CAN be on honest fair reporting system in the absolute healthiest of work cultures, with trust i.e. if I disclose I did something wrong I am confident that I won't be in trouble (that also means nobody can character assassinate someone else who crossed them - they won't be in trouble). But medicine, it's the opposite of such a culture, viciously political, unethical interpersonal practices, obsession with preserving ones career, license, and legal exposure over the common sense best interests of the patients.
Every user I’ve heard coming from a Cerner facility has said that Cerner is an unmitigated disaster.
It almost makes sense that the only way to get a customer was to essentially lobby and force your way into a government contract for it, and it’s still an unmitigated disaster.
My heart goes out to those that are going to get fucked over by this piss-poor deployment and be actually, physically harmed by bad EMR decisions and implementation choices.
Oracle acquired them. So, now you have the full power of Oracle standing behind them to convince customers to adopt it, and the same power to keep customers from migrating.
I'm amazed that city, county, state, and federal tech projects never want to clone best-of-show systems instead of starting from scratch. City needs a web site? Clone the best one you can find amongst the tens of thousands of cities already doing that. County jail needs tracking of inmate transports? Clone the best one you can find amongst the thousands of counties already doing that. State needs a sales tax system? Clone whatever other state system is the best. VA needs a system for hospital records? Don't develop from scratch, start by cloning the best system you can find amongst the thousands of existing hospital networks, and customize from there.
That's what they did. If you read the article, it discussed the whole program as being a change from an in house developed system, to an off the shelf system.
> The program launched in 2018 to replace the aging computer system used across VA’s health care network, which serves more than 9 million veterans, with an off-the-shelf product that could handle many of the same tasks: organizing important information including appointments, referrals, prescriptions and patient histories.
> David Shulkin, the secretary at the time, announced that VA would negotiate a contract to buy the records system from Cerner without competitive bidding.
VA leaders said they selected the program because the Pentagon already had purchased a similar Cerner system for the military’s more than 700 hospitals and clinics.
VistA is an old system, and it's definitely "aging." But the thing is that it actually works really, really well. For instance, it kills a remarkably low number of people, which is one of the benchmarks I personally value in an EHR.
One of the interesting things about this is that, from my perspective, VistA's sort of a mesh of servers rather than the hierarchy we might expect from a federal system. Perhaps that's because of the complex interplay between federal and state and local laws. But anyway, there's probably a "station" for VistA near you that serves your area, and that's very similar (though not identical) to the "station" in the next neighboring area/metropolis/state/whatever.
But weirdly it seemed like the plan to roll this out was to replace all of the functionality at a given VistA station, rather than to do a strangler fig sort of thing and work on supplanting VistA's functionality in a specific functional area (whether locally or nationally). I don't know if that's because of the aforementioned complexity of laws, or the complexity of how the system(s) is/are administered, or other reasons that would elude me.
VistA EHR works reasonably well for end users but the problem is that the underlying platform is kind of a dead end. There's no practical technical path to keep it moving forward with major new enhancements (some of which are legally mandated for compliance). Hardly any developers have the platform skills, no one wants to learn (career suicide in most cases), and modern tools don't support it. It's a shame but that's the reality.
I don't disagree at all (disclaimer: I don't work on VistA, but systems that communicate with VistA... close enough that the "career suicide" phrase hits a little close to home).
I think the bigger problem is that we're not meaningfully grappling with the reality of what it takes to replace legacy government systems.
Another grain of sand on the beach of things that we're completely unequipped to deal with, I guess.
A lot of government procurement is bound by strict "competitive bidding" laws that seek to give everyome and their grandmother a fair shake at the contract, in the name of avoiding graft, corruption, and bribery.
This has led to somewhat of an arms race where government workers desperately collaborate with contractors to find a way to sidestep or subvert the bid process and other contractors aggressively seek to inspect and enforce the process.
Developing in-house governmental talent, institutional knowledge, and capacity is of course strictly off the table, as it would reduce opportunities for private profit in basic government services.
I once flagged a bug in Epic, the big EHR system. The system had somehow mixed up kilograms and pounds. For example, a normal adult male weight of 150lbs would be ~68kg, But accidentally save it without converting and get 150kg. Convert back and it becomes 330lbs. Suddenly our reasonably slim man becomes grossly obese.
It's not just wrong, it's extremely dangerous. In an emergency situations, where morphine is commonly administered for extreme pain, the dosage needed to relieve the pain of a 330lb man would kill a 150lb man. Granted the responder at the patient's side would probably realize something is amiss, but a pharmacist in another room filling an order wouldn't have the context, and could make the error.
I wouldn't trust that a nurse or doctor that is bedside to flag that either, though. Hospitals are woefully understaffed, and while they will do there best, we are all just humans.
My wife's grandmother was killed by a second dose of metformin (well kidney failure after a second dose) because the attending that administered the first dose left the room, planning on coming back a moment later, when the next round nurse came in, they noticed the does hadn't been administered (wasn't in the chart), ordered another dose, and injected it.
There were multiple layers that should have prevented that. The prescription shouldn't have been filled for a second time without someone noticing. The first doctor should have filled in the chart before leaving. And the pharmacist should have noticed that it had already been requested.
Too many patients, too few doctors, and with Epic, too many button clicks.
I am very sad to hear that your grandmother died as a result of a medical error, but the details of the story as you remember them aren’t quite plausible.
1) Metformin is not available in an IV formulation 2) Metformin itself is not nephrotoxic
It certainly is believable that a medical error caused kidney failure but it is very unlikely to have been caused by an incorrect second IV dose of Metformin.
I expect GP mistook metformin-associated lactic acidosis in a patient with kidney disease as something metformin-caused. A separate but coincident IV misadministration could be an exacerbating factor. It would have been a rare case, but a plausible explanation for the misunderstanding.
Metformin is relatively kidney safe and not administered by IV. Thank you for clarifying that for anyone that may currently on or considering Metformin. It would be great if medical professionals were infallible communicators and had time to verify understanding, but they are human and we need more doctors and less time-pressure by profit extracting private equity.
Disclaimer: I am not a doctor.
Also even when medical professionals communicate flawlessly, they're communicating to non-professionals who will necessarily encode their understanding in an imperfect mental model that will likely degrade over time.
I'll defer to your expertise. I wasn't there, and just sharing what aunt was "told".
I don't see any mention of IV in the OP comment.
It says the second dose was "injected."
yeah, you can do that with needle and syringe.
Oh absolutely, especially in an emergency. I mentioned the other scenario because when I've told this story before people have been skeptical that a bedside provider could make that kind of error.
Damn, sorry to hear that a nurse killed your grandma :(
In emergency situations (or even routine ones) where I'm administering morphine, I don't need a computer to help me figure out the dose. There are more complex dose calculations where good tech matters far more. Harold Thimbleby has some very accessible talks on safety in health tech: https://www.youtube.com/watch?v=AobMb3S5OtY&t=1034s
I suppose I'd better change my example to something that is more dependent on accurate weight. I do know that at one point there were weight/dose charts for morphine administration, e.g. https://nursing.duke.edu/sites/default/files/documents/SCD-O... but that could be an outdated practice.
The calculation charts are valid but you can give it slowly and stop at whatever dose the patient reports good effect. I do highly recommend Prof Thimbleby's talks in this topic.
Thank you, I'll give them a look.
Wired covered the story twelve years ago of an Epic implementation failure that led to a child’s overdose (and recovery), of note:
https://www.wired.com/2015/03/how-technology-led-a-hospital-... https://archive.is/1QPmK
Opioids are not weight based dosed for adults. Typically pain protocols start at fixed doses based on prior opioid use and titrate up for effect.
Also was this a bug in Epic proper or a site specific customization?
OK, so my example is flawed, but the scenario remains valid. I was going off charts like https://nursing.duke.edu/sites/default/files/documents/SCD-O...
I don't know anything about the bug other than my provider who I'd communicated and demonstrated the behavior to came back to me confirming that it was a real bug and was being fixed.
Meanwhile in Sweden:
> Scrapping the millennium: introduction of a health record in Sweden fails
> The introduction of a new, heavily criticized electronic journal system from Oracle in two Swedish health districts has failed spectacularly.
https://www.heise.de/en/news/Scrapping-the-millennium-introd...
"An external audit by the auditing firm KPMG..." Piling one nonsense upon another. I worked with them on a different health project. After much song and dance we ended up with another bland slide deck.
The audit and consulting practices at KMPG and similar organizations are almost completely separate. And in fairness to KPMG, a bland slide deck is exactly what their customers want. If a hospital wanted groundbreaking disruptive innovation then they wouldn't hire a management consulting firm in the first place.
After some 25 years of consulting/professional services in IT and working probably a dozen or more health care industry businesses or hospitals themselves, I can honestly say they are the worst clients and most I will simply refuse now as a rule. I've never met one that wasn't run poorly run, overly political, low paying, high spending, morale/morally deprived... I could go on, but mostly mismanaged applies, and more so this is universal when it comes to their EMR software systems, as every one needs one.
One need only google "Oracle failure" or visit theregister.com to see a list of atrocities committed yearly by Oracle for ERP and Financial system implementation fiascos, buying Cerner for their EMR that was already bad to use could only make it worse. It's like bad and worse got together to make a baby, but what are your alternatives in the space? I never met an ERP that wasn't a mess, or the staff that tend to manage them. Oracle just keeps expanding, because why be a lesser evil?
If you work in the medical IT field, you have my condolences, and my best advice is don't. The only ones that win are the executives above their meat shields and the investors. If you have to visit one in the US, you also have my condolences, as you'll see first hand the product above commentary.
Vanilla is the best flavor. The decision makers at large health systems often waste fortunes customizing EHR configurations based on the mistaken belief that they know better than the vendor. But often it would be better to stick with the software defaults and retrain the users. Of course this is politically difficult when experienced physicians who bring in a lot of revenue threaten to leave if they don't get their way.
I used to work at Cerner for a couple years. Honestly working at Cerner was a decent experience. Biggest private employer in Kansas City so everyone knew about the company. Huge campuses across Missouri and Kansas. Huge gym in HQ campus.
I was aware of this VA project at the time, although I was no part of it. I knew it was going to take years and we basically had no competition. Shortly before I left the company we got a new CEO from Google (Health?). Now looking back this guy was probably brought specifically for preparing for the Oracle acquisition.
We were a team of 5. After the Oracle acquisition everyone left. That team basically disappeared. Three of my teammates left for another IT company in Kansas. Things must have gotten really tough after the acquisition.
I supported Cerner RHEL servers for 6 years and Cerner is a joke.
Everything has to run as root.
All their scripts are Korn shell hacks from the 90s that have been carrier over.
Their god awful Kickstart scripts were written in stone even if you prove that there were better ways.
Cerner controlled the root password.
Cerner123!
They hard coded passwords with ROT13.
No wonder Oracle bought them.
I'll tell you how "dangerous errors" work in practice in a hospital
Let's say we design a "dangerous errors" system. What it's going to consist of is some "are you sure you want to do that" style popups, and some sort of error reporting system.
The error reporting system (human entered) is where it gets interesting. The real nasty errors dutifilly get reported, no choice. The nasty and/or embarrassing errors are not reported. You get some of what they call "virtue signalling" errors where a person may report themself and pat themselves on the back for it - small errors. And than finally, about 1-5% of staff, the sorts that don't belong in medicine, will use it as their personal vendetta machine, ham up, falsify, fictionalize stories written against whoever it was that crossed them that day. Every day.
There CAN be on honest fair reporting system in the absolute healthiest of work cultures, with trust i.e. if I disclose I did something wrong I am confident that I won't be in trouble (that also means nobody can character assassinate someone else who crossed them - they won't be in trouble). But medicine, it's the opposite of such a culture, viciously political, unethical interpersonal practices, obsession with preserving ones career, license, and legal exposure over the common sense best interests of the patients.
Now take your mother in to have her suffering fixed by these people!
https://archive.ph/4f6JE
Every user I’ve heard coming from a Cerner facility has said that Cerner is an unmitigated disaster.
It almost makes sense that the only way to get a customer was to essentially lobby and force your way into a government contract for it, and it’s still an unmitigated disaster.
My heart goes out to those that are going to get fucked over by this piss-poor deployment and be actually, physically harmed by bad EMR decisions and implementation choices.
Every single time I've heard about an org onboarding Cerner, it was a disaster. One wonders how they can possibly still be in business.
Oracle acquired them. So, now you have the full power of Oracle standing behind them to convince customers to adopt it, and the same power to keep customers from migrating.
Nobody got fired from buying Oracle
What will it take to blacklist oracle from government contracts?
an end to crony capitalism
I'm amazed that city, county, state, and federal tech projects never want to clone best-of-show systems instead of starting from scratch. City needs a web site? Clone the best one you can find amongst the tens of thousands of cities already doing that. County jail needs tracking of inmate transports? Clone the best one you can find amongst the thousands of counties already doing that. State needs a sales tax system? Clone whatever other state system is the best. VA needs a system for hospital records? Don't develop from scratch, start by cloning the best system you can find amongst the thousands of existing hospital networks, and customize from there.
That's what they did. If you read the article, it discussed the whole program as being a change from an in house developed system, to an off the shelf system.
> The program launched in 2018 to replace the aging computer system used across VA’s health care network, which serves more than 9 million veterans, with an off-the-shelf product that could handle many of the same tasks: organizing important information including appointments, referrals, prescriptions and patient histories.
> David Shulkin, the secretary at the time, announced that VA would negotiate a contract to buy the records system from Cerner without competitive bidding. VA leaders said they selected the program because the Pentagon already had purchased a similar Cerner system for the military’s more than 700 hospitals and clinics.
VistA is an old system, and it's definitely "aging." But the thing is that it actually works really, really well. For instance, it kills a remarkably low number of people, which is one of the benchmarks I personally value in an EHR.
One of the interesting things about this is that, from my perspective, VistA's sort of a mesh of servers rather than the hierarchy we might expect from a federal system. Perhaps that's because of the complex interplay between federal and state and local laws. But anyway, there's probably a "station" for VistA near you that serves your area, and that's very similar (though not identical) to the "station" in the next neighboring area/metropolis/state/whatever.
But weirdly it seemed like the plan to roll this out was to replace all of the functionality at a given VistA station, rather than to do a strangler fig sort of thing and work on supplanting VistA's functionality in a specific functional area (whether locally or nationally). I don't know if that's because of the aforementioned complexity of laws, or the complexity of how the system(s) is/are administered, or other reasons that would elude me.
It's, uh, it's a fun situation.
VistA EHR works reasonably well for end users but the problem is that the underlying platform is kind of a dead end. There's no practical technical path to keep it moving forward with major new enhancements (some of which are legally mandated for compliance). Hardly any developers have the platform skills, no one wants to learn (career suicide in most cases), and modern tools don't support it. It's a shame but that's the reality.
https://worldvista.org/
I don't disagree at all (disclaimer: I don't work on VistA, but systems that communicate with VistA... close enough that the "career suicide" phrase hits a little close to home).
I think the bigger problem is that we're not meaningfully grappling with the reality of what it takes to replace legacy government systems.
Another grain of sand on the beach of things that we're completely unequipped to deal with, I guess.
Any time you hear a vendor is being selected "without competitive bidding", you're probably not getting the best solution.
A lot of government procurement is bound by strict "competitive bidding" laws that seek to give everyome and their grandmother a fair shake at the contract, in the name of avoiding graft, corruption, and bribery.
This has led to somewhat of an arms race where government workers desperately collaborate with contractors to find a way to sidestep or subvert the bid process and other contractors aggressively seek to inspect and enforce the process.
Developing in-house governmental talent, institutional knowledge, and capacity is of course strictly off the table, as it would reduce opportunities for private profit in basic government services.