Hey, there! Log in / Register
Global security-software glitch forces cancellation of all elective surgery and office visits at Mass General Brigham hospitals
By adamg on Fri, 07/19/2024 - 9:17am
An attempt by a computer-security firm called CrowdStrike to send out an update for its Windows security software has taken down computers and mobile devices around the world, including at Mass General Brigham, which sent out an urgent message to staff this morning not only canceling all non-emergency procedures and visits, but telling doctors and nurses they shouldn't even try to access the hospital patient-information sytem:
We continue to experience a major Digital incident that is affecting all Mass General Brigham hospitals and sites as well as many major businesses across the country. All Mass General Brigham sites should be using downtime procedures for all systems. Due to the severity of this issue, all elective and non-emergent surgeries, procedures, and non-urgent ambulatory care at Mass General Brigham sites will be cancelled for Friday, July 19, 2024.
All staff who were scheduled today should report to work even if their work area clinics and cases are canceled, unless otherwise directed by their manager.
If you have a MGB device and you have not already used it today, please do not turn on the device or log in. This will help to expedite our resolution to the issue.
We have activated the incident command across the system. All leadership and technical teams are working on addressing this incident.
The problem relates to an update to CrowdStrike's software and how it doesn't play nicely with the Windows platforms on which it was installed.
Mass General Brigham IT sent a second message:
All staff who were scheduled today should report to work even if their work area clinics and cases are canceled, unless otherwise directed by their manager.
If you have a MGB device and you have not already used it today, please do not turn on the device or log in. This will help to expedite our resolution to the issue.
This is a reminder that MGB is on downtime procedures. DO NOT ATTEMPT TO ACCESS EPIC UNTIL THE ALL CLEAR IS SENT.
Topics:
Free tagging:
Ad:
Support Universal Hub
Help keep Universal Hub going. If you like what we're up to and want to help out, please consider a (completely non-deductible) contribution.
Comments
Ouch
Thankfully my work.. a cloud provider.. is not affected.
But ouch. What a way to bring down a hospital to nearly a halt.
But the cloud is suppose to be better. LOL Remember, the cloud is just someone else's computer and not this magical thing.
I don't think the cloud has
I don't think the cloud has anything directly to do with this glitch, though. The cloud is involved in what the software is supposed to do, but the software runs locally on companies' computers, and that software updated itself in a way that causes Windows to crash. (Downloading updates from a server might be one of the many things tech companies try to sell as "the cloud" but I don't think that's what CrowdStrike means by it.)
Yeah
It was a normal update that installed fine but caused the computer to hang on reboot. Fairly easy fix for someone who knows how to boot windows in safe mode but that's a huge number of systems to fix for the few (if any) techs who are in person at these places.
I've always been skeptical of things like Crowdstrike. It seems like trading one risk for another.
Nope
As someone who used to work for a cybersecurity company...
Nope It is not. Why? It will take one breach that is caught by crowdstrike to pay for itself for years. If a breach is not caught, it could cost a company millions in lost revenue.
I've watched breaches happen in real time, its amazing how fast this stuff works. And when you have a product like crowdstrike, it will intercept it and/or at least be able to log everything so the forensics are easy to do.
Fwiw, not a huge fan of CrowdStrike. I prefer Rapid7 instead. Local company, solid product, and their CEO can play a fine game of table tennis (when they were in at 100 summer, we had a challenge with them.. as the company I worked with was across the hall)
Hundreds of millions in revenue lost today too
You point is valid which is why I'm strongly encouraged or required to use CS on my systems. (And do so willingly.) But today's failure is going to cost some firms just as much as a breach.
Last night's Crowdstrike failure wasn't that bad, all told, but the failure shows how it would be possible for a malicious group to push out an attack via CS if they gained access to that firm's tools.
It's basically how security guards are useful to protect against intruders. But if the security guards become the intruders themselves, you have even bigger problems.
Cloud
Yes its the cloud because CloudStrike lives in the cloud (the calling home server). And its the lack of update coming from it is the issue and why things are crashing..
Today cloud can be anything that isn't local to your computer. Cloud does not have to be things like Azure or AWS and such. If it lives somewhere else... its "cloud"
(remember I work for a cloud outfit.. and we aren't AWS or Azure or etc. Yet we are still "cloud")
Isn't redundancy in IT
Isn't redundancy in IT systems of the state's health care monopoly a public health issue? Will there be an investigation or at least a 311 report?
You're special
You're special aren't you.
No it is not because there are other hospitals that are in functioning order like BI-Lahey and (sadly) Steward Healtcare hospitals.
Huh? What's this utter nonsense about?
State healthcare monopoly? Last I checked, MGB is an independent corporation. And it's far from a monopoly. I work for a different large healthcare provider and our systems are fine. The biggest threat to health care tech wouldn't be something attacking Windows anyway, but something attacking EMR systems, such as the EPIC system mentioned in the MGB text alerts.
Regardless, you clearly have no idea what you are talking about. If you did, you'd be embarrassed by this sad attempt to troll. Trolling is an art form, and you are no artist.
The man without a job has opinions!
Guess what, chump - this was NOT limited to MA.
I have family hospitalized in VT and they were not discharged on time due to this glitch. I also have family working in other state's hospitals and a VA facility and they have had issues, too.
New Hampshire's 911 system went down last night
So did all the display boards at North and South Stations. They were just coming back online this morning when I got on my train.
So, CrowdStrike feller gives
So, CrowdStrike feller gives a full-throated apology for his service glitch. That’s good. And, we’re confident it’s a matter of the system not playing nicely with the environment and not a long undetected adversary, or allied agency?
But I wanted to get my lipo
But I wanted to get my lipo NOW!
Elective =/= unimportant
Elective just means it isn't an emergency. My cataract surgery was considered elective, as was my lumpectomy and my polyp removal. They all had to happen, preferably very soon, but I wasn't going to die if they were postponed a few days.
It's alarming that some obscure company...
... can shut down so many vital resouces by accident.
Imagine what a malicious actor could do in that position. If this country ever goes to war, we'll probably learn of it when all our TVs go blue-screen.
Obscure?
Uh. No.
Over $1 billion in sales is definitely non-obscure.
LOL
"Obscure?" Solely because you haven't heard of it before today?
And yet MGB Patient Gateway still works fine
I was able to log into it today, and found everything there that I expected to find.
I'm just glad I was able to
I'm just glad I was able to get through my appointment w/the orthopedic surgeon and blood/urine testing done at MGH yesterday. I don't have a UTI, my thyroid is still wonky, and I'll be getting a knee replaced in November, but at least I can plan around the surgery now.
windows > linux ?
ī fīnd it fassinnating how deep ħis bug iz.
ī olwaze ħot gnu/linux wuz domminnent in sərvərz.
@adamg: hwut iz hōsting ħis website ?