Hey, there! Log in / Register
Shorting power supplies spark fire in Downtown Crossing data center
By adamg on Fri, 06/22/2018 - 6:09pm
Firefighters responded to the eighth and seventh floors of 1 Summer Street in Downtown Crossing late this afternoon where rooms containing uninterruptible power supplies and large batteries at Markley Group's data center began filling with arc'ing and smoke and then water when the sprinklers went off.
The outage hit MIT, which uses the data center for many of its servers. MIT says it hopes to have a bypass in place by 10 a.m. on Saturday.
Massive @BostonFire presence in DTX. @universalhub pic.twitter.com/snNQyh8aV3
— Aaron Kravitz (@aaron_kravitz) June 22, 2018
Neighborhoods:
Free tagging:
Ad:
Support Universal Hub
Help keep Universal Hub going. If you like what we're up to and want to help out, please consider a (completely non-deductible) contribution.
Comments
Data Center
There is a data center in that building. Lots of power cords, UPSes, HVAC equipment, etc.
Yet they still have water sprinklers
Yikes!
Fire
Yes - many of these DC in old bldg's still have sprinklers...insane.
Anyone have any updates?
Hearing from a couple of folks
That the Markley data center was knocked offline. If I find out more, will post.
The whole thing. Yikes. That
The whole thing. Yikes. That's impressive if true, but I have services live in there and they didn't go offline, so I sorta don't think so.
Yes, this cut off MIT's email
Yes, this cut off MIT's email for most of the evening (along with several other IT services). See https://3down.mit.edu/
What do you propose as an
What do you propose as an alternative? Halon, Inergon, FM-200? They all have additional issues in a large data center environment.
halon hasn't been legal for decades
Inergen, not inergon.
Often times you don't have a choice in which agent to use, because they have different requirements in terms of agent delivery rate/quantity, how sealed the space is, and so on.
Try Nitrogen
Suffocation hazard, yes, but non corrosive, cheap, easy to find a supply (like when you havea GC lab built in an ISO module shuttling between Johnson Island and Hawaii) and nontoxic. After all, we aren't talking about a football field - just smaller sealed rooms rented to individual companies.
N also works - this is a system used for power turbine enclosures, among other water sensitive facilities:
/factsplaining
Life Safety Hazard
That looks like a great system for use in enclosed space with no possibility of a trapped person. A container on a ship is a fine use for it.
If you want to avoid water, probably the best would be an Ansul system. Dry chemical. Messy, but they do the job. They're everywhere. Go to a restaurant , look in the kitchen. Go to a gas station and look up. Fairly inexpensive, meets codes and the insurance companies love them.
No life safety hazard. 1010 Mass Ave will be happy.
I'm a native Code Talker. A 'codesplainer' but for real answers, call 1010. They'll know.
Batteries present a problem. How do you kill the power to the source of the power?
duplicate
.
Not just "old" DCs
Wet fire suppression has been the choice for most companies for over a decade. Dry (gaseous) suppression is too difficult and expensive both from capex and opex perspectives for most datacenter facilities and clients. Fires in datacenters are incredibly rare for a lot of different reasons, dry suppression is difficult and expensive, and the money is usually better spent elsewhere.
You only see dry suppression in specific applications where protection of physical infrastructure is paramount, or where water can't be used because it (or shutting off power, as all sprinkler systems in DCs do first) would cause critical failure.
Stick to what you know, swirrrly
They don't "still", that's what was spec'd from the very beginning.
1)The systems shut off power to the room as part of the activation process and there are pre-activation alarms to give on-site staff a chance to intervene if the fire is manageable with an extinguisher or if the alarm is accidental (ie construction for example kicking up a lot of dust.)
2)Gas suppression systems are incredibly expensive to install (both in planning, preparatory construction ie necessary sealing of the room, and actual system installation), require at least one full system test with the suppressant (which is also pretty expensive), and a lot more inspection and testing, and if the room is big enough, you could be looking at needing to purchase, install, maintain, and train people on SCBA equipment.
3)In today's world, it is more resilient and generally useful to have clustered or replicated systems because they can provide coverage for more incidents than just fire.
4)Doing gas suppression for a datacenter the size of Markley is difficult because it's massive. It has the footprint of a football field if not bigger, and numerous floors high. All full of computers, networking, and telecom equipment. That said, *some* MDC customers have gas suppression; I've seen the systems. They tend to be used in areas where it the physical equipment and wiring/fibre is itself very important.
In short: spend the money on computers elsewhere, because they can cover you for things like equipment failure, theft, power failure, malicious action IRL, and so on. Money spent on gaseous suppression is money that only protects you from fire.
Swirrrly, you're one of the most prolific commenters here and you're honestly just like a mansplaining dude....you speak authoritatively with your opinion on all sorts of things you clearly know absolutely nothing about. I dunno if it's your PhD or just your personality but you seriously need to put a sticky note on your monitor that says:
"Do I have any education or training on the topic at hand?"
Her Knowledge Is Vast, And Her Anecdotes Are Prescient Treasures
My two cents...
...on that solid and excellent foundation...
NFPA sets guidelines.
Rule one...life safety is paramount.
Rule two...ya, property.
If there's a life safety hazard, that comes first. Burning computer cables can probably floor you in seconds. Nasty stuff, like a car fire or other burning plastic. Dumping water on it is not the worst of ideas. It's like a manhole fire. Primary concern is to stop the spread of the fire. Nothing like a heavy stream of sprinkler water to cool and knock down a fire, especially in its incipient stages. So, you lose some equipment and save the rest.
Also good point about backups...how often do you replace servers? Is it worth it to cover all of them if they're trash in eight or nine years?
This was a battery backup room. All bets are off. Think Teslas in a building.
Not so fast there
You make it pretty clear that you have never been to the Markley Data Center. It isn't open rooms, it isn't the size of a football field.
You might want to read what someone described below because you have never been to this facility. Your description of a wide-open rural warehouse-style data center says a lot about the last time you set foot in such a place.
Markley is made up of self-contained sectors - rooms, really - that contain an emergency in any single area. Like bulkheads on a boat. They do offer gas fire suppression if you want to pay for it, too. You might want to do that if your equipment is specialized.
Maybe you should stick to what you know.
When the building is worth
When the building is worth more than the datacenter inside it, you have sprinklers.
Data Center
Google the address 1 Summer St and look at the Satellite view
The reason the building is worth protecting is that in addition to whatever overheated and caught fire in some rack sits under all the stuff on the roof and most likely above the floor with all the telecom stuif to connect to wherever at incredible rates
ps my e-mail which is an mit.edu was hardly affected at all
Sprinklers Required
Boston Fire Dept is the Authority Having Jurisdiction and I believe they require a "wet" system for fire suppression in the Data Center. Most Data Center Suppression systems in Boston are double zone interlock pre-action. Dry pipe sprinklers until two smoke detectors are triggered filling the pipes with water. Sprinklers won't discharge until ceiling temps hit 165F when the fusible link in the pendant melts.
Data Center in Somerville Available
If anyone affected by the Markley fire needs an alternative data center, endoffice (endoffice.com) is available all night and tomorrow to install equipment at a data center a mile away on the Charlestown-Somerville line.
Coresite
Coresite on Inner Belt Road in Somerville is the best DC around
Jeff and the crew there are a
Jeff and the crew there are a great bunch
Unfortunate this happened - has been a well run facility for many years
I Imagine Those Uninterruptible Power Supplies Were Interrupted
Markley
I've had a few companies who have had equipment in that data center.
That DC is the Boston Internet Exchange, which over 100 companies have cross connects in that building. Essentially its the hub for much of the internet for the northeast.
http://www.markleygroup.com/services/boston-internet-exchange
As far as that building itself.. I think Markley has 4 floors in that building (1,2 are Macy's, 3-6 are Markley.. 6 being Markley's corp offices, 3,4, 5 being DC). Depending on who you are with and what service you're there for will depend where you go. But from what I've seen are large hallways of doors. Many have familiar names (i.e. Comcast, Partners, MIT, Harvard, RCN, etc) on them. Some doors just lead to are just shared spaces.. like my former company's. We had a cage where we had a few cabinets. Some companies just rent cabinet space too.
But each one of these rooms.. it's all separated and air locked. Each space probably has its own fire suppression, alarms, heating, cooling, and electrical systems. This is by design, and is one of things that make Markley.. well Markley (this DC is highly regarded in tech circles). It's pretty impressive if you ever can go inside... it's also pretty impressive knowing that it used to be a Jordan Marsh :-)
Regardless, I am guessing only *parts* of Markley are down. I just did a few traces to customers I know who are inside that inside, and they are still online. So I am sure its restricted to what customers used that particular equipment. MIT must be one of them.
but again it's what makes Markley.. Markley. One part can go down and the rest can still go. And tbh, if they were entirely down, we'd have so many internet issues in our region right now....... (Comcast & RCN having large presences in that DC)
There's definitely DC space
There's definitely DC space on 7.
FWIW (or speculate away..) there was a notice for UPS Static Bypass scheduled for June 28th. That same UPS that was identified, is the one that malfunctioned. Stuff happens, just glad no one got hurt.
Floors
Yeah I only went by what i saw (remember) in the elevator :p Maybe there was a 7. I know they have a shitton of crap on the roof.
I worked all day yesterday on getting stuff back up
I work at one of the affected companies.
floor 3 is office space and the shipping and receiving area.
4 is datacenter. 5 is the office and some more datacenter.
6, 7, and 8 are all datacenter space.
This affected floors 7 and 8, they did a full power shutdown on those floors.
There was water in the
There was water in the basement as well.
Why is so much of the
Why is so much of the Internet routed through one building? You'd think in this day and age, with the cloud and everything, that there would be multiple redundant sites and paths.
it's getting there
That is indeed a problem for much of New England, and also in other metro markets too. NYC is getting more diverse lately with new players providing meet-me rooms outside of Manhattan (165 Halsey, NJFX, 1025Connect, etc).
Boston now has the Massachusetts Internet Exchange (MASS-IX), which provides internet peering exchange across multiple redundant sites and paths, as opposed to housing peering exchange in just one location.
Water sprinklers bad for a data center
Anyone who knows the basics about water and electricity knows that water sprinklers are bad design. Non water fire suppressing system is what's needed.
Well ...
Lets see how many people attack you for daring to mention that such systems exist and are even an option in this particular center - while describing them as a football field or even mentioning halogens.
Nitrogen exists and many places use it.
I think someone has things mixed up with a grow.
That is incorrect
Gaseous fire suppression is incredibly expensive, and they have gone off accidentally many more times than they have put out fires. Pre action sprinkler system (i.e. dual interlock and dry pipe) are the norm in electronic environments such as data centers.
In fact, a leading industry standards group is in process of recommending the cessation of gaseous fire suppression systems in that environment. If anything thing, gas systems are the outdated method.
Wasn't just MIT
Oracle went down for a time, too.
Let's face it: there's no good time to do maintenance on power systems - what's good timing for one client is never good for another.
I just hope whomever was working on the systems are OK.
My rule of thumb: "if it's more voltage than what it takes to start a car, I don't mess with it."