This article is more than 1 year old
What Murphy’s law has to teach you about data centres
'Our IT guy just let me trigger a total network outage'
Data centres are loud, noisy places. So loud, it's often hard for anyone to hear your screams and cries of frustration when Murphy decides to make you the example for his law.
For some, this is a good thing. For others, this is why social media was invented. Outages of all kinds echo out into social media like waves in a pond. Just a few weeks ago, Linux-based CSP Linode had an outage in their Hurricane Electric Fremont 2 colo facility. As you can imagine, many a service provider were not particularly thrilled about the outage, despite it only lasting a few hours.
So, HE's Fremont 2 datacenter has a power outage. One out of eight generators fails, which takes down the entire suite Linode is in.
— Sparkly Securobod (@sindarina) May 31, 2015
In fact, it's hard to find anyone sympathetic to the cause of a CSP or MSP suffering an outage of any kind and for any reason - whether it be incompetence, acts of $deity or a plain old accident. Regardless of price point, reputation or location, the expectation is 100 per cent uptime.
Relaxed support staff during datacenter outage! pic.twitter.com/x0sCF1ODde
— gethash (@gethash) May 13, 2015
Murphy's law is typically stated as "anything that can go wrong, will go wrong." Taken at face value, it deserves respect in all of your actions as an administrator. Trifle with the wrath of Murphy and you may find yourself in this unfortunate scenario. Should have bought a lottery ticket!
Note to self, it is possible to take down one data center and have a lightning strike take power out for the other. Achievement unlocked.
— Jeremiah McClain (@jer_mcclain) June 14, 2015
You know you're truly working in a utopian heaven when you've never been dragged out of bed in the middle of the night to go and poke a switch, let alone call contractors and repair teams to do things like emergency repair a genny or deal with an HVAC malfunction.
Back home from nightly excursion to a data center. Dragons had to be slayed ... major HVAC malfunction.
— WorkingHardInIT (@WorkingHardInIT) February 4, 2012
It’s a public holiday so OF COURSE there’s a network outage.
— mikeforbes! (@mikeforbes) May 31, 2015
One of the worst feelings in the world is knowing the services and equipment you're responsible for maintaining going up in figurative smoke for reasons entirely beyond your control and then having to futilely explain to the boss "but wait, it's not my fault!" This is usually accompanied by a flood of people asking you (while you're trying to fix the issue, no less) "Is it up yet? When will it be back up? Why is it taking so long? Can't you do anything to make it go faster?"
Power outages in #ymm today interrupting our network services. It's one of those days :-( @FMPSDTech @FMPSD pic.twitter.com/oHZmcMQzuE
— Cloudy Miguel (@CloudyMiguel) June 3, 2015
It's good to have some form of backup plan. In a perfect world, we would all have more redundancy and disaster recovery than we know what to do with, but budgets don't always allow for that. Thankfully, 4G and LTE are there for us in the darkest of times.
Our network system is down due to Hawaiian Tel major circuit outage in Aiea. Thanks to data on phone, we can still do -some- work...
— Joannie (@trulyjoannies) January 15, 2013
When you do have your disaster planning in order, you can shrug off all kinds of major issues. This team didn't bat an eye or lift a finger during a power outage thanks to their foresight and testing.
Watching a webinar with @NwoodsChurch Comm team even during a power outage... Love it when Disaster planning works pic.twitter.com/bdnNJz6XHB
— Jason Lee (@JasonMLee) June 11, 2015
Not everyone is quite so lucky. As a society, we've made carbon our slave; but in IT it's easy to feel like it's the other way around.
nice 2nd circuit outage on our network...so happy to be a one man IT team..
— Ben (@suidroot) November 28, 2009
You can run an entire business out of a cardboard box if you really put your mind to it. Office space is not cheap, and as IT requirements have grown over the years sometimes one has to make compromises in order to fit all of their gear physically into the building. Sometimes, there are oversights in those compromises. Murphy isn't having any of that.
Office network outage due to my pot pie causing the microwave to pop the circuit it shares with the server wall
— John Todd (@john_todd) December 3, 2012
And finally, some people just want to watch the world network burn.
Take it down! Our IT guy just let me trigger a total network outage (scheduled) at #Fox13. #FeelThePower pic.twitter.com/5PI9PcvbAw
— Chris Chmura ✈ FoxTV (@Chris_Chmura) June 4, 2015
®