Original URL: https://www.theregister.com/2014/02/21/city_lifeline_london_colocation_data_centre/
Hot racks and cool customers: Colocating in the capital
Where diesel, data and dark fibre meet
Posted in On-Prem, 21st February 2014 10:01 GMT
Feature In a world where data centre duties are increasingly relocated to lower cost territories in the East and beyond, why would anyone want to set up shop in the heart of London? Even if there was a requirement for your data to be UK-based, surely a location in the shires where rents are lower would make sense.
City Lifeline colocation data centre building
That may appear to be the case in most instances, but it all depends on the nature of your business, according to Roger Keenan, managing director of City Lifeline, a colocation centre in based in the heart of City of London. This year is the company’s 21st birthday and although it was originally set up as a disaster recovery facility for traders in the City, this tech stronghold has moved with the times, swapping out coax for fibre and traders for telcos.
Recently, it upgraded its generators to 2N taking it up to a Tier 3 rating, if you buy into that US trade association rating system – presentation here [PDF].
When The Register came calling, the operations guys were busy putting one of the backup generators through the monthly power testing routine. We’ll be looking at this process in more detail later and taking a look at the massive diesel engines and the UPS buffer that keeps those colocation racks up and running for City Lifeline’s clients 24/7/365.
More from the generator room later
Ten years ago, explains Keenan, the clientele was 100 per cent voice telephony, but that has dropped down to about 50 per cent. Services such as Skype have had an impact, so the phone operators that remain are mostly those involved in pre-paid phonecards, enabling cheap calls to far-flung lands.
There’s a broader mix of clients than that, although some he can’t talk about – such as a fashion retailer, which he finds perplexing given the nature of its business – and others he can discuss, which can be equally perplexing.
The International Institute of Strategic Studies (IISS) is a case in point. As a leading global security think tank, you’d think it would rather keep quiet about its data activities but it has even given its blessing for a case study of its colocation implementation to appear on City Lifeline’s website.
The IISS project had several needs according to its CTO Neil Goodman, who sought to virtualise the IT operating environment with additional resources featuring Disaster Recovery and a High Availability environment. Goodman went through the motions of finding a colocation provider, and it was City Lifeline’s London presence that was a major factor, not simply for its metropolitan convenience, but for the benefits that the capital’s global connectivity offers.
Optical collusion
Fibre Alley is a phrase that City Lifeline likes to bandy about to describe its position “at the centre of the world’s internet backbone”, which sounds akin to a spinal tap, but reflects the company’s claim that London is home to more than half of the UK’s data centres and these services need fast, high capacity networks to do business.
Dark fibre makes its mark outside City Lifeline's premises
To prove the point, Keenan suggested I take a look at the outside of the building, which had getting on for 20 different cabinets (manholes) in the space of 50 yards (46m). Many of the cast iron covers were emblazoned with the names of providers such as euNetworks and companies long since gone or acquired – such as Metro Media Fibre Networks (now Zayo) and WorldCom, all conveniently situated on City Lifeline’s doorstep.
“We’ve got 25 carriers in the building and they bring their fibre in from the outside world. They will have a presence here – a rack or even a private suite – and from there they will connect out to customers. City Lifeline itself doesn’t bring fibre into the building, but there are miles and miles of fibre coming into the site.”
The fact that so much of the world’s fibre terminates in London gives the businesses here a distinct advantage for global communications. Add to that the availability of dark fibre in the capital, which, among other things delivers low latency between data centres in Docklands and beyond.
London traffic: outside City Lifeline a cable guy connects fibre for Vodafone and performs tests in his van
OK, so there are some obvious comms and capacity benefits to the London location as well as being able to draw upon the talents of the capital’s highly skilled IT community. However, there is a price to be paid, and Keenan reckons his clients are having to cough up around 30 per cent more to be located here than in a more remotely located data centre, but for many, it's worth it.
Secure holdings
The whole notion of colocation does help contain costs. Customers are not paying for a dedicated area on their premises to be fitted out with their IT kit, which in some instances could be an expensive use of office space. That’s all taken care of here, as it would be in similar facilities around the country.
“Another interesting thing that happens,” says Keenan “is their insurance premiums go down because now they don’t have all their IT on-site.”
You need to get the thumbs-up to enter the building
Indeed, security is another factor and just to get around the place you'll find biometric tech installed on doors everywhere. The racks are all locked up and those sharing the aisles only get keys to their own kit. Security knows who goes in and out and I couldn’t get beyond reception without first producing photo ID and consenting to fingerprint scanning.
With pinkie pressed on the scanner, entry functions much like a glass-panelled air lock for a single person; the door behind you closing before the other one opens. Called an anti-tailgating pod, it’s not for claustrophobes.
If you’d rather not colocate in the same rack space, then you can opt for a suite. Indeed, IISS might not mind chatting about its tech activities but it’s not complacent about its operations and has its own secure room within City Lifeline featuring various sensors and CCTV that link back to its own HQ monitoring it all, to ensure it’s a tamper-free environment.
“If there’s a fire, security always has an override on everything so it can do something about it,” says Keenan. “If there was a cooling problem that needed fixing, we’d tell them first before going in there, otherwise we’d never go in there.”
Blowing hot and cold
Suites aside and fingerprints permitting, you can enter the more open equipment rooms. In a slightly older area, the floor space is taken up with half racks and quarter racks – mostly low power, although the air conditioning still rages. In the more critical areas, City Lifeline takes a hot aisle, cold aisle approach to cooling.
“Here we have the hot aisle as the general area outside, and this is the cold aisle here, other people run it the other way round,” explains Keenan as we step into the chilled chamber of racks, the alarm bleeping away – a shrill reminder to visitors to keep the sliding door shut.
Inside the cold aisle: alarms sound to remind visitors to keep the door closed
“We keep a relatively small floor void with all the power and signal cabling underneath the floor, separated, with all the air handling above. The basic idea is cold air falls into the cold aisle, it goes through all the equipment, it’s heated, and vents into the hot aisle. The hot air rises, goes into the evaporators, cools, and is blown back down across the top of the racks, falls into here and goes right round in a circle.”
Eyeing the equipment items, Keenan points out a couple of items of rack gear and remarks, “A specialised telephony company looks like that now. Ten years ago, that would have been a private suite. Back then a voice telecoms operation needed all sorts of specialised TDM [time division multiplex] equipment and every voice operation needed at least a full rack of kit, often more."
"TDM has largely gone now, and most voice telephony is carried as TCP/IP packets on VoIP (voice over internet protocol). You can run a voice network in a quarter rack on two standard servers plus a router. You need one server to do all the switching and the other to do all the billing, control and management around it. And the router to switch to the IP platform, plus all the software, but that’s all you need.
"You could even manage it all using one server and no router, but most wouldn't consider that to be a good design strategy. So nowadays, the colo requirement for a voice operation can easily be just a separately lockable quarter rack and 10Meg of IP, rather than a whole rack and miles of coax."
City Lifeline argonite gas and hot aisle
Alarming situation
Stepping into the hot aisle again, a line of six red argonite gas cylinders catches the eye in this, one of many, server rooms. There’s a set procedure for fire though. The building is 24/7 manned and if there’s any smoke, the VESDA system alarms security, which sends somebody up to investigate.
The standing instructions basically come down to: if there’s smoke coming out of a piece of equipment, pull all the bank’s plugs out and contact a supervisor. In what would most likely be an isolated incident, this approach would cover most scenarios. If all else fails, the gas floods the area to extinguish a fire, but is operated manually by security. You can’t have folk in these areas when the gas goes off as they would suffocate.
Keenan recalls a fire hazard of a different sort that City Lifeline had to deal with. “A long, long time ago, we used to have a customer whose engineer would go to the pub at lunchtime and come back pissed at 4 o’clock. He’d say he was perfectly all right, and then go round the back of the equipment and fall asleep. We always said to him: what happens if there was a fire and somebody came in shouting if anyone was in there and he was asleep and then let the gas off?“ A sobering thought, but it’s all changed now. If you’re boozed up, City Lifeline’s security won’t let you in.
Dream machines? Hot aisle hiding places perfect for a boozy snooze
Tricky customers
There are other reasons for not being let in at all at City Lifeline. While Keenan is pleased to report that he can’t recall any troublesome clients, there are always exceptions and he knows the customers well.
Naturally, there’s an acceptable use policy in place for those who rely on the site’s IP. If other clients were affected by the antics of a particular customer, they’d get disconnected. He admits, those involved in scammy phone behaviour haven’t affected other users, but he’s had no cause to disconnect anyone, but wouldn’t hesitate to do so if it was appropriate.
He concedes that you never refuse customers, instead you declare that the required services aren’t available or you double the price, as there are some outfits whose reputations precede them as being “not nice people” which sounds intriguing. So how "not nice" is not nice?
“Shall we say, the kind of people who if you asked them to pay a bill and they didn’t want to, might break your legs. There are some of those around. About 10 years ago and all through the dotcom boom there was a lot of ‘wild west’ stuff around like that. It’s pretty well all gone now. The cowboys all went out of business, went bust and were replaced by professional companies who knew what they were doing and did things properly.”
Special ops
As we wander around the facility, five tech ops are busy checking out alarms, aircon equipment and power to server rooms; communicating their findings on walkie talkies. City Lifeline Operations Manager Bill Walsh orchestrates the proceedings as today is the monthly generator test, when the power is switched over from the mains – that keeps the UPS batteries charged – to the one of the two diesel generators. He explains the procedure.
Diesel generator with massive black silencers above it
“We have workshops regularly with my team to go through the electrical schematics. That ensures everyone is aware of who should be switching when. During the switchover, the team is deployed in different parts of the building. However, the first thing we do is make sure the generators are all working, make sure they’ve got fuel. One in duty, one in standby and every month we swap them over. We also run generator tests every week. They are run off-load for 10 minutes because if you run them for any longer off-load, they don’t like it, they coke themselves up. They're designed to be run on load.”
And today’s the day when one of the 30 litre V16 generators is taking the strain. The switchover happened as I chatted to Keenan in the boardroom and the only way to tell was the building lights went out, as they rely on mains power and only back-up lighting would kick in.
The actual client equipment is always fed by the banks of UPS lead-acid batteries producing what Keenan refers to as “synthetic mains” and we went along to the UPS area where these batteries were hard at work.
Taking charge
“The electronic equipment in the colocation racks here is always fed from synthetic mains, it never sees the conventional mains power supply. So the synthetic mains powers the equipment and the "real" mains is only used to charge the batteries up. So if the real mains stops, all that happens is the synthetic mains carries on working but it takes its power from the batteries. The batteries start to discharge and when you put the diesel generator in, it begins to charge the batteries up again.”
UPS room with banks of Yuasa lead acid batteries and four Reillo AROS power conversion units at the back
Similar to 12V DC batteries found in cars, banks of Yuasa SWL2500EFR lead-acid cells are configured to deliver an output of around 480V. Rectifiers and other circuitry is utilised to convert the DC to AC which is distributed around the data centre to the server room equipment, the air conditioning and alarm systems too. Needless to say, these batteries stack up to a considerable weight, as Keenan remarks, “You don’t put them in the middle or top floor, you put it all on the ground floor.”
Reillo UPS monitoring – click for a larger image
The UPS room fizzed with the sound of electrical activity as if teeming with android insects rising above the noise of the low frequency boom of the air conditioning that was the backdrop to all of the equipment rooms in the building.
There are several UPS rooms in the building but on show was an area recently fitted out with four Reillo AROS UPS units capable of delivering 120kVA apiece. Two UPS systems form a 2N pair to support the customer load and the other two handle the chillers. On the wall at the back was a curious arrangement of Dent Instruments ElitePro XC power meter, coupled with a DataProbe iBoot Hub.
Dent Instruments and Dataprobe monitoring kit
Used for monitoring the power loads from the various sources – mains, generator and battery – it allows Keenan to see how all this is shaping up from an app on his iPhone. From the graphs on the screen he refers to the bottom-centre one first. “That was the generator which was generating no current and then went up and started showing that change. Over on the left, that’s the switchover, as the two mains feeds were failed intentionally. It’s all completely automated. We all get alerts, so we know what’s happening.”
Power monitoring from an iPhone app – click for a larger image
New power generation
Around the back of City Lifeline is where the generators are kept in a separate outhouse. Although it’s quite cramped inside, with each generator having its own separate room, the building itself is quite imposing. Silencers keep the outside noise to a purr – I've heard noisier ice-cream vans – with chrome exhaust pipes snaking out of the generator room and climbing the wall of the main building.
Inside, one of the generators is on load, proving its worth for the next hour. The noise inside is deafening, but you can nose around with ear defenders on. The MTU diesel engine hails from Friedrichshafen, and was built in the old Zeppelin motor factory. These days, MTU is owned by Rolls Royce Power Systems AG. The engine itself is capable of delivering one megaWatt on the shaft. It turns a 1150kVA Stamford alternator which is oversized in order to suppress all the harmonics that the UPSes generate from the rectifiers.
MTU 30-litre V16 diesel generator with Stamford alternator in the foreground
After a quick gander, Keenan points out the safety features from inside the quieter spare generator room. “We’ve various fire suppression devices acting on three levels. One is the engine itself, which says: 'I’ve got a fault, shut down'. The second level behind that is a rate of rise smoke detector, which says: 'this is getting awfully hot very quickly' and sends a signal into the main building fire system.”
Valve shut-off relies on suspended red weight mechanism - click to enlarge
The third system is a curious set-up of taut wire suspended above the generator that features a tiny piece of metal that melts at 100˚C. With a touch of sarcasm, Keenan remarks, “It has a wonderful, sophisticated, totally mechanical system that consists of two weights. If the metal melts and the weight drops, it closes that big valve. That’s the connection between the ‘day tank’ and the engine, so it drops and stops the fuel going to the engine. The other weight acts to stop any more fuel coming into the day tank container.”
Tanked up
A day tank is a reservoir of fuel that’s pumped out at the correct pressure to the diesel engine injectors. And talking of fuel, City Lifeline keeps 14,500 litres on site, which is expected to last around three to four days in the event of an outage. Pour that into your family saloon and at 47mpg it’ll get you 150,000 miles – about 400 miles a day for a year. Or if you reckon you could siphon it away, you’d not be far off 20 grand’s worth of fuel.
Having that amount of diesel hanging around involves some maintenance of its own. The fuel needs ‘polishing’ and City Lifeline calls in contractors for this particular job. It’s a process that removes traces of water, bacteria and other particulates that can accumulate in the tanks, ensuring that the generator gets a clean and healthy fuel supply every time its drawn upon.
Load shedding between power sources is managed automatically in the Switch Room
As the tests come to a close, we’re taken to the switch room which manages the handover of power to the UPS array from the mains to the generators and vice versa. One of the team is monitoring the current draw as a clatter of switching is heard.
There’s a sequence to how the switch gear operates, as the power isn’t flipped over in one go. A load-shedder co-ordinates a gradual return to normal. The amp readings on the generator meter he’s checking steadily drop as the load shedder returns the power to a full building load and the generator has less work to do.
Reading the meters – click for a larger image
Back in generator room it’s quieter now as the diesel engine is in its wind down phase for cooling, eventually switching off automatically. You hear the engine whine a descending portamento for a few seconds as it comes to rest.
Even though these tests are performed regularly each month customers are kept informed, as some of them might choose to have engineers on standby during that window. The test routine gives customers some reassurance that there are no problems and they receive notification of the next monthly test window.
Thermal protection
A few years ago, City Lifeline upgraded its generators, something its customers would have definitely noticed. It was a challenge of a different order, as Walsh explains.
Roger Keenan puts the size of the generator installation into perspective
"From outside, it’s a pretty tight road and the task involved transferring the old generators out and bringing the new generators in with temporary generators on site all the while. You plan it, you have the necessary equipment in place, the personnel in place and hope that on the day everything does fall into place."
He adds: "At the end of the day, you’ve always got to rely on people. You only need one person to ring in sick and it has a knock-on effect. Once you’ve met with that problem, it’s how you address it and move forward. In this instance, it all went according to plan and on time and we were running fluently on power which was our objective."
Power distribution panels
Planning for the big stuff is one thing but often problems occur in hidden areas that can take some time to get to a critical stage. At City Lifeline the company runs through an annual test of its electrical distribution systems using thermal imaging cameras from Testo. To the trained eye, failing components can be detected from the heat profile they produce and replaced before they break down completely.
“It’s quite a big exercise,” says Keenan. “There are hundreds of panels and distribution boards in this building and it means somebody has to go through every single one, take all the covers off, check them with a thermal imaging camera and then put all the covers back on again. It’s not something you can do in an afternoon; it involves two electricians spending a week doing that.”
While electrical distribution gets the once over every year, the communications side of City Life is continually evolving. These days Walsh grapples with Fujikura fibre optic splicers more than Cat5 crimping. “It’s delicate,” he says “You need steady hands. Stripping it and getting it clean before it goes into the machine is the hard part. Six years ago it was just all coax. Now, a lot more are going over to fibre.”
Economies of scale: quarter racks colocate with full size racks
Keenan observes the thinking behind this transition. “Fibre isn’t as difficult as it used to be and it’s not that much more expensive. Put a fibre in, then you can say, if I ever need to go to 10Gb/s I can do that. Cat6 you can do 1Gb/s; Cat5, generally you wouldn’t go to more than 100Mb/s. For the voice people, 100Mb/s is all right – that’s 1,000 simultaneous phone calls. But for those who want expansion capacity, they would put fibre in to start with.”
Trend spotting
So what else is new? With years of experience running a central London colocation data centre, Roger Keenan must have noticed some trends.
“What’s happened in the past 10 years is that the space requirements have gone down, but the power requirements have gone up. Generally, everybody’s experienced that. Ten years ago, typically what we sold was a 4 amp rack. Then it went from an average of being a 4 amp racks to an 8 amp rack with some 4 amp racks in it. Then it went to being an average of a 16 amp rack with some 8 amp racks – and we stopped making 4 amp racks all together.
"And that’s the way the world has gone and it may continue to do that, except that there’s a whole trend away from Intel chips towards ARM-based servers. If that trend moves – and I’ve every reason to believe it will, it’s a very good thing – you can envisage a situation where somebody, instead of building something with two Intel processors in it, builds something with six ARM-based processors in it. And it’s just as powerful and consumes a quarter of the electrical power… and you can see that coming. That will take us back in the same direction, so that we’ll end up needing more space, than power. These balances change all the time.”
Could a mix of Intel and ARM servers give a new meaning to hot aisle, cold aisle in future?
Backtracking a bit, he adds, “Actually, it never really happens like that. Once there’s the opportunity to use more computing power, and it becomes available, people will start to use it. The demand for computing power will never go down, it keeps going up.”
It’s a fair point but given the need to balance energy costs and computing power, perhaps we’ll see hot aisle, cold aisle configurations of a different nature – with Intel and ARM servers co-existing in colocation data centres. ®