Can we talk about the little backdoors in data center servers, please?

Black Hat Data centers are vital in this cloudy world – yet little-understood management chips potentially give hackers easy access to their servers in ways sysadmins may not have imagined.

The components in question are known as baseband management controllers (BMCs). They are discrete microcontrollers popped into boxes by the likes of Dell, HPE, and Lenovo to allow data-center managers to control machines without having to brave the chilly confines of a server farm. They can be accessed in various ways, from dedicated wired networks to Ethernet LANs.

BMCs can be used to remotely monitor system temperature, voltage and power consumption, operating system health, and so on, and power cycle the box if it runs into trouble, tweak configurations, and even, depending on the setup, reinstall the OS – all from the comfort of an operations center, as opposed to having to find an errant server in the middle of a data center to physically wrangle. They also provide the foundations for IPMI.

"They are basically a machine inside a machine – even if the server is down, as long as it has power, the BMCs will work,” said Nico Waisman, VP of security shop Immunity, in a talk at this year's Black Hat USA hacking conference on Thursday.

“They have a full network stack, KVM, serial console, and power management. It’s kind of like the perfect backdoor: you can remotely connect, reboot a device, and manage keyboard and mouse.”

It’s a situation not unlike Intel’s Active Management Technology, a remote management component that sits under the OS or hypervisor, has total control over a system, and been exploited more than once over the years.

Waisman and his colleague Matias Soler, a senior security researcher at Immunity, examined these BMC systems, and claimed the results weren’t good. They even tried some old-school hacking techniques from the 1990s against the equipment they could get hold of, and found them to be very successful. With HP's BMC-based remote management technology iLO4, for example, the builtin web server could be tricked into thinking a remote attacker was local, and so didn’t need to authenticate them.

"We decided to take a look at these devices and what we found was even worse than what we could have imagined," the pair said. "Vulnerabilities that bring back memories from the 1990s, remote code execution that is 100 per cent reliable, and the possibility of moving bidirectionally between the server and the BMC, making not only an amazing lateral movement angle, but the perfect backdoor too."

The fear is that once an intruder gets into a data center network, insecure BMC firmware could be used to turn a drama into a crisis: vulnerabilities in the technology could be exploited to hijack more systems, install malware that persists across reboots and reinstalls, or simple hide from administrators.

Sadly, the security of the BMCs is lax – and that's perhaps because manufacturers made the assumption that once a miscreant gets access to a server rack's baseboard controllers, it's game over completely anyway. Here's the stinging conclusion of their study:

The BMCs, by the way, use fruity hardware. Take HP’s Integrated Lights-Out (iLO) system, which is embedded in the ProLiant server range. The older version, iLO2 uses an antiquated NEC CPU core that was popular in optical drives back in the day, while iLO4 has a more modern Arm-compatible core. Dell’s version is the Integrated Dell Remote Access Controller (iDRAC) that uses Linux running on a variant of the SuperH chips once used in some gaming consoles.

Most BMC chips run their own web server, typically based on the popular Appweb code. This can reveal the exact operating system and hardware setup of the chip if pinged correctly. Waisman and Soler also found a list, published by Rapid7, of the default passwords for most BMC systems.

The duo probed whatever kit they could get hold of – mainly older equipment – and it could be that modern stuff is a lot better in terms of security with firmware that follows secure coding best practices. On the other hand, what Waisman and Soler have found and documented doesn't inspire a terrible amount of confidence in newer gear.

Their full findings can be found here, and their slides here.

Of course, data center managers aren’t stupid, and BMC services are typically kept behind firewalls, segmented on the network, or only accessible via dedicated serial lines – and certainly shouldn't be facing the public internet. However, Waisman and Soler said they found plenty exposed to the web.

The bottom line is that IT admins need to assess the routes to their BMC services, make sure none are internet facing, and harden up access. Once an attacker establishes persistence with a BMC, you'll really wish you'd taken their advice. ®