The strange case of the data breach that stayed online for a month
Your security is only as good as your partners' ability to fix messes and flush caches
A couple of weeks ago Jeff* quit his job at the Singaporean branch of a major enterprise technology vendor that is, if not quite a household name, certainly known to most IT professionals.
Not long afterwards he Googled his old work employee ID number and was unpleasantly surprised to see the first result was a link to a spreadsheet with a name suggesting it contained details of the company's Singaporean payroll.
Relief came in the form of a 404 error after his first click. But as an IT type, Jeff was curious to learn if Google had cached the file.
Unfortunately, his hunch was correct: the spreadsheet opened and divulged over 160 names of the company's employees, the salaries they were paid, their home addresses and bank account details. Even marital status was listed.
Worse still, the spreadsheet suggested staff were being paid vastly different rates depending on gender and origin: not only were female employees' salaries low, expat staff were being paid vastly more than their Singaporean colleagues. In these diversity-aware times, the spreadsheet was a corporate reputation nightmare waiting to happen.
Jeff wondered if only his ID number had this problem, so tried other employees' ID numbers. All quickly brought up the same cached spreadsheet.
At which point Jeff complained to his former employer and contacted The Register.
D'oh, facepalm and WTF?
It was simple to verify his claims about how to find the leaked data. Jeff's former employer quickly told us it was aware of the breach and had notified its staff and offered them help. The multinational company declined to name the source of the breach, told us staff were confident the breach wasn't its fault and hinted that a third party was to blame.
Which wasn't hard to conclude, because the URL Google produced included the domain name of a Singaporean service provider. That URL included directory names that suggested a test and development server had been exposed to the internet.
Such a mistake could come as the result of a simple fat-fingered fumble, but the Singaporean service provider told us the cause was a ransomware infection that reset the server's security configuration. During the effort to repair the server, staff realised it was now in an insecure state, fixed that and tried to ensure the data was not accessible from the public Web.
You're the IT worker in charge of securing the cloud for your company. Welcome to HellREAD MORE
The service provider told us they also contacted Google, asking it to flush its cache so leaked data would not remain visible to the world. By January 9th, 2018, the service provider's IT staff were satisfied that their security had been restored and the personal data was no longer available.
Sadly, they were wrong. Jeff contacted The Register in the week of February 5th and we were able to view the personal data not long afterwards.
We therefore asked Google if it offers service levels for requests to flush its cache. The company told us it wouldn't comment on an individual case and directed us to its instructions on how to "remove outdated content" and pointed out that document explains that to remove personal information there's a Legal Removal Requests facility. Neither really explains how it would respond to a request to remove data from its cache.
It was El Reg wot won it?
Not long after The Register started asking questions of the Singaporean service provider, the cached data disappeared, leaving us more than a little suspicious about the service provider's claim to have asked Google for a cache flush in early January.
At the conclusion of our inquiries we can say with certainty that the Singaporean service provider should have had ransomware-proof defences and that the multinational technology company should have done better due diligence of the companies it permits to access its payroll data. Security best practice has long recognised that you are only as secure as your partners permit you to be: one weak link in the chain can be enough to break you.
But we're left uncertain if consideration of that chain also needs to factor in the quixotic nature of a web titan and its well-meaning but clearly dangerous cache. ®
*Not his real name. Nor have we used the names of the companies involved, or revealed the exact nature of how personal data was accessed, as we are not entirely satisfied the breach has been closed and the people whose personal information was leaked do not deserve increased risk.