Feeds

Running queries on the HMRC database fiasco

Dis-information systems management

High performance access to file storage

Why wasn't the sensitive, non-required, data removed?

According to the Telegraph, the National Audit Office (NAO) asked for the names, National Insurance numbers, and child benefit numbers of every child so that it could select 100 cases at random for its annual audit of Revenue and Customs. The NAO asked for bank and other details to be removed, but an HMRC official replied that, to keep costs down, the HMRC could only provide all of the details on the database.

Now, let's go back to Database 101 for a moment (not a theoretical Database 101; I actually designed and teach on the database course at Dundee University). The first example in the first lecture on querying shows the students how to subset the data by column and then by row. Database engines are built, from the ground up, to perform sub-setting. It doesn't get any easier than this. So, where's the problem?

But, let's assume the worst: that the HMRC uses a very complex, unwieldy engine that cannot subset data easily. Well, no matter how complex the original database was, it is reasonable to assume that it was reduced to a simple format for the CDs. In which case, importing it into an engine that can subset easily is trivial.

So, unless there are some very odd circumstances to which we are not privy, I find it impossible to believe that removing the bank and other details would have involved significant cost.

Go on, Mark, stick your neck out. How much?

Well, the Telegraph has an estimate:

However The Telegraph has established that a typical clean-up operation would cost around £5,000 and take a software engineer less than a week. A spokesman for HMRC said that the £5,000 cost of removing the information "was not a figure we recognise" and declined to discuss the cost because the matter is the subject of a review.

I don't recognise the figure either, but that's because I think the Telegraph is being far, far too generous to the HMRC. Assuming 25 million CSV records, I would estimate half a day's work to subset by column. If I was familiar with the data structure and had done the job before, maybe an hour. Any competent DBA/DBA could do it in the same time. Now DBAs are expensive, but not £10,000 per day. I'd do it for £500.

A spokesperson for the HMRC said: "We don't have infinite resources, we have to use our resources rationally."

One has to wonder about the definition of the word "rationally" here.

High performance access to file storage

More from The Register

next story
Obama allows NSA to exploit 0-days: report
If the spooks say they need it, they get it
Parent gabfest Mumsnet hit by SSL bug: My heart bleeds, grins hacker
Natter-board tells middle-class Britain to purée its passwords
Web data BLEEDOUT: Users to feel the pain as Heartbleed bug revealed
Vendors and ISPs have work to do updating firmware - if it's possible to fix this
OpenSSL Heartbleed: Bloody nose for open-source bleeding hearts
Bloke behind the cockup says not enough people are helping crucial crypto project
One year on: diplomatic fail as Chinese APT gangs get back to work
Mandiant says past 12 months shows Beijing won't call off its hackers
Call of Duty 'fragged using OpenSSL's Heartbleed exploit'
So it begins ... or maybe not, says one analyst
German space centre endures cyber attack
Chinese code retrieved but NSA hack not ruled out
Experian subsidiary faces MEGA-PROBE for 'selling consumer data to fraudster'
US attorneys general roll up sleeves, snap on gloves
prev story

Whitepapers

Securing web applications made simple and scalable
In this whitepaper learn how automated security testing can provide a simple and scalable way to protect your web applications.
Five 3D headsets to be won!
We were so impressed by the Durovis Dive headset we’ve asked the company to give some away to Reg readers.
HP ArcSight ESM solution helps Finansbank
Based on their experience using HP ArcSight Enterprise Security Manager for IT security operations, Finansbank moved to HP ArcSight ESM for fraud management.
The benefits of software based PBX
Why you should break free from your proprietary PBX and how to leverage your existing server hardware.
Mobile application security study
Download this report to see the alarming realities regarding the sheer number of applications vulnerable to attack, as well as the most common and easily addressable vulnerability errors.