SaaS data loss: The problem you didn’t know you had

The world's source for global temperature record admits it's lost or destroyed all the original data that would allow a third party to construct a global temperature record. The destruction (or loss) of the data comes at a convenient time for the Climatic Research Unit (CRU) in East Anglia - permitting it to snub FoIA requests to see the data.

The CRU has refused to release the raw weather station data and its processing methods for inspection - except to hand-picked academics - for several years. Instead, it releases a processed version, in gridded form. NASA maintains its own (GISSTEMP), but the CRU Global Climate Dataset, is the most cited surface temperature record by the UN IPCC. So any errors in CRU cascade around the world, and become part of "the science".

Professor Phil Jones, the activist-scientist who maintains the data set, has cited various reasons for refusing to release the raw data. Most famously, Jones told an Australian climate scientist in 2004:

Even if WMO agrees, I will still not pass on the data. We have 25 or so years invested in the work. Why should I make the data available to you, when your aim is to try and find something wrong with it.

In 2007, in response to Freedom of Information Act requests, CRU initially said it didn't have to fulfil the requests because "Information accessible to applicant via other means Some information is publicly available on external websites".

Now it's citing confidentiality agreements with Denmark, Spain, Bahrain and our own Mystic Met Office. Others may exist, CRU says in a statement, but it might have lost them because it moved offices. Or they were made verbally, and nobody at CRU wrote them down.

As for the raw station data,

"We are not in a position to supply data for a particular country not covered by the example agreements referred to earlier, as we have never had sufficient resources to keep track of the exact source of each individual monthly value. Since the 1980s, we have merged the data we have received into existing series or begun new ones, so it is impossible to say if all stations within a particular country or if all of an individual record should be freely available. Data storage availability in the 1980s meant that we were not able to keep the multiple sources for some sites, only the station series after adjustment for homogeneity issues. We, therefore, do not hold the original raw data but only the value-added (i.e. quality controlled and homogenized) data."

Canadian statistician and blogger Steve McIntyre, who has been asking for the data set for years, says he isn't impressed by the excuses. McIntyre obtained raw data when it was accidentally left on an FTP server last month. Since then, CRU has battened down the hatches, and purged its FTP directories lest any more raw data escapes and falls into the wrong hands.

McIntyre says he doesn't expect any significant surprises after analysing the raw data, but believes that reproducibility is a cornerstone of the scientific principle, and so raw data and methods should be disclosed. ®

Magic Quadrant for Enterprise Backup/Recovery