Auditor rains on Bureau of Met's data warehousing parade

Drought databases needs a scorched-earth approach if it is to be fixed

An ambitious project to create a single national water database is going to need a complete reboot, with the Australian National Audit Office saying complexity, non-standard approaches and supplier capture caused a multi-million-dollar blowout in the system.

The Bureau of Meteorology was given the task of creating the database after Australia's last long drought, and a doozy of a database it is, with a daily incoming load of 10,000 data files. According to the ANAO, AWRIS 1 (Australian Water Resources Information Systems) “has been managing more than 21 million water data files containing more than four billion time‐series observations since the Water Regulations came into effect on 30 June 2008.”

The scale of the project, ANAO explains, comes from the ambitious scope of the brief given to the BoM: to concentrate state-based water observations (from streamflow monitoring gauges, which generated the 10,000 daily data files) into a single database, which stakeholders were supposed to be able to search for water availability and quality information.

According to the ANAO, both architectural and governance problems plagued the implementation, and as a result, the expected $AU2.5 million annual price tag went by the board: instead, the data warehouse alone cost around $AU12.5 million and the total project has cost $AU38.5 million since the contract was awarded to SMS Management and Technology in 2009.

“The development of AWRIS has been problematic with unclear business and system requirements, inadequate technical solutions, shortcomings in governance arrangements, changes in design and approach and unanticipated costs and delays that have limited the functionality of the system,” the report states.

It continues: “The Bureau has advised that because of the complex design of the system, if any significant changes were required, it would not always be possible to modify the system to ingest or store data without the original consultant.”

Data duplication, the scourge of large datasets, has also proven problematic. Data can't be easily deleted from the system once it's been validated by a data provider, the report states. “This has led to significant duplication of data, with the Bureau estimating that some line items had 98 per cent redundancy prior to a recent time‐consuming de‐duplication activity.”

According to ITNews, the original system was built on Sun servers running RedHat Linux, with MySQL as the database, Talend's Open Studio for data integration, and Java as the front end.

AWRIS 1 is now to be replaced, with the ANAO noting that the Bureau has budgeted $AU10 million on its replacement. ANAO says it's to be based on a proof-of-concept, based on both open source and off-the-shelf commercial components, which confirmed the design of the data warehouse and datamart schemas. The Bureau told ANAO it has verified that AWRIS 1 data can be migrated into the proposed AWRIS 2 environment.

The new system will be mostly handled in-house, the report says, with contractors providing supplementary skills. ®

Sponsored: Today’s most dangerous security threats