Feeds

Auditor rains on Bureau of Met's data warehousing parade

Drought databases needs a scorched-earth approach if it is to be fixed

Mobile application security vulnerability report

An ambitious project to create a single national water database is going to need a complete reboot, with the Australian National Audit Office saying complexity, non-standard approaches and supplier capture caused a multi-million-dollar blowout in the system.

The Bureau of Meteorology was given the task of creating the database after Australia's last long drought, and a doozy of a database it is, with a daily incoming load of 10,000 data files. According to the ANAO, AWRIS 1 (Australian Water Resources Information Systems) “has been managing more than 21 million water data files containing more than four billion time‐series observations since the Water Regulations came into effect on 30 June 2008.”

The scale of the project, ANAO explains, comes from the ambitious scope of the brief given to the BoM: to concentrate state-based water observations (from streamflow monitoring gauges, which generated the 10,000 daily data files) into a single database, which stakeholders were supposed to be able to search for water availability and quality information.

According to the ANAO, both architectural and governance problems plagued the implementation, and as a result, the expected $AU2.5 million annual price tag went by the board: instead, the data warehouse alone cost around $AU12.5 million and the total project has cost $AU38.5 million since the contract was awarded to SMS Management and Technology in 2009.

“The development of AWRIS has been problematic with unclear business and system requirements, inadequate technical solutions, shortcomings in governance arrangements, changes in design and approach and unanticipated costs and delays that have limited the functionality of the system,” the report states.

It continues: “The Bureau has advised that because of the complex design of the system, if any significant changes were required, it would not always be possible to modify the system to ingest or store data without the original consultant.”

Data duplication, the scourge of large datasets, has also proven problematic. Data can't be easily deleted from the system once it's been validated by a data provider, the report states. “This has led to significant duplication of data, with the Bureau estimating that some line items had 98 per cent redundancy prior to a recent time‐consuming de‐duplication activity.”

According to ITNews, the original system was built on Sun servers running RedHat Linux, with MySQL as the database, Talend's Open Studio for data integration, and Java as the front end.

AWRIS 1 is now to be replaced, with the ANAO noting that the Bureau has budgeted $AU10 million on its replacement. ANAO says it's to be based on a proof-of-concept, based on both open source and off-the-shelf commercial components, which confirmed the design of the data warehouse and datamart schemas. The Bureau told ANAO it has verified that AWRIS 1 data can be migrated into the proposed AWRIS 2 environment.

The new system will be mostly handled in-house, the report says, with contractors providing supplementary skills. ®

Bridging the IT gap between rising business demands and ageing tools

More from The Register

next story
Arrr: Freetard-bothering Digital Economy Act tied up, thrown in the hold
Ministry of Fun confirms: Yes, we're busy doing nothing
ONE EMAIL costs mining company $300 MEEELION
Environmental activist walks free after hoax sent share price over a cliff
'Blow it up': Plods pop round for chat with Commonwealth Games tweeter
You'd better not be talking about the council's housing plans
Help yourself to anyone's photos FOR FREE, suggests UK.gov
Copyright law reforms will keep m'learned friends busy
Apple smacked with privacy sueball over Location Services
Class action launched on behalf of 100 million iPhone owners
UK government officially adopts Open Document Format
Microsoft insurgency fails, earns snarky remark from UK digital services head
You! Pirate! Stop pirating, or we shall admonish you politely. Repeatedly, if necessary
And we shall go about telling people you smell. No, not really
prev story

Whitepapers

Implementing global e-invoicing with guaranteed legal certainty
Explaining the role local tax compliance plays in successful supply chain management and e-business and how leading global brands are addressing this.
Consolidation: The Foundation for IT Business Transformation
In this whitepaper learn how effective consolidation of IT and business resources can enable multiple, meaningful business benefits.
Application security programs and practises
Follow a few strategies and your organization can gain the full benefits of open source and the cloud without compromising the security of your applications.
How modern custom applications can spur business growth
Learn how to create, deploy and manage custom applications without consuming or expanding the need for scarce, expensive IT resources.
Securing Web Applications Made Simple and Scalable
Learn how automated security testing can provide a simple and scalable way to protect your web applications.