Feeds

Auditor rains on Bureau of Met's data warehousing parade

Drought databases needs a scorched-earth approach if it is to be fixed

High performance access to file storage

An ambitious project to create a single national water database is going to need a complete reboot, with the Australian National Audit Office saying complexity, non-standard approaches and supplier capture caused a multi-million-dollar blowout in the system.

The Bureau of Meteorology was given the task of creating the database after Australia's last long drought, and a doozy of a database it is, with a daily incoming load of 10,000 data files. According to the ANAO, AWRIS 1 (Australian Water Resources Information Systems) “has been managing more than 21 million water data files containing more than four billion time‐series observations since the Water Regulations came into effect on 30 June 2008.”

The scale of the project, ANAO explains, comes from the ambitious scope of the brief given to the BoM: to concentrate state-based water observations (from streamflow monitoring gauges, which generated the 10,000 daily data files) into a single database, which stakeholders were supposed to be able to search for water availability and quality information.

According to the ANAO, both architectural and governance problems plagued the implementation, and as a result, the expected $AU2.5 million annual price tag went by the board: instead, the data warehouse alone cost around $AU12.5 million and the total project has cost $AU38.5 million since the contract was awarded to SMS Management and Technology in 2009.

“The development of AWRIS has been problematic with unclear business and system requirements, inadequate technical solutions, shortcomings in governance arrangements, changes in design and approach and unanticipated costs and delays that have limited the functionality of the system,” the report states.

It continues: “The Bureau has advised that because of the complex design of the system, if any significant changes were required, it would not always be possible to modify the system to ingest or store data without the original consultant.”

Data duplication, the scourge of large datasets, has also proven problematic. Data can't be easily deleted from the system once it's been validated by a data provider, the report states. “This has led to significant duplication of data, with the Bureau estimating that some line items had 98 per cent redundancy prior to a recent time‐consuming de‐duplication activity.”

According to ITNews, the original system was built on Sun servers running RedHat Linux, with MySQL as the database, Talend's Open Studio for data integration, and Java as the front end.

AWRIS 1 is now to be replaced, with the ANAO noting that the Bureau has budgeted $AU10 million on its replacement. ANAO says it's to be based on a proof-of-concept, based on both open source and off-the-shelf commercial components, which confirmed the design of the data warehouse and datamart schemas. The Bureau told ANAO it has verified that AWRIS 1 data can be migrated into the proposed AWRIS 2 environment.

The new system will be mostly handled in-house, the report says, with contractors providing supplementary skills. ®

High performance access to file storage

More from The Register

next story
Android engineer: We DIDN'T copy Apple OR follow Samsung's orders
Veep testifies for Samsung during Apple patent trial
MtGox chief Karpelès refuses to come to US for g-men's grilling
Bitcoin baron says he needs another lawyer for FinCEN chat
Did a date calculation bug just cost hard-up Co-op Bank £110m?
And just when Brit banking org needs £400m to stay afloat
One year on: diplomatic fail as Chinese APT gangs get back to work
Mandiant says past 12 months shows Beijing won't call off its hackers
Don't let no-hire pact suit witnesses call Steve Jobs a bullyboy, plead Apple and Google
'Irrelevant' character evidence should be excluded – lawyers
EFF: Feds plan to put 52 MILLION FACES into recognition database
System would identify faces as part of biometrics collection
Big Content goes after Kim Dotcom
Six studios sling sueballs at dead download destination
Ex-Tony Blair adviser is new top boss at UK spy-hive GCHQ
Robert Hannigan to replace Sir Iain Lobban in the autumn
Alphadex fires back at British Gas with overcharging allegation
Brit colo outfit says it paid for 347KVA, has been charged for 1940KVA
Jack the RIPA: Blighty cops ignore law, retain innocents' comms data
Prime minister: Nothing to see here, go about your business
prev story

Whitepapers

Securing web applications made simple and scalable
In this whitepaper learn how automated security testing can provide a simple and scalable way to protect your web applications.
Five 3D headsets to be won!
We were so impressed by the Durovis Dive headset we’ve asked the company to give some away to Reg readers.
HP ArcSight ESM solution helps Finansbank
Based on their experience using HP ArcSight Enterprise Security Manager for IT security operations, Finansbank moved to HP ArcSight ESM for fraud management.
The benefits of software based PBX
Why you should break free from your proprietary PBX and how to leverage your existing server hardware.
Mobile application security study
Download this report to see the alarming realities regarding the sheer number of applications vulnerable to attack, as well as the most common and easily addressable vulnerability errors.