Feeds

Auditor rains on Bureau of Met's data warehousing parade

Drought databases needs a scorched-earth approach if it is to be fixed

Secure remote control for conventional and virtual desktops

An ambitious project to create a single national water database is going to need a complete reboot, with the Australian National Audit Office saying complexity, non-standard approaches and supplier capture caused a multi-million-dollar blowout in the system.

The Bureau of Meteorology was given the task of creating the database after Australia's last long drought, and a doozy of a database it is, with a daily incoming load of 10,000 data files. According to the ANAO, AWRIS 1 (Australian Water Resources Information Systems) “has been managing more than 21 million water data files containing more than four billion time‐series observations since the Water Regulations came into effect on 30 June 2008.”

The scale of the project, ANAO explains, comes from the ambitious scope of the brief given to the BoM: to concentrate state-based water observations (from streamflow monitoring gauges, which generated the 10,000 daily data files) into a single database, which stakeholders were supposed to be able to search for water availability and quality information.

According to the ANAO, both architectural and governance problems plagued the implementation, and as a result, the expected $AU2.5 million annual price tag went by the board: instead, the data warehouse alone cost around $AU12.5 million and the total project has cost $AU38.5 million since the contract was awarded to SMS Management and Technology in 2009.

“The development of AWRIS has been problematic with unclear business and system requirements, inadequate technical solutions, shortcomings in governance arrangements, changes in design and approach and unanticipated costs and delays that have limited the functionality of the system,” the report states.

It continues: “The Bureau has advised that because of the complex design of the system, if any significant changes were required, it would not always be possible to modify the system to ingest or store data without the original consultant.”

Data duplication, the scourge of large datasets, has also proven problematic. Data can't be easily deleted from the system once it's been validated by a data provider, the report states. “This has led to significant duplication of data, with the Bureau estimating that some line items had 98 per cent redundancy prior to a recent time‐consuming de‐duplication activity.”

According to ITNews, the original system was built on Sun servers running RedHat Linux, with MySQL as the database, Talend's Open Studio for data integration, and Java as the front end.

AWRIS 1 is now to be replaced, with the ANAO noting that the Bureau has budgeted $AU10 million on its replacement. ANAO says it's to be based on a proof-of-concept, based on both open source and off-the-shelf commercial components, which confirmed the design of the data warehouse and datamart schemas. The Bureau told ANAO it has verified that AWRIS 1 data can be migrated into the proposed AWRIS 2 environment.

The new system will be mostly handled in-house, the report says, with contractors providing supplementary skills. ®

The essential guide to IT transformation

More from The Register

next story
Hello, police, El Reg here. Are we a bunch of terrorists now?
Do Brits risk arrest for watching beheading video nasty? We asked the fuzz
Snowden on NSA's MonsterMind TERROR: It may trigger cyberwar
Plus: Syria's internet going down? That was a US cock-up
UK government accused of hiding TRUTH about Universal Credit fiasco
'Reset rating keeps secrets on one-dole-to-rule-them-all plan', say MPs
Caught red-handed: UK cops, PCSOs, specials behaving badly… on social media
No Mr Fuzz, don't ask a crime victim to be your pal on Facebook
Felony charges? Harsh! Alleged Anon hackers plead guilty to misdemeanours
US judge questions harsh sentence sought by prosecutors
e-Borders fiasco: Brits stung for £224m after US IT giant sues UK govt
Defeat to Raytheon branded 'catastrophic result'
Yes, but what are your plans if a DRAGON attacks?
Local UK gov outs most ridiculous FoI requests...
EU justice chief blasts Google on 'right to be forgotten'
Don't pretend it's a freedom of speech issue – interim commish
Munich considers dumping Linux for ... GULP ... Windows!
Give a penguinista a hug, the Outlook's not good for open source's poster child
This'll end well: US govt says car-to-car jibber-jabber will SAVE lives
Department of Transportation starts cogs turning for another wireless comms standard
prev story

Whitepapers

Implementing global e-invoicing with guaranteed legal certainty
Explaining the role local tax compliance plays in successful supply chain management and e-business and how leading global brands are addressing this.
Top 10 endpoint backup mistakes
Avoid the ten endpoint backup mistakes to ensure that your critical corporate data is protected and end user productivity is improved.
Top 8 considerations to enable and simplify mobility
In this whitepaper learn how to successfully add mobile capabilities simply and cost effectively.
Rethinking backup and recovery in the modern data center
Combining intelligence, operational analytics, and automation to enable efficient, data-driven IT organizations using the HP ABR approach.
Reg Reader Research: SaaS based Email and Office Productivity Tools
Read this Reg reader report which provides advice and guidance for SMBs towards the use of SaaS based email and Office productivity tools.