Feeds

Auditor rains on Bureau of Met's data warehousing parade

Drought databases needs a scorched-earth approach if it is to be fixed

Choosing a cloud hosting partner with confidence

An ambitious project to create a single national water database is going to need a complete reboot, with the Australian National Audit Office saying complexity, non-standard approaches and supplier capture caused a multi-million-dollar blowout in the system.

The Bureau of Meteorology was given the task of creating the database after Australia's last long drought, and a doozy of a database it is, with a daily incoming load of 10,000 data files. According to the ANAO, AWRIS 1 (Australian Water Resources Information Systems) “has been managing more than 21 million water data files containing more than four billion time‐series observations since the Water Regulations came into effect on 30 June 2008.”

The scale of the project, ANAO explains, comes from the ambitious scope of the brief given to the BoM: to concentrate state-based water observations (from streamflow monitoring gauges, which generated the 10,000 daily data files) into a single database, which stakeholders were supposed to be able to search for water availability and quality information.

According to the ANAO, both architectural and governance problems plagued the implementation, and as a result, the expected $AU2.5 million annual price tag went by the board: instead, the data warehouse alone cost around $AU12.5 million and the total project has cost $AU38.5 million since the contract was awarded to SMS Management and Technology in 2009.

“The development of AWRIS has been problematic with unclear business and system requirements, inadequate technical solutions, shortcomings in governance arrangements, changes in design and approach and unanticipated costs and delays that have limited the functionality of the system,” the report states.

It continues: “The Bureau has advised that because of the complex design of the system, if any significant changes were required, it would not always be possible to modify the system to ingest or store data without the original consultant.”

Data duplication, the scourge of large datasets, has also proven problematic. Data can't be easily deleted from the system once it's been validated by a data provider, the report states. “This has led to significant duplication of data, with the Bureau estimating that some line items had 98 per cent redundancy prior to a recent time‐consuming de‐duplication activity.”

According to ITNews, the original system was built on Sun servers running RedHat Linux, with MySQL as the database, Talend's Open Studio for data integration, and Java as the front end.

AWRIS 1 is now to be replaced, with the ANAO noting that the Bureau has budgeted $AU10 million on its replacement. ANAO says it's to be based on a proof-of-concept, based on both open source and off-the-shelf commercial components, which confirmed the design of the data warehouse and datamart schemas. The Bureau told ANAO it has verified that AWRIS 1 data can be migrated into the proposed AWRIS 2 environment.

The new system will be mostly handled in-house, the report says, with contractors providing supplementary skills. ®

Top 5 reasons to deploy VMware with Tegile

More from The Register

next story
Facebook pays INFINITELY MORE UK corp tax than in 2012
Thanks for the £3k, Zuck. Doh! you're IN CREDIT. Guess not
DOUBLE BONK: Testy fanbois catch Apple Pay picking pockets
Users wail as tapcash transactions are duplicated
Happiness economics is bollocks. Oh, UK.gov just adopted it? Er ...
Opportunity doesn't knock; it costs us instead
Google Glassholes are UNDATEABLE – HP exec
You need an emotional connection, says touchy-feely MD... We can do that
YARR! Pirates walk the plank: DMCA magnets sink in Google results
Spaffing copyrighted stuff over the web? No search ranking for you
prev story

Whitepapers

Why cloud backup?
Combining the latest advancements in disk-based backup with secure, integrated, cloud technologies offer organizations fast and assured recovery of their critical enterprise data.
A strategic approach to identity relationship management
ForgeRock commissioned Forrester to evaluate companies’ IAM practices and requirements when it comes to customer-facing scenarios versus employee-facing ones.
Security for virtualized datacentres
Legacy security solutions are inefficient due to the architectural differences between physical and virtual environments.
Reg Reader Research: SaaS based Email and Office Productivity Tools
Read this Reg reader report which provides advice and guidance for SMBs towards the use of SaaS based email and Office productivity tools.
New hybrid storage solutions
Tackling data challenges through emerging hybrid storage solutions that enable optimum database performance whilst managing costs and increasingly large data stores.