Structured data is boring and useless

And unstructructured data? Hmmm, sexy

By David Norfolk, 23rd June 2006

Editor's blog We all know that structured data is boring and useless; while unstructured data is sexy and chock full of value. Well, only up to a point, Lord Copper.

Genuinely unstructured data can be a real nuisance - imagine extracting the return address from an unstructured letter, without letterhead and any of the formatting usually applied to letters.

A letter may be thought of as unstructured data, but most business letters are, in fact, highly-structured. This issue exercises Duncan Pauly, founder and chief technology officer of Coppereye, which sells indexing technology that exploits data structures, even when the data is in an unstructured container.

Duncan comes from the Oracle database world and has a deep understanding of data analysis and structure - and he finds that a fundamental knowledge of database and data structures is becoming increasingly rare.

I have to agree. At an IDC conference on Business Performance Management and Business Intelligence yesterday, I heard a speaker use Google Earth as an example of the use of unstructured data: "You just type in your postcode..."

Hold on a second. The data is geocoded by postcode of all things and identified by latitude and longitude (and, presumably, there's catalogued metadata associated with the photographs) - how on Earth [sorry] can it be called unstructured?

And, no, Google has not rendered data analysis and so on obsolete - my wife, an information scientist, never tires of pointing out that an unstructured search engine should be the last resort of the researcher, not the first. If you understand the structure and semantics of your data, there are usually more effective ways of retrieving the data you need than such a search engine can provide.

So, Duncan, what is the issue that's concerning you, exactly?

The labels "structured data" and "unstructured data" are often used ambiguously by different interest groups; and often used lazily to cover multiple distinct aspects of the issue. In reality, there are at least three orthogonal aspects to structure:

The structure of the data itself.
The structure of the container that hosts the data.
The structure of the access method used to access the data.

These three dimensions are largely independent and one does not need to imply another. For example, it is absolutely feasible and reasonable to store unstructured data in a structured database container and access it by unstructured search mechanisms.

What does that actually mean for the user of the data?

Unstructured data I take to be a sequence of tokens that require human interpretation or heuristics to derive semantic content. Structured data, however, has associated or implied metadata to define or label its semantics. Typically, unstructured data tends to be textual, where structured data tends contain vectors of quantitative scalar or referential identities.

OK, so (as I take it) truly unstructured data resists automation, a human has to be involved to provide the semantics necessary. The processing of structured data, even in unstructured containers, can be automated routinely - which is the business of developers. Perhaps we should throw this subject open to the readers of this blog; who are, presumably, mostly developers - what aspects of this issue do you think they should consider?

Well, perhaps:

Do storage vendors, database vendors and search vendors misuse the "structured/unstructured" labels, and why?
As search-engine user interfaces become more widely used and data-stores federate databases proper and document-style datastores, do we need to think more clearly about the distinction between unstructured and structured data?
What are the implications of separating data-models from container models from access models? An RDBMS with an XML engine and a BLOB datatype, for example, stores a lot more than just relational data.

All right, all you practising developers. It's over to you...

Regcast training : Hyper-V 3.0, VM high availability and disaster recovery

COMMENTS

House Rules Send Corrections

All Reader Comments (5)

Latest Comments

Posted Monday 26th June 2006 09:54 GMT

Duncan Pauly

One Man's Structure...

I agree David. The methods exist for dealing with data structure, but the “freedom” associated with unstructured data has become seductive to many who are happy to perceive structure merely as XML semantic tagging; and meanwhile nomenclature continues to confuse with the term “structure” meaning different things to different interest groups.

Posted Saturday 24th June 2006 22:07 GMT

David Norfolk

OODBMS

Well, I'm sorry about posting this piece on Friday (even sorrier to be posting responses at midnight on Friday <grin>) but the OODBMS comments are interesting - and quite valid, although perhaps not all that can be said on the subject.

The trouble is, that many people don't think there's an issue here and that using an OODBMS (I've been told by people who've tried them) needs quite as much discipline as using an RDBMS properly.

More trivially, perhaps, things like Ontology make peoples' heads hurt. I was at an IDC Business Performance Management and BI conference this week where someone bought up ontologies. Everyone looked blank - analysts, speaker panal and delegates.

I think that there are technologies that can cope with managing structured and unstructured data and everything in between. I don't think that is the issue - I think Kingsley's remark - we are supposed to be in the throws of the "Information Age", but for some reason this appears to have no correlation with data and "data access" in the minds of many - is about right. And it will lead to problems.

A lot of "information processing" (or integration) efforts still fail because of data quality issues are underestimated and the politics of data semantics and ownership are overlooked. I think that the confusion between "structured" and "unstructured" data that Duncan

identifies is just one symptom of a general malaise involving a lack of understanding of data issues and their importance.

Posted Friday 23rd June 2006 20:29 GMT

Kingsley Idehen

Unstructured vs Structured Data terminology does matter

Your article addresses some very important issues.

1. Data understanding and appreciation is dwindling at a time when the reverse should be happening. We are supposed to be in the throws of the "Information Age", but for some reason this appears to have no correlation with data and "data access" in the minds of many -- as reflected in the broad contradictory positions taken re. unstructured data vs structured data.

2. The difference between "Structured Containers" and "Structured Data" are clearly misunderstood by most.

"Structured Containers" (most DBMS products) have been limited by proprietary data access APIs and underlying data model specificity to date, when looking at the needs of the loosely coupled "Open-World" web of data called the World Wide Web. Naturally, in the "Open-world" model of the Web this is unacceptable. But things are changing fast, and the concept of multi-model DBMS products is beginning to crystalize.

For instance, the Semantic Web (a vision that most don't understand due to the lack of coherent annecdotal material for the less technical) will ultimately manifest itself as a collection of loosely coupled databases that possess object-relational DBMS functionality.

ORDBMS engines can extend data model support capabilities via object-relational functionality as exemplified by OpenLink Virtuoso which enables SQL, XML, and RDF management from one place (Unified Storage) with support for SPARQL, GData, OpenSearch, and other emerging Query Protocols).

Please note that I am not implying that ORDBMS knowldge is required to make the Semantic Web more coherent than it is to date. I am designating ORDBMS engines as the DBMS engine form best suited for building the applications layer that is ultimately exposed as an endpoint in the eventual "web of databases". Personally, I prefer to call these endpoints "Data Spaces" since this is what ultimately fuses the Web 2.0 and the Semantic Web paradigms (that are currently perceived as mutually exclusive).

For information on Virtuoso you can take a look at the Open Source project at: http://virtuoso.openlinksw.com/wiki/main/

Nice article!

Regards,

Kingsley Idehen

President & CEO

OpenLink Software Web: http://www.openlinksw.com

Personal Blog: http://www.openlinksw.com/blog/~kidehen