Metadata manipulation by Alation seeks needles in data haystack

A disparate bunch of Oracle, Apple and Microsoft guys have started up Alation, which makes metadata manipulations to present all of an enterprise's data to analysts and others so they can find stuff out.

Alation, founded in 2012, has emerged from stealth having just gained $9 million of A-round funding, from Andreesen Horowitz and other VCs.

The co-founders are Satyen Sangani, CEO and ex-Oracle warehousing, Venky Ganti, CTO and ex-Microsoft, and Aaron Kalb who is head of product and ex-Apple with Siri involvement.

Their product technology, already in development with customers such as eBay, provides data search and discovery, collaborative analytics, data optimisation and governance. The firm's announcement release is full of customer love stuff so we thought we'd ignore that and look at the data search and discovery to find out what's going on since, without that, the other things wouldn't work.

Alation starts from the fact that there are multiple different structured and semi-structured data silos, "spread across disparate databases, wikis, data dictionaries, metadata tools, glossaries, and systems like Hadoop," each built with their own declared and default and hidden assumptions.

Assume this data forms a haystack, you want to find a needle, then Alation's software is the compass and needle finder. It will locate data you ask for using English terms (shipment date and not "shp_dte", China and not "cntry_cd=156") "whether it lives in Hadoop, MySQL, Oracle, Vertica, Teradata or any of the many databases and file systems that we support." And whether it is one place or many and it will "help you understand the results with technical metadata, usage metadata, and documentation."

Okay, so straight away it must have a means of locating, querying and analysing metadata from "the many databases and file systems that we support." The metadata examined includes the time since last access, by whom, and data source and type, like transactions.

"Machine learning algorithms leverage statistics on usage and visits to rank every data object in your company," teh firm states. Humans can annotate and rate the value if data.

The software makes, we think, rankings of data accessors and creators so that experts can be identified. An Alation Converse function lets you ask such an expert a question and the question and resulting conversation effectively becomes more metadata about the subject data.

It creates a data glossary, a dictionary, and the fields in each can be edited and customised to extend the overall metadata model.

In the collaborative analytics area Alation's software "automatically builds a Catalog of useful data documentation, covering all of your data sources. Access relevant information (including experts, lineage, keys and indexes, relevant queries) and documentation on all tables, across your organisation’s data sources, in seconds."

It can help you write SQL by providing examples and suggestions. You an use "Query Search over all the SQL ever written at your company," to locate SQL.

Think of Alation as a kind of super SQL-ish workhorse in its alpha-test/development phase; SQL Siri without the voice input so to speak. It doesn't merge storage silos or virtualise them into a single logical entity, but it does turn them into a single searchable logical metadata entity; a metadata lake. ®