Real-time data warehousing

A change in the air?

HP ProLiant Gen8: Integrated lifecycle automation

Comment Data warehousing has historically been regarded as an environment which was specifically about analysing historic data, either to understand what has happened or, more recently, in order to try to predict what will happen: for example, to try to predict customers likely to churn.

However, there are also a number of environments in which you need to combine this historic information with real-time transactional data in order to make operational business decisions or to detect fraud or for anti-money laundering, and so on.

Traditionally (if we can use that term for something that has only come to the fore in the last few years), the way to implement this sort of capability has been through real-time data warehousing. This is what Teradata refers to as active data warehousing.

The way real-time data warehousing is typically implemented is that you use change data capture to grab relevant data after it is committed to the transactional database(s) and you then trickle feed that information to the warehouse via a message queue or similar mechanism. Then, in the warehouse, that new information is combined with historic data and, if relevant thresholds are crossed, alerts can be generated, rules triggered and action taken.

The question I want to pose is whether this is actually the best way to implement this sort of capability? It seems to me that using an event processing engine is a more appropriate technology to start from.

Event processing engines do not need you to commit data to a database before you start to action it; many of them can process deltas (changes to data) directly, so you don't need any change data capture; and you don't have any issues with getting data onto a message queue. The only potential problem is with the historic data that you may need from the data warehouse.

Most event processing engines have some sort of associated data store, though this is usually only for storing the events that have been captured. Some companies have specific capabilities for pattern recognition with these event stores. However, for real-time data warehousing you may well need to access historic information that would not form part of this database. So, vendors provide facilities to extract information from other sources (including data warehouses) where appropriate.

Now, consider the data needed from the warehouse in these sorts of circumstances. Are these large ad hoc or complex queries? No. In order to build the sort of workflow-style decisioning that is central to active and real-time data warehousing you have to know what you are looking for. In other words, queries are pre-built and the data sets you are going to be exploring are known.

This leads to two possible scenarios: in the first, the dataset you need to explore is relatively limited, in which case why not replicate a copy of the data into the event processing database and have the rules built on that platform. If you truly want real-time performance this will be much faster than the "traditional" approach to real-time data warehousing.

Secondly, there is the scenario where the dataset that needs to be explored is very large. You could sample it, of course, but if it is considered impractical to do this or to replicate the data to the event store, then I think that it is still (potentially) preferable to do the processing on the event engine and then to link that to the warehouse for query purposes: what you would need is a fast pipe between the event engine and the warehouse, and you want a warehouse that is optimised for these specific queries and, I would argue, this will be best served by a data warehouse appliance.

If this is a valid proposition then there are two things that are likely to happen in the market. First, we may see event processing and appliance suppliers forming partnerships (and developing fast pipes between them) and, secondly, we will see Teradata coming under even more pressure in the market, as active data warehousing is currently one of its major differentiators.

Copyright © 2006, IT-Analysis.com

Eight steps to building an HP BladeSystem

More from The Register

next story
THUD! WD plonks down SIX TERABYTE 'consumer NAS' fatboy
Now that's a LOT of porn or pirated movies. Or, you know, other consumer stuff
EU's top data cops to meet Google, Microsoft et al over 'right to be forgotten'
Plan to hammer out 'coherent' guidelines. Good luck chaps!
US judge: YES, cops or feds so can slurp an ENTIRE Gmail account
Crooks don't have folders labelled 'drug records', opines NY beak
Manic malware Mayhem spreads through Linux, FreeBSD web servers
And how Google could cripple infection rate in a second
FLAPE – the next BIG THING in storage
Find cold data with flash, transmit it from tape
Seagate chances ARM with NAS boxes for the SOHO crowd
There's an Atom-powered offering, too
prev story


Seven Steps to Software Security
Seven practical steps you can begin to take today to secure your applications and prevent the damages a successful cyber-attack can cause.
Consolidation: The Foundation for IT Business Transformation
In this whitepaper learn how effective consolidation of IT and business resources can enable multiple, meaningful business benefits.
Designing a Defense for Mobile Applications
Learn about the various considerations for defending mobile applications - from the application architecture itself to the myriad testing technologies.
Build a business case: developing custom apps
Learn how to maximize the value of custom applications by accelerating and simplifying their development.
Consolidation: the foundation for IT and business transformation
In this whitepaper learn how effective consolidation of IT and business resources can enable multiple, meaningful business benefits.