Feeds

Real-time data warehousing

A change in the air?

3 Big data security analytics techniques

Comment Data warehousing has historically been regarded as an environment which was specifically about analysing historic data, either to understand what has happened or, more recently, in order to try to predict what will happen: for example, to try to predict customers likely to churn.

However, there are also a number of environments in which you need to combine this historic information with real-time transactional data in order to make operational business decisions or to detect fraud or for anti-money laundering, and so on.

Traditionally (if we can use that term for something that has only come to the fore in the last few years), the way to implement this sort of capability has been through real-time data warehousing. This is what Teradata refers to as active data warehousing.

The way real-time data warehousing is typically implemented is that you use change data capture to grab relevant data after it is committed to the transactional database(s) and you then trickle feed that information to the warehouse via a message queue or similar mechanism. Then, in the warehouse, that new information is combined with historic data and, if relevant thresholds are crossed, alerts can be generated, rules triggered and action taken.

The question I want to pose is whether this is actually the best way to implement this sort of capability? It seems to me that using an event processing engine is a more appropriate technology to start from.

Event processing engines do not need you to commit data to a database before you start to action it; many of them can process deltas (changes to data) directly, so you don't need any change data capture; and you don't have any issues with getting data onto a message queue. The only potential problem is with the historic data that you may need from the data warehouse.

Most event processing engines have some sort of associated data store, though this is usually only for storing the events that have been captured. Some companies have specific capabilities for pattern recognition with these event stores. However, for real-time data warehousing you may well need to access historic information that would not form part of this database. So, vendors provide facilities to extract information from other sources (including data warehouses) where appropriate.

Now, consider the data needed from the warehouse in these sorts of circumstances. Are these large ad hoc or complex queries? No. In order to build the sort of workflow-style decisioning that is central to active and real-time data warehousing you have to know what you are looking for. In other words, queries are pre-built and the data sets you are going to be exploring are known.

This leads to two possible scenarios: in the first, the dataset you need to explore is relatively limited, in which case why not replicate a copy of the data into the event processing database and have the rules built on that platform. If you truly want real-time performance this will be much faster than the "traditional" approach to real-time data warehousing.

Secondly, there is the scenario where the dataset that needs to be explored is very large. You could sample it, of course, but if it is considered impractical to do this or to replicate the data to the event store, then I think that it is still (potentially) preferable to do the processing on the event engine and then to link that to the warehouse for query purposes: what you would need is a fast pipe between the event engine and the warehouse, and you want a warehouse that is optimised for these specific queries and, I would argue, this will be best served by a data warehouse appliance.

If this is a valid proposition then there are two things that are likely to happen in the market. First, we may see event processing and appliance suppliers forming partnerships (and developing fast pipes between them) and, secondly, we will see Teradata coming under even more pressure in the market, as active data warehousing is currently one of its major differentiators.

Copyright © 2006, IT-Analysis.com

SANS - Survey on application security programs

More from The Register

next story
This time it's 'Personal': new Office 365 sub covers just two devices
Redmond also brings Office into Google's back yard
Kingston DataTraveler MicroDuo: Turn your phone into a 72GB beast
USB-usiness in the front, micro-USB party in the back
Dropbox defends fantastically badly timed Condoleezza Rice appointment
'Nothing is going to change with Dr. Rice's appointment,' file sharer promises
BOFH: Oh DO tell us what you think. *CLICK*
$%%&amp Oh dear, we've been cut *CLICK* Well hello *CLICK* You're breaking up...
Just what could be inside Dropbox's new 'Home For Life'?
Biz apps, messaging, photos, email, more storage – sorry, did you think there would be cake?
IT bods: How long does it take YOU to train up on new tech?
I'll leave my arrays to do the hard work, if you don't mind
Amazon reveals its Google-killing 'R3' server instances
A mega-memory instance that never forgets
Cisco reps flog Whiptail's Invicta arrays against EMC and Pure
Storage reseller report reveals who's selling what
prev story

Whitepapers

Designing a defence for mobile apps
In this whitepaper learn the various considerations for defending mobile applications; from the mobile application architecture itself to the myriad testing technologies needed to properly assess mobile applications risk.
3 Big data security analytics techniques
Applying these Big Data security analytics techniques can help you make your business safer by detecting attacks early, before significant damage is done.
Five 3D headsets to be won!
We were so impressed by the Durovis Dive headset we’ve asked the company to give some away to Reg readers.
The benefits of software based PBX
Why you should break free from your proprietary PBX and how to leverage your existing server hardware.
Securing web applications made simple and scalable
In this whitepaper learn how automated security testing can provide a simple and scalable way to protect your web applications.