All aboard the Enterprise Service Bus
Event processing, anyone?
Analysis Sonic Software has a good claim to the invention of the Enterprise Service Bus (ESB) and has done more than anyone else to evangelise the concept, backed by the resources of its parent company, Progress Software.
Sonic runs regular seminars (we attended one, entitled “SOA: Best Practices, an Architect's forum” in London this week); and Enterprise Service Bus by Dave Chappell, Sonic's VP and chief technology evangelist, (O'Reilly, ISBN 0-596-00675-6), is an excellent introduction to ESB.
The ESB concept seems to be maturing now, which means that everybody and his dog is talking ESB – and some of them - Iona's Artix, perhaps - are strong competition for Sonic's own ESB.
An ESB is a “standards-based integration platform that combines messaging, web services, data transformation to reliably connect and coordinate the interaction of significant numbers of diverse applications across extended enterprises with transactional integrity” (from Chappell's book).
It is better than a hub-and-spoke enterpris application integration (EAI) broker because there is no single bottleneck to scalability, but with that specification, appserver vendors such as IBM or BEA shouldn't have much trouble building one.
Starting more or less from scratch like Sonic has advantages (it should be leaner and faster with the latest technology), but re-using legacy technology has its points too (for a start, people are used to it).
So how do you distinguish yourself in this game? Well, by good marketing and by providing more, and more useful, services.
Sonic Software has done both. Its Service-Oriented Architecture SOA Maturity Model (developed in collaboration with AmberPoint and Systinet and due for publication at the end of October 2005) is partly a marketing tool as far as we can see - it exploits the Maturity Model concept associated with CMMI. CMMI deals with process management, not specific technologies, but it still has some practical uses.
You gain little over traditional approaches when you implement SOA at Maturity Level 1 (ML1) to achieve some new business functionality. But you are at the start of a progression towards something better and the SOA Maturity model helps you track your progress. Level 2 is about architected services and starts to bring cost reduction and control; Level 3 is about business and collaborative services and increases the responsiveness of your business process; Level 4 adds service metrics and promises to start delivering business transformation, from reactive to real-time process; and Level 5 adds automatic proactive optimisation of services using these metrics.
The various stages more-or-less correspond to the maturity levels in CMMI – but we suspect that you need to be operating at CMMI ML4 or 5 to really achieve SOA ML 4 or 5 benefits reliably, and installing an ESB is not a short cut to CMMI maturity.
At SOA ML4 and 5 the key technologies are, respectively, Business Activity Modelling or Event Stream Processing (ESP); and Event Stream Automation for optimisation. This is interesting, cutting-edge stuff (best exemplified by event-driven securities trading) – and probably provides Sonic with “more useful services” differentiation from its competition.
In essence, rather than analysing static records of the day's events, ESP looks at the pattern of business events in (very) near-real-time and then fires off alerts, or even instigates actions, quickly enough to allow emerging situations to be controlled before they become critical. At a recent Gartner Integration conference (Barcelona 2005), both SeeBeyond and WebMethods identified event processing of some sort as the next driver for integration and Progress' Real Time Division is well-placed to exploit this trend.
It has the SOA concept to hold everything together, the Sonic ESB to pass events along to where they can be acted on and object-oriented database technology (originally from ObjectStore) to implement an Event Store, capable of storing multitudes of real-time events and making them accessible in real-time too. The event store is fundamental to ESP and Progress's Real Time Division implements it as an in-memory database mapped onto disk storage for persistence using virtual-memory-like techniques; with indexes for after-the-fact, static, analysis.
Progress acquired its fundamental ESP knowledge by buying Apama, a leading innovator in algorithmic equities and futures trading, which is now extending its capabilities into the FX (foreign exchange) markets.
According to Mark Palmer, VP, Progress Real Time Division ESP, ESP as adds a missing part to an event-driven SOA - events are intrinsic to the ESB concept.
ESP can be differentiated from previous technologies, such as Business Activity Monitoring dashboards (although ESP drives dashboards too), by its event processing language and an engine that can output complex or derived events, he says. Complex events are based on processing real time events against, possibly, data in a conventional database and/or previous events in the Event Store and then generating new events, that can be transmitted to other applications over, for example, the Sonic ESB.
So, what can you use this new technology for? Well, clever financial trading and compliance checking are the current hot applications. A trader may split up very large stock purchases or sales so as to avoid moving the market price; but an ESP engine might notice a series of fixed size purchases a fixed distance apart, infer the placing of a large deal in total and react accordingly.
So, traders will start randomising the splitting up of such deals and the ESP will have to get cleverer. At the same time, ESP can be used to react to events such as, say, two brokers selling stock at exactly the same time (if deliberate, this can be used to manipulate prices; but it might happen by chance) and to enforce dealing rules generally.
Palmer, however points out that there are many other applications – processing the stream of events from RFID (Radio Frequency ID) tag readers for example, possibly acting on the discovery that a lot of tagged objects coming into a distribution facility and rather fewer are leaving each stage of breakdown and packaging. For example, Delta Airlines' “Delta Nervous System", which analyses weather patterns, aircraft positions etc, determines necessary flight re-routings and generates new events such as passenger rebookings, all in the interests of providing a better passenger experience (and more loyal customers), is another example of what you can do with event processing.
Nevertheless, ESP will have its issues. How easily can you test and validate complex event processing logic, for instance – is it possible that it works on a few events during testing but that aberrant behaviours emerge when hundreds of thousands of events, and derived events, are being issued and acted on in real time?
And, of course, there is governance - how do you show regulators that your real-time processing always followed the rules (which is why the event store, and an ability to replay events afterwards, is important)? We may suspect that mature SOA services such as ESP will require a certain level of maturity from the organisations operating them – although this is probably true of any advanced technology. ®