Feeds

Sumo Logic slings log file ops into the clouds

Meet me at the Chatsubo bar for a big data bash

Top three mobile application threats

Sumo Logic CTO and co-founder Christian Beedgen sports a Chatsubo Bar banner on his t-shirt. When you meet a startup guy wearing a Neuromancer t-shirt you know the product is going to be a dreadful self-indulgent failure or an extremely cool resource that delivers the goods. Sumo Logic's Log Management and Analytics Service is the latter.

Christian Beedgen

Sumo Logic co-founder and CTO Christian Beedgen (IT Press Tour)

But log files? Log files are not cool. However, they are vital when apps degrade or crash. The log files – the record of the events that occur when the app executes – are the only detailed info into the application's run-time history that IT administrators have. The collected log file records are like a data centre flight recorder, a black box. When there is an airplane crash or failure the black box is the first port of call.

Every app has its own log file format, though they are often ASCII files. There can be tens of thousands of these records, terabytes of the stuff. And the problem of checking log files grows because we have application suites with many components running on different servers linked by network devices. Soon there are millions of log file records. The problem expands to finding the ones of interest, the needles in the vast haystack, and finding them quickly when something goes wrong in an application suite that is used by hundreds if not thousands of users and is critical to the running of your business.

Kumar Saurabh

Sumar Logic co-founder and acting CEO Kumar Saurabh

There is a second aspect to this, and that is the storing of the log files and ensuring you have enough storage capacity for that and then enough server capacity to run the analytic queries. Sumo Logic aims to deal with both problems.

Log file library operations

What co-founder and acting CEO Kumar Saurabh and Beedgen have done is to provide a cloud-based log file recording and storing service together with analytics software to help admin staff search and identify log file records of interest. We could think of Sumo Logic's Log Management and Analytics Service as working like a librarian. Books are stored in a library and classified, by the Dewey Decimal scheme for example.

Researchers need to find books but don't understand book type classification. They tell the librarians what they want and the library staff find it, without themselves knowing about the detailed contents of books. The analytics service provides algorithms for finding log files of interest. It has a specific function called Log reduce, a tongue-in-cheek reference to Hadoop's Map Reduce. Although it works in a different way the net effect is the same in that it produces a subset of records of interest out of the thousands of possible hits.

Operators might be looking for events connected with a particular user, IP address or business code of some sort and Log Reduce will quickly sort through a large population of log files and produce a manageable subset that a human can look through and assess and use to decide what to investigate next.

Patterns and collectors

The analytics software looks for patterns and can learn what patterns to look out for, further helping operators to refine subsets of interest.

The log files are collected by collectors, software agents that can operate in the servers where the events are occurring or remote from them, and thus not using any CPU cycles in them. They are collected and then sent up to the cloud for subsequent access. Although there can be many of them they are individually small and don't consume much network bandwidth.

Whatever cost is involved here is inconsequential when compared to the increase in productivity of the operators investigating log files for the cause of a failure, the founders say. Sumo Logic uses its own software to manage the operations of its own service and so has direct experience of its utility and a direct connection to its users' experience of the service.

CTO Beedgen said that the days of developing software and shipping copies to customers all over the world are gone ... and as far as he is concerned, they are not lamented. It is far easier for developers to look after software in one place which provides a service to thousands of users than thousands of copies of software in thousands of remote places needing a substantial support and software distribution operation.

The cloud is real

You listen to people like Beedgen and Saurabh explaining how their cloud service works, how it improves the necessary grunt work involved in log file collection, storage and analysis, and you realise that for applications like this the cloud is just a natural and logical fit. The pair are providing log monitoring and analytics to cloud computing environments via a Software-as-a-Service model.

Sumo Logic is doing its bit to transform the world of log file collection, storage and analysis into a cloud service. For its users, once they have experienced log file operations in the cloud, there is pretty much no going back. ®

Combat fraud and increase customer satisfaction

More from The Register

next story
This time it's 'Personal': new Office 365 sub covers just two devices
Redmond also brings Office into Google's back yard
Kingston DataTraveler MicroDuo: Turn your phone into a 72GB beast
USB-usiness in the front, micro-USB party in the back
AMD's 'Seattle' 64-bit ARM server chips now sampling, set to launch in late 2014
But they won't appear in SeaMicro Fabric Compute Systems anytime soon
Brit boffins use TARDIS to re-route data flows through time and space
'Traffic Assignment and Retiming Dynamics with Inherent Stability' algo can save ISPs big bucks
Microsoft's Nadella: SQL Server 2014 means we're all about data
Adds new big data tools in quest for 'ambient intelligence'
prev story

Whitepapers

Securing web applications made simple and scalable
In this whitepaper learn how automated security testing can provide a simple and scalable way to protect your web applications.
Combat fraud and increase customer satisfaction
Based on their experience using HP ArcSight Enterprise Security Manager for IT security operations, Finansbank moved to HP ArcSight ESM for fraud management.
The benefits of software based PBX
Why you should break free from your proprietary PBX and how to leverage your existing server hardware.
SANS - Survey on application security programs
In this whitepaper learn about the state of application security programs and practices of 488 surveyed respondents, and discover how mature and effective these programs are.
3 Big data security analytics techniques
Applying these Big Data security analytics techniques can help you make your business safer by detecting attacks early, before significant damage is done.