Feeds

Sumo Logic slings log file ops into the clouds

Meet me at the Chatsubo bar for a big data bash

High performance access to file storage

Sumo Logic CTO and co-founder Christian Beedgen sports a Chatsubo Bar banner on his t-shirt. When you meet a startup guy wearing a Neuromancer t-shirt you know the product is going to be a dreadful self-indulgent failure or an extremely cool resource that delivers the goods. Sumo Logic's Log Management and Analytics Service is the latter.

Christian Beedgen

Sumo Logic co-founder and CTO Christian Beedgen (IT Press Tour)

But log files? Log files are not cool. However, they are vital when apps degrade or crash. The log files – the record of the events that occur when the app executes – are the only detailed info into the application's run-time history that IT administrators have. The collected log file records are like a data centre flight recorder, a black box. When there is an airplane crash or failure the black box is the first port of call.

Every app has its own log file format, though they are often ASCII files. There can be tens of thousands of these records, terabytes of the stuff. And the problem of checking log files grows because we have application suites with many components running on different servers linked by network devices. Soon there are millions of log file records. The problem expands to finding the ones of interest, the needles in the vast haystack, and finding them quickly when something goes wrong in an application suite that is used by hundreds if not thousands of users and is critical to the running of your business.

Kumar Saurabh

Sumar Logic co-founder and acting CEO Kumar Saurabh

There is a second aspect to this, and that is the storing of the log files and ensuring you have enough storage capacity for that and then enough server capacity to run the analytic queries. Sumo Logic aims to deal with both problems.

Log file library operations

What co-founder and acting CEO Kumar Saurabh and Beedgen have done is to provide a cloud-based log file recording and storing service together with analytics software to help admin staff search and identify log file records of interest. We could think of Sumo Logic's Log Management and Analytics Service as working like a librarian. Books are stored in a library and classified, by the Dewey Decimal scheme for example.

Researchers need to find books but don't understand book type classification. They tell the librarians what they want and the library staff find it, without themselves knowing about the detailed contents of books. The analytics service provides algorithms for finding log files of interest. It has a specific function called Log reduce, a tongue-in-cheek reference to Hadoop's Map Reduce. Although it works in a different way the net effect is the same in that it produces a subset of records of interest out of the thousands of possible hits.

Operators might be looking for events connected with a particular user, IP address or business code of some sort and Log Reduce will quickly sort through a large population of log files and produce a manageable subset that a human can look through and assess and use to decide what to investigate next.

Patterns and collectors

The analytics software looks for patterns and can learn what patterns to look out for, further helping operators to refine subsets of interest.

The log files are collected by collectors, software agents that can operate in the servers where the events are occurring or remote from them, and thus not using any CPU cycles in them. They are collected and then sent up to the cloud for subsequent access. Although there can be many of them they are individually small and don't consume much network bandwidth.

Whatever cost is involved here is inconsequential when compared to the increase in productivity of the operators investigating log files for the cause of a failure, the founders say. Sumo Logic uses its own software to manage the operations of its own service and so has direct experience of its utility and a direct connection to its users' experience of the service.

CTO Beedgen said that the days of developing software and shipping copies to customers all over the world are gone ... and as far as he is concerned, they are not lamented. It is far easier for developers to look after software in one place which provides a service to thousands of users than thousands of copies of software in thousands of remote places needing a substantial support and software distribution operation.

The cloud is real

You listen to people like Beedgen and Saurabh explaining how their cloud service works, how it improves the necessary grunt work involved in log file collection, storage and analysis, and you realise that for applications like this the cloud is just a natural and logical fit. The pair are providing log monitoring and analytics to cloud computing environments via a Software-as-a-Service model.

Sumo Logic is doing its bit to transform the world of log file collection, storage and analysis into a cloud service. For its users, once they have experienced log file operations in the cloud, there is pretty much no going back. ®

High performance access to file storage

More from The Register

next story
Seagate brings out 6TB HDD, did not need NO STEENKIN' SHINGLES
Or helium filling either, according to reports
European Court of Justice rips up Data Retention Directive
Rules 'interfering' measure to be 'invalid'
Dropbox defends fantastically badly timed Condoleezza Rice appointment
'Nothing is going to change with Dr. Rice's appointment,' file sharer promises
Cisco reps flog Whiptail's Invicta arrays against EMC and Pure
Storage reseller report reveals who's selling what
Just what could be inside Dropbox's new 'Home For Life'?
Biz apps, messaging, photos, email, more storage – sorry, did you think there would be cake?
IT bods: How long does it take YOU to train up on new tech?
I'll leave my arrays to do the hard work, if you don't mind
Amazon reveals its Google-killing 'R3' server instances
A mega-memory instance that never forgets
USA opposes 'Schengen cloud' Eurocentric routing plan
All routes should transit America, apparently
prev story

Whitepapers

Mainstay ROI - Does application security pay?
In this whitepaper learn how you and your enterprise might benefit from better software security.
Five 3D headsets to be won!
We were so impressed by the Durovis Dive headset we’ve asked the company to give some away to Reg readers.
3 Big data security analytics techniques
Applying these Big Data security analytics techniques can help you make your business safer by detecting attacks early, before significant damage is done.
The benefits of software based PBX
Why you should break free from your proprietary PBX and how to leverage your existing server hardware.
Mobile application security study
Download this report to see the alarming realities regarding the sheer number of applications vulnerable to attack, as well as the most common and easily addressable vulnerability errors.