Sumo Logic slings log file ops into the clouds

Sumo Logic CTO and co-founder Christian Beedgen sports a Chatsubo Bar banner on his t-shirt. When you meet a startup guy wearing a Neuromancer t-shirt you know the product is going to be a dreadful self-indulgent failure or an extremely cool resource that delivers the goods. Sumo Logic's Log Management and Analytics Service is the latter.

Sumo Logic co-founder and CTO Christian Beedgen (IT Press Tour)

But log files? Log files are not cool. However, they are vital when apps degrade or crash. The log files – the record of the events that occur when the app executes – are the only detailed info into the application's run-time history that IT administrators have. The collected log file records are like a data centre flight recorder, a black box. When there is an airplane crash or failure the black box is the first port of call.

Every app has its own log file format, though they are often ASCII files. There can be tens of thousands of these records, terabytes of the stuff. And the problem of checking log files grows because we have application suites with many components running on different servers linked by network devices. Soon there are millions of log file records. The problem expands to finding the ones of interest, the needles in the vast haystack, and finding them quickly when something goes wrong in an application suite that is used by hundreds if not thousands of users and is critical to the running of your business.

Sumar Logic co-founder and acting CEO Kumar Saurabh

There is a second aspect to this, and that is the storing of the log files and ensuring you have enough storage capacity for that and then enough server capacity to run the analytic queries. Sumo Logic aims to deal with both problems.

Log file library operations

What co-founder and acting CEO Kumar Saurabh and Beedgen have done is to provide a cloud-based log file recording and storing service together with analytics software to help admin staff search and identify log file records of interest. We could think of Sumo Logic's Log Management and Analytics Service as working like a librarian. Books are stored in a library and classified, by the Dewey Decimal scheme for example.

Researchers need to find books but don't understand book type classification. They tell the librarians what they want and the library staff find it, without themselves knowing about the detailed contents of books. The analytics service provides algorithms for finding log files of interest. It has a specific function called Log reduce, a tongue-in-cheek reference to Hadoop's Map Reduce. Although it works in a different way the net effect is the same in that it produces a subset of records of interest out of the thousands of possible hits.

Operators might be looking for events connected with a particular user, IP address or business code of some sort and Log Reduce will quickly sort through a large population of log files and produce a manageable subset that a human can look through and assess and use to decide what to investigate next.

Patterns and collectors

The analytics software looks for patterns and can learn what patterns to look out for, further helping operators to refine subsets of interest.

The log files are collected by collectors, software agents that can operate in the servers where the events are occurring or remote from them, and thus not using any CPU cycles in them. They are collected and then sent up to the cloud for subsequent access. Although there can be many of them they are individually small and don't consume much network bandwidth.

Whatever cost is involved here is inconsequential when compared to the increase in productivity of the operators investigating log files for the cause of a failure, the founders say. Sumo Logic uses its own software to manage the operations of its own service and so has direct experience of its utility and a direct connection to its users' experience of the service.

CTO Beedgen said that the days of developing software and shipping copies to customers all over the world are gone ... and as far as he is concerned, they are not lamented. It is far easier for developers to look after software in one place which provides a service to thousands of users than thousands of copies of software in thousands of remote places needing a substantial support and software distribution operation.

The cloud is real

You listen to people like Beedgen and Saurabh explaining how their cloud service works, how it improves the necessary grunt work involved in log file collection, storage and analysis, and you realise that for applications like this the cloud is just a natural and logical fit. The pair are providing log monitoring and analytics to cloud computing environments via a Software-as-a-Service model.

Sumo Logic is doing its bit to transform the world of log file collection, storage and analysis into a cloud service. For its users, once they have experienced log file operations in the cloud, there is pretty much no going back. ®