Feeds

Sumo Logic slings log file ops into the clouds

Meet me at the Chatsubo bar for a big data bash

Internet Security Threat Report 2014

Sumo Logic CTO and co-founder Christian Beedgen sports a Chatsubo Bar banner on his t-shirt. When you meet a startup guy wearing a Neuromancer t-shirt you know the product is going to be a dreadful self-indulgent failure or an extremely cool resource that delivers the goods. Sumo Logic's Log Management and Analytics Service is the latter.

Christian Beedgen

Sumo Logic co-founder and CTO Christian Beedgen (IT Press Tour)

But log files? Log files are not cool. However, they are vital when apps degrade or crash. The log files – the record of the events that occur when the app executes – are the only detailed info into the application's run-time history that IT administrators have. The collected log file records are like a data centre flight recorder, a black box. When there is an airplane crash or failure the black box is the first port of call.

Every app has its own log file format, though they are often ASCII files. There can be tens of thousands of these records, terabytes of the stuff. And the problem of checking log files grows because we have application suites with many components running on different servers linked by network devices. Soon there are millions of log file records. The problem expands to finding the ones of interest, the needles in the vast haystack, and finding them quickly when something goes wrong in an application suite that is used by hundreds if not thousands of users and is critical to the running of your business.

Kumar Saurabh

Sumar Logic co-founder and acting CEO Kumar Saurabh

There is a second aspect to this, and that is the storing of the log files and ensuring you have enough storage capacity for that and then enough server capacity to run the analytic queries. Sumo Logic aims to deal with both problems.

Log file library operations

What co-founder and acting CEO Kumar Saurabh and Beedgen have done is to provide a cloud-based log file recording and storing service together with analytics software to help admin staff search and identify log file records of interest. We could think of Sumo Logic's Log Management and Analytics Service as working like a librarian. Books are stored in a library and classified, by the Dewey Decimal scheme for example.

Researchers need to find books but don't understand book type classification. They tell the librarians what they want and the library staff find it, without themselves knowing about the detailed contents of books. The analytics service provides algorithms for finding log files of interest. It has a specific function called Log reduce, a tongue-in-cheek reference to Hadoop's Map Reduce. Although it works in a different way the net effect is the same in that it produces a subset of records of interest out of the thousands of possible hits.

Operators might be looking for events connected with a particular user, IP address or business code of some sort and Log Reduce will quickly sort through a large population of log files and produce a manageable subset that a human can look through and assess and use to decide what to investigate next.

Patterns and collectors

The analytics software looks for patterns and can learn what patterns to look out for, further helping operators to refine subsets of interest.

The log files are collected by collectors, software agents that can operate in the servers where the events are occurring or remote from them, and thus not using any CPU cycles in them. They are collected and then sent up to the cloud for subsequent access. Although there can be many of them they are individually small and don't consume much network bandwidth.

Whatever cost is involved here is inconsequential when compared to the increase in productivity of the operators investigating log files for the cause of a failure, the founders say. Sumo Logic uses its own software to manage the operations of its own service and so has direct experience of its utility and a direct connection to its users' experience of the service.

CTO Beedgen said that the days of developing software and shipping copies to customers all over the world are gone ... and as far as he is concerned, they are not lamented. It is far easier for developers to look after software in one place which provides a service to thousands of users than thousands of copies of software in thousands of remote places needing a substantial support and software distribution operation.

The cloud is real

You listen to people like Beedgen and Saurabh explaining how their cloud service works, how it improves the necessary grunt work involved in log file collection, storage and analysis, and you realise that for applications like this the cloud is just a natural and logical fit. The pair are providing log monitoring and analytics to cloud computing environments via a Software-as-a-Service model.

Sumo Logic is doing its bit to transform the world of log file collection, storage and analysis into a cloud service. For its users, once they have experienced log file operations in the cloud, there is pretty much no going back. ®

Beginner's guide to SSL certificates

More from The Register

next story
Docker's app containers are coming to Windows Server, says Microsoft
MS chases app deployment speeds already enjoyed by Linux devs
'Hmm, why CAN'T I run a water pipe through that rack of media servers?'
Leaving Las Vegas for Armenia kludging and Dubai dune bashing
SDI wars: WTF is software defined infrastructure?
This time we play for ALL the marbles
'Urika': Cray unveils new 1,500-core big data crunching monster
6TB of DRAM, 38TB of SSD flash and 120TB of disk storage
Facebook slurps 'paste sites' for STOLEN passwords, sprinkles on hash and salt
Zuck's ad empire DOESN'T see details in plain text. Phew!
Windows 10: Forget Cloudobile, put Security and Privacy First
But - dammit - It would be insane to say 'don't collect, because NSA'
Oracle hires former SAP exec for cloudy push
'We know Larry said cloud was gibberish, and insane, and idiotic, but...'
Symantec backs out of Backup Exec: Plans to can appliance in Jan
Will still provide support to existing customers
prev story

Whitepapers

Forging a new future with identity relationship management
Learn about ForgeRock's next generation IRM platform and how it is designed to empower CEOS's and enterprises to engage with consumers.
Why cloud backup?
Combining the latest advancements in disk-based backup with secure, integrated, cloud technologies offer organizations fast and assured recovery of their critical enterprise data.
Win a year’s supply of chocolate
There is no techie angle to this competition so we're not going to pretend there is, but everyone loves chocolate so who cares.
High Performance for All
While HPC is not new, it has traditionally been seen as a specialist area – is it now geared up to meet more mainstream requirements?
Intelligent flash storage arrays
Tegile Intelligent Storage Arrays with IntelliFlash helps IT boost storage utilization and effciency while delivering unmatched storage savings and performance.