Feeds

Twitter to open source MySQL-to-Hadoop tool

Data Crane

3 Big data security analytics techniques

Hadoop Summit Twitter intends to open source an additional piece of the Hadoop-happy infrastructure it uses for internal data analysis. Known as Crane, this is a tool for moving data from MySQL into Hadoop, the open source data-crunching platform based on Google's proprietary infrastructure.

Twitter uses Hadoop for ad hoc analysis of data collected from its famous microblogging service, but the platform also crunches data for use by live tools on the site, including Twitter's name-search function.

Speaking today at the Yahoo!-sponsored Hadoop Summit in Santa Clara, California, Twitter analytics man Kevin Wiel explained that the company handles Hadoop data input in essentially two ways. It does log collection with the open source Scribe developed at Facebook, logging seven terabytes of data into the Hadoop File System (HDFS) each day, and it handles tabular data with Crane.

Most of Twitter's tabular data is stored in MySQL, though "a little" is stored in the Cassandra open source distributed database and Twitter's open source "social graph" data store, Flock. "Other than that," Wiel said. "Everything you do on Twitter ends up in a MySQL table somewhere."

Crane was developed to move data from MySQL to the HDFS or to the Hadoop-friendly distributed database known as HBase, but also to other MySQL databases. "We needed to have a flexible data-moving tool, so we built Crane, which is a configuration-driven ETL [extract, transform, and load] tool," Wiel says.

The tool moves data not only into MySQL, HDFS, and HBase, but also into Flock, Google Analytics, and Facebook Insights.

Like Yahoo! — and unlike Facebook — Twitter does its Hadoop programming in Pig. Developed by Yahoo!, the open source Pig is a lower-level language than the Facebook-developed Hive. But it operates at a significantly higher level than raw Hadoop MapReduce code.

According to Wiel, Pig requires five per cent of the coding and five percent of the code compared to Hadoop MapReduce, and it comes within 30 per cent of the execution time.

Twitter employees access Hadoop via dashboard known as BirdBrain, much like Facebookers use a Hive GUI known as HiPal.

A more general Hadoop interface was just open sourced by all-star startup Cloudera. Formerly known as the Cloudera Desktop, HUE — short for Hadoop User Interface — provides a web-based graphical user interface for creating and submitting jobs on a Hadoop cluster, monitoring the cluster's health, and browsing stored data. ®

SANS - Survey on application security programs

More from The Register

next story
This time it's 'Personal': new Office 365 sub covers just two devices
Redmond also brings Office into Google's back yard
Oh no, Joe: WinPhone users already griping over 8.1 mega-update
Hang on. Which bit of Developer Preview don't you understand?
Microsoft lobs pre-release Windows Phone 8.1 at devs who dare
App makers can load it before anyone else, but if they do they're stuck with it
Half of Twitter's 'active users' are SILENT STALKERS
Nearly 50% have NEVER tweeted a word
Internet-of-stuff startup dumps NoSQL for ... SQL?
NoSQL taste great at first but lacks proper nutrients, says startup cloud whiz
Next Windows obsolescence panic is 450 days from … NOW!
The clock is ticking louder for Windows Server 2003 R2 users
Ditch the sync, paddle in the Streem: Upstart offers syncless sharing
Upload, delete and carry on sharing afterwards?
Microsoft TIER SMEAR changes app prices whether devs ask or not
Some go up, some go down, Redmond goes silent
Batten down the hatches, Ubuntu 14.04 LTS due in TWO DAYS
Admins dab straining server brows in advance of Trusty Tahr's long-term support landing
prev story

Whitepapers

SANS - Survey on application security programs
In this whitepaper learn about the state of application security programs and practices of 488 surveyed respondents, and discover how mature and effective these programs are.
Combat fraud and increase customer satisfaction
Based on their experience using HP ArcSight Enterprise Security Manager for IT security operations, Finansbank moved to HP ArcSight ESM for fraud management.
The benefits of software based PBX
Why you should break free from your proprietary PBX and how to leverage your existing server hardware.
Top three mobile application threats
Learn about three of the top mobile application security threats facing businesses today and recommendations on how to mitigate the risk.
3 Big data security analytics techniques
Applying these Big Data security analytics techniques can help you make your business safer by detecting attacks early, before significant damage is done.