Feeds

Twitter to open source MySQL-to-Hadoop tool

Data Crane

7 Elements of Radically Simple OS Migration

Hadoop Summit Twitter intends to open source an additional piece of the Hadoop-happy infrastructure it uses for internal data analysis. Known as Crane, this is a tool for moving data from MySQL into Hadoop, the open source data-crunching platform based on Google's proprietary infrastructure.

Twitter uses Hadoop for ad hoc analysis of data collected from its famous microblogging service, but the platform also crunches data for use by live tools on the site, including Twitter's name-search function.

Speaking today at the Yahoo!-sponsored Hadoop Summit in Santa Clara, California, Twitter analytics man Kevin Wiel explained that the company handles Hadoop data input in essentially two ways. It does log collection with the open source Scribe developed at Facebook, logging seven terabytes of data into the Hadoop File System (HDFS) each day, and it handles tabular data with Crane.

Most of Twitter's tabular data is stored in MySQL, though "a little" is stored in the Cassandra open source distributed database and Twitter's open source "social graph" data store, Flock. "Other than that," Wiel said. "Everything you do on Twitter ends up in a MySQL table somewhere."

Crane was developed to move data from MySQL to the HDFS or to the Hadoop-friendly distributed database known as HBase, but also to other MySQL databases. "We needed to have a flexible data-moving tool, so we built Crane, which is a configuration-driven ETL [extract, transform, and load] tool," Wiel says.

The tool moves data not only into MySQL, HDFS, and HBase, but also into Flock, Google Analytics, and Facebook Insights.

Like Yahoo! — and unlike Facebook — Twitter does its Hadoop programming in Pig. Developed by Yahoo!, the open source Pig is a lower-level language than the Facebook-developed Hive. But it operates at a significantly higher level than raw Hadoop MapReduce code.

According to Wiel, Pig requires five per cent of the coding and five percent of the code compared to Hadoop MapReduce, and it comes within 30 per cent of the execution time.

Twitter employees access Hadoop via dashboard known as BirdBrain, much like Facebookers use a Hive GUI known as HiPal.

A more general Hadoop interface was just open sourced by all-star startup Cloudera. Formerly known as the Cloudera Desktop, HUE — short for Hadoop User Interface — provides a web-based graphical user interface for creating and submitting jobs on a Hadoop cluster, monitoring the cluster's health, and browsing stored data. ®

Endpoint data privacy in the cloud is easier than you think

More from The Register

next story
PEAK LANDFILL: Why tablet gloom is good news for Windows users
Sinofsky's hybrid strategy looks dafter than ever
Leaked Windows Phone 8.1 Update specs tease details of Nokia's next mobes
New screen sizes, dual SIMs, voice over LTE, and more
POW! Apple smites Macbook Air EFI firmware update borkage
Fruity firm provides digital balm for furious fanbois
Call off the firing squad: HP grants stay of execution to OpenVMS
Startup to take over support for today's Itaniums and beyond
Fiendishly complex password app extension ships for iOS 8
Just slip it in, won't hurt a bit, 1Password makers urge devs
Mozilla keeps its Beard, hopes anti-gay marriage troubles are now over
Plenty on new CEO's todo list – starting with Firefox's slipping grasp
Apple: We'll unleash OS X Yosemite beta on the MASSES on 24 July
Starting today, regular fanbois will be guinea pigs, it tells Reg
Another day, another Firefox: Version 31 is upon us ALREADY
Web devs, Mozilla really wants you to like this one
prev story

Whitepapers

7 Elements of Radically Simple OS Migration
Avoid the typical headaches of OS migration during your next project by learning about 7 elements of radically simple OS migration.
Implementing global e-invoicing with guaranteed legal certainty
Explaining the role local tax compliance plays in successful supply chain management and e-business and how leading global brands are addressing this.
Consolidation: The Foundation for IT Business Transformation
In this whitepaper learn how effective consolidation of IT and business resources can enable multiple, meaningful business benefits.
Solving today's distributed Big Data backup challenges
Enable IT efficiency and allow a firm to access and reuse corporate information for competitive advantage, ultimately changing business outcomes.
A new approach to endpoint data protection
What is the best way to ensure comprehensive visibility, management, and control of information on both company-owned and employee-owned devices?