Feeds

Twitter to open source MySQL-to-Hadoop tool

Data Crane

Choosing a cloud hosting partner with confidence

Hadoop Summit Twitter intends to open source an additional piece of the Hadoop-happy infrastructure it uses for internal data analysis. Known as Crane, this is a tool for moving data from MySQL into Hadoop, the open source data-crunching platform based on Google's proprietary infrastructure.

Twitter uses Hadoop for ad hoc analysis of data collected from its famous microblogging service, but the platform also crunches data for use by live tools on the site, including Twitter's name-search function.

Speaking today at the Yahoo!-sponsored Hadoop Summit in Santa Clara, California, Twitter analytics man Kevin Wiel explained that the company handles Hadoop data input in essentially two ways. It does log collection with the open source Scribe developed at Facebook, logging seven terabytes of data into the Hadoop File System (HDFS) each day, and it handles tabular data with Crane.

Most of Twitter's tabular data is stored in MySQL, though "a little" is stored in the Cassandra open source distributed database and Twitter's open source "social graph" data store, Flock. "Other than that," Wiel said. "Everything you do on Twitter ends up in a MySQL table somewhere."

Crane was developed to move data from MySQL to the HDFS or to the Hadoop-friendly distributed database known as HBase, but also to other MySQL databases. "We needed to have a flexible data-moving tool, so we built Crane, which is a configuration-driven ETL [extract, transform, and load] tool," Wiel says.

The tool moves data not only into MySQL, HDFS, and HBase, but also into Flock, Google Analytics, and Facebook Insights.

Like Yahoo! — and unlike Facebook — Twitter does its Hadoop programming in Pig. Developed by Yahoo!, the open source Pig is a lower-level language than the Facebook-developed Hive. But it operates at a significantly higher level than raw Hadoop MapReduce code.

According to Wiel, Pig requires five per cent of the coding and five percent of the code compared to Hadoop MapReduce, and it comes within 30 per cent of the execution time.

Twitter employees access Hadoop via dashboard known as BirdBrain, much like Facebookers use a Hive GUI known as HiPal.

A more general Hadoop interface was just open sourced by all-star startup Cloudera. Formerly known as the Cloudera Desktop, HUE — short for Hadoop User Interface — provides a web-based graphical user interface for creating and submitting jobs on a Hadoop cluster, monitoring the cluster's health, and browsing stored data. ®

Choosing a cloud hosting partner with confidence

More from The Register

next story
Nexus 7 fandroids tell of salty taste after sucking on Google's Lollipop
Web giant looking into why version 5.0 of Android is crippling older slabs
Be real, Apple: In-app goodie grab games AREN'T FREE – EU
Cupertino stands down after Euro legal threats
Download alert: Nearly ALL top 100 Android, iOS paid apps hacked
Attack of the Clones? Yeah, but much, much scarier – report
Microsoft: Your Linux Docker containers are now OURS to command
New tool lets admins wrangle Linux apps from Windows
Bada-Bing! Mozilla flips Firefox to YAHOO! for search
Microsoft system will be the default for browser in US until 2020
Facebook, working on Facebook at Work, works on Facebook. At Work
You don't want your cat or drunk pics at the office
prev story

Whitepapers

Why and how to choose the right cloud vendor
The benefits of cloud-based storage in your processes. Eliminate onsite, disk-based backup and archiving in favor of cloud-based data protection.
Getting started with customer-focused identity management
Learn why identity is a fundamental requirement to digital growth, and how without it there is no way to identify and engage customers in a meaningful way.
Reg Reader Research: SaaS based Email and Office Productivity Tools
Read this Reg reader report which provides advice and guidance for SMBs towards the use of SaaS based email and Office productivity tools.
The hidden costs of self-signed SSL certificates
Exploring the true TCO for self-signed SSL certificates, including a side-by-side comparison of a self-signed architecture versus working with a third-party SSL vendor.
Intelligent flash storage arrays
Tegile Intelligent Storage Arrays with IntelliFlash helps IT boost storage utilization and effciency while delivering unmatched storage savings and performance.