Feeds

Facebook reveals 700TB of tiered RAM and flash power Graph Search

The Social NetworkTM lifts skirts, shows glimpses of MySQL, Hadoop and Unicorn

Designing a Defense for Mobile Applications

Facebook has revealed a little more detail about how it built Graph Search, the feature it took live a couple of weeks back to enable better searching of the billion posts it now collects each day.

The reveal came in a blog post, as is the fashion at this time, that explains building better search across posts is hard because Facebook has “70 different kinds of data we sort and index on, many of them specific to certain types of posts.”

That's all managed by MySQL, which passes data to a HBase cluster before Hadoop gets to work. At this point “Unicorn”, Facebook's name for the tech powering its search engine, gets involved and the result is a colossal index with a very substantial cache that the company tries to treat as gently as possible.

Keeping the cache happy is not a trivial task, so the company has devised the following tiering regime:

“However, storing more than 700 terabytes in RAM imposes a large amount of overhead, as it involves maintaining an index that is spread across many racks of machines. The performance cost of having these machines coordinate with each other drove the Unicorn team to look into new solutions for serving the posts index. The solution we decided on involves storing the majority of the index on solid-state flash memory. We managed to preserve performance by carefully separating out the most frequently accessed data structures and placing those in RAM.”

RAM as tier zero and flash as tier one are suggested as the recipe for fast access to data by just about the whole storage and big data industries, so Facebook's infrastructure will please many a sales and marketing person and probably make its way into lots of slideware before too much time passes.

Perhaps a little more interesting is the post's statement that Graph Search was built “in the past year by a few dozen engineers”. Plenty of lesser projects take more time and more people, suggesting the most interesting thing IT pros and their managers can learn from Graph Search might be how the project was managed and the hardware designed and implemented, rather than the configurations chosen. ®

The Power of One eBook: Top reasons to choose HP BladeSystem

More from The Register

next story
Apple fanbois SCREAM as update BRICKS their Macbook Airs
Ragegasm spills over as firmware upgrade kills machines
Attack of the clones: Oracle's latest Red Hat Linux lookalike arrives
Oracle's Linux boss says Larry's Linux isn't just for Oracle apps anymore
THUD! WD plonks down SIX TERABYTE 'consumer NAS' fatboy
Now that's a LOT of porn or pirated movies. Or, you know, other consumer stuff
EU's top data cops to meet Google, Microsoft et al over 'right to be forgotten'
Plan to hammer out 'coherent' guidelines. Good luck chaps!
US judge: YES, cops or feds so can slurp an ENTIRE Gmail account
Crooks don't have folders labelled 'drug records', opines NY beak
Manic malware Mayhem spreads through Linux, FreeBSD web servers
And how Google could cripple infection rate in a second
FLAPE – the next BIG THING in storage
Find cold data with flash, transmit it from tape
prev story

Whitepapers

Designing a Defense for Mobile Applications
Learn about the various considerations for defending mobile applications - from the application architecture itself to the myriad testing technologies.
How modern custom applications can spur business growth
Learn how to create, deploy and manage custom applications without consuming or expanding the need for scarce, expensive IT resources.
Reducing security risks from open source software
Follow a few strategies and your organization can gain the full benefits of open source and the cloud without compromising the security of your applications.
Boost IT visibility and business value
How building a great service catalog relieves pressure points and demonstrates the value of IT service management.
Consolidation: the foundation for IT and business transformation
In this whitepaper learn how effective consolidation of IT and business resources can enable multiple, meaningful business benefits.