Feeds

Facebook reveals 700TB of tiered RAM and flash power Graph Search

The Social NetworkTM lifts skirts, shows glimpses of MySQL, Hadoop and Unicorn

Maximizing your infrastructure through virtualization

Facebook has revealed a little more detail about how it built Graph Search, the feature it took live a couple of weeks back to enable better searching of the billion posts it now collects each day.

The reveal came in a blog post, as is the fashion at this time, that explains building better search across posts is hard because Facebook has “70 different kinds of data we sort and index on, many of them specific to certain types of posts.”

That's all managed by MySQL, which passes data to a HBase cluster before Hadoop gets to work. At this point “Unicorn”, Facebook's name for the tech powering its search engine, gets involved and the result is a colossal index with a very substantial cache that the company tries to treat as gently as possible.

Keeping the cache happy is not a trivial task, so the company has devised the following tiering regime:

“However, storing more than 700 terabytes in RAM imposes a large amount of overhead, as it involves maintaining an index that is spread across many racks of machines. The performance cost of having these machines coordinate with each other drove the Unicorn team to look into new solutions for serving the posts index. The solution we decided on involves storing the majority of the index on solid-state flash memory. We managed to preserve performance by carefully separating out the most frequently accessed data structures and placing those in RAM.”

RAM as tier zero and flash as tier one are suggested as the recipe for fast access to data by just about the whole storage and big data industries, so Facebook's infrastructure will please many a sales and marketing person and probably make its way into lots of slideware before too much time passes.

Perhaps a little more interesting is the post's statement that Graph Search was built “in the past year by a few dozen engineers”. Plenty of lesser projects take more time and more people, suggesting the most interesting thing IT pros and their managers can learn from Graph Search might be how the project was managed and the hardware designed and implemented, rather than the configurations chosen. ®

The Power of One eBook: Top reasons to choose HP BladeSystem

More from The Register

next story
Sysadmin Day 2014: Quick, there's still time to get the beers in
He walked over the broken glass, killed the thugs... and er... reconnected the cables*
Auntie remains MYSTIFIED by that weekend BBC iPlayer and website outage
Still doing 'forensics' on the caching layer – Beeb digi wonk
SHOCK and AWS: The fall of Amazon's deflationary cloud
Just as Jeff Bezos did to books and CDs, Amazon's rivals are now doing to it
BlackBerry: Toss the server, mate... BES is in the CLOUD now
BlackBerry Enterprise Services takes aim at SMEs - but there's a catch
The triumph of VVOL: Everyone's jumping into bed with VMware
'Bandwagon'? Yes, we're on it and so what, say big dogs
Carbon tax repeal won't see data centre operators cut prices
Rackspace says electricity isn't a major cost, Equinix promises 'no levy'
prev story

Whitepapers

Implementing global e-invoicing with guaranteed legal certainty
Explaining the role local tax compliance plays in successful supply chain management and e-business and how leading global brands are addressing this.
Consolidation: The Foundation for IT Business Transformation
In this whitepaper learn how effective consolidation of IT and business resources can enable multiple, meaningful business benefits.
Application security programs and practises
Follow a few strategies and your organization can gain the full benefits of open source and the cloud without compromising the security of your applications.
How modern custom applications can spur business growth
Learn how to create, deploy and manage custom applications without consuming or expanding the need for scarce, expensive IT resources.
Securing Web Applications Made Simple and Scalable
Learn how automated security testing can provide a simple and scalable way to protect your web applications.