Feeds

Facebook knuckle-raps Intel, AMD

'The server vendors have failed us'

Combat fraud and increase customer satisfaction

While social networking site Facebook doesn't seem inclined to pull a Google and build all of its own servers, the company's top techies are frustrated enough with the current crop of x64 boxes that they may be giving the idea some thought.

Speaking at the Structure 09 conference in San Francisco yesterday, Jonathan Heiliger, vice president of technical operations at Facebook, first expressed his disappointment with chip makers Intel and AMD and then server vendors in addressing the infrastructure challenges the company faces. (You can see a video of Heiliger's chat here).

"The biggest thing that surprised us - or is about to surprise us - is the less-than-anticipated performance gains from new microarchitectures," Heiliger explained in a Q&A session with GigaOM's Om Malik, referencing the new chips from Intel and AMD in particular. "The performance gains they are touting in the press, we are not seeing in our applications. We are literally in real-time trying to figure out why that is and if there are optimizations that we can do. Otherwise, we are kind of left with current-generation technology and current-generation scale."

Of course, it's unlikely that Facebook will return to using systems from one or two generations ago. But the situation may point out that the performance gains that Intel and AMD are showing on tests may not translate well into the heavily customized Facebook stack, which is written in PHP and uses memcached as a temporary cache store for data serving up its pages and MySQL as a permanent store for that data behind the caching servers. Or - and this seems more likely - it may simply point out the limits of the open source software Facebook has chosen to run its site.

Both MySQL and memcached are not particularly good at scaling on many cores and threads, which is why companies like Facebook scale horizontally in the first place. The scalability issue is why a number of vendors have come out recently with multithreaded versions of memcached (the appliance made by Schooner, which launched in April comes to mind), and it is also why Sun Microsystems has been touting that its future MySQL 8.4 (or rather, Oracle's future MySQL 5.4) will scale across 16 processor threads, whereas the current MySQL 5.1 only spans four threads.

With the new two-socket x64 boxes having 12 threads for an Istanbul Opteron and 16 threads for a Nehalem EP Xeon, it should not be all that much of a surprise to Heiliger that new iron is not making Faceook's caching and database tiers run faster. The software can't use the threads. And the only reason that MySQL 5.4 will be able to span up to 16 threads is because Google is contributing code to the MySQL project. (I have no idea how PHP is or is not making use of all those extra threads in two-socket Nehalem EP and Istanbul servers).

With regard to the other pet peeve that Facebook has about its server infrastructure - power consumption - Heiliger heaped a certain amount of scorn on the world's server makers, who are no doubt constantly knocking on his door to try to win the next 10,000 server deal that Facebook does.

"I am not sure whether to be embarrassed or pleased for the OEM and system vendors in the audience," Heiliger said, "but you guys just don't get it." Heiliger said that it was not enough to just have servers with low-voltage chips and more efficient power supplies, but that server makers had to do what Google has done, which is to trim out every watt it can and minimize the wall power that a server consumes as it does its work.

"I am not sure why the server vendors have failed us," Heiliger added when asked why Facebook wasn't getting the machines it really wanted to buy. But he conceded that there is a bit of a chicken-and-egg problem in that server makers have to design for the belly of the market and that they have come around to doing custom designs for hyperscale customers like Facebook for orders of tens of thousands of nodes at a time.

Heiliger's own advice for providing scalability - the kind that a hyperscale company like Facebook needs - was simple, and it is fair to say that maybe Facebook needs to take its own advice a little more literally to be happy. "Pick an area where you are going to own - whether it is the application or the software infrastructure or the hardware infrastructure - and start with that area," Heiliger advised. "As you grow and as you scale, you can add the other two. You can start anywhere, but I would probably recommend starting with the application first."

Following this advice, it would seem that Facebook needs to buy lots more new x64 servers and to get down into the guts of its open source code and make it more scalable. Or design its own machines that give the best performance per watt on its existing code using older or at least different iron. Might I suggest some racks of Nano or Atom servers to start? ®

Combat fraud and increase customer satisfaction

More from The Register

next story
This time it's 'Personal': new Office 365 sub covers just two devices
Redmond also brings Office into Google's back yard
Kingston DataTraveler MicroDuo: Turn your phone into a 72GB beast
USB-usiness in the front, micro-USB party in the back
Dropbox defends fantastically badly timed Condoleezza Rice appointment
'Nothing is going to change with Dr. Rice's appointment,' file sharer promises
BOFH: Oh DO tell us what you think. *CLICK*
$%%&amp Oh dear, we've been cut *CLICK* Well hello *CLICK* You're breaking up...
AMD's 'Seattle' 64-bit ARM server chips now sampling, set to launch in late 2014
But they won't appear in SeaMicro Fabric Compute Systems anytime soon
Amazon reveals its Google-killing 'R3' server instances
A mega-memory instance that never forgets
Cisco reps flog Whiptail's Invicta arrays against EMC and Pure
Storage reseller report reveals who's selling what
Microsoft builds teleporter weapon to send VMware into Azure
Updated Virtual Machine Converter now converts Linux VMs too
prev story

Whitepapers

Securing web applications made simple and scalable
In this whitepaper learn how automated security testing can provide a simple and scalable way to protect your web applications.
3 Big data security analytics techniques
Applying these Big Data security analytics techniques can help you make your business safer by detecting attacks early, before significant damage is done.
The benefits of software based PBX
Why you should break free from your proprietary PBX and how to leverage your existing server hardware.
Top three mobile application threats
Learn about three of the top mobile application security threats facing businesses today and recommendations on how to mitigate the risk.
Combat fraud and increase customer satisfaction
Based on their experience using HP ArcSight Enterprise Security Manager for IT security operations, Finansbank moved to HP ArcSight ESM for fraud management.