SoftLayer sticks with Super Micro servers, ponders Open Compute
It's all about the supply chain - THEN the price
Google, Yahoo!, Facebook, Amazon, and Microsoft are in a class all by themselves when it comes to the size and scope of their data center infrastructure, but hosting and cloud provider SoftLayer is no slouch, either. The company has created its own control freak and is not about to reflexively jump on the OpenStack or Open Compute bandwagons as cross-state rival Rackspace Hosting has done.
With over 100,000 servers under management in 13 data centers, SoftLayer has one of the largest IT operations in the world. And it has enough data center capacity to handle even more machines than that. The company, which was founded in 2005, merged with rival The Planet to embiggen itself and is now privately held by private equity giant GI Partners.
One of the things that the homegrown Infrastructure Management System (IMS) control freak does well - which other cloud controllers do not as yet - is provision bare metal machines as well as hypervisor-based cloudy server slices. And the company has come up with its own methods of plotting IT infrastructure so it can roll out chunks of processing capacity in 5,000-server increments.
SoftLayer has some of its own facilities and rents space from Digital Realty Trust, one of the big co-location data center providers. It just opened up its data center for the Asia/pacific region in Singapore in the fall of 2011, and already has added 500 customers to this facility in the past six months.
SoftLayer has over 100,000 servers and has capacity to do a lot more
In March, SoftLayer was boasting that the global reach of its data centers has made its infrastructure popular with online gaming companies, who want to get the iron closer to the end users to reduce latencies. The company's servers support hundreds of mobile, PC, and social games with an aggregate of more than 100 million users.
In March 2011, SoftLayer had more than 80,000 active servers across its data centers, and chief scientist Nathan Day tells El Reg that the hosting and cloud provider added 19,000 shiny new servers in 2011 and another 21,000 in 2012. All of this is very good news for Super Micro, which has been the server supplier of choice for SoftLayer since the company was founded.
"When we are trying to pick our server hardware, we've got to try to balance two competing factors," explains Day. "One is maintaining a variety of configurations for our customers because we have a seemingly infinite variety of workloads running in the SoftLayer data centers.
"We don't have the luxury of some other hyperscale data center operators that have a set number and mix of workloads, a number you can count on one or two hands. We have to have a huge amount of customization available. It is a balancing act we have been doing for the past seven years and it is what has driven most of our decisions."
Sam Fleitman, chief operations officer at the hosting and cloud provider, says that there was never an option of getting customized hardware when SoftLayer began building its infrastructure. So the company did what you and I would probably do in the same situation.
"We looked at the Super Micro catalog and bought what we thought we could sell," explains Fleitman. "In the past seven years, as we have evolved, we have been able to shore up our relationship with Super Micro with qualifications and then tell them what we want to purchase in bulk.
"They know what our purchasing patterns are, they know what configurations have been purchased, and then they show us the best motherboard and system configuration options and we turn that into our standardized products."
In the beginning, SoftLayer had a mix of servers based on Intel Xeon and Advanced Micro Devices Opteron processors, reflecting the preferences of customers years ago. But according to Fleitman, and over time, as demand for AMD machines has shrunk, the older Opteron boxes have been removed from the SoftLayer fleet. "Not because SoftLayer didn't like AMD systems – we are a big fan – but because our customers stopped asking for them."
There are still some SoftLayer customers who want AMD boxes, and Super Micro is happy to take orders for them, and both Super Micro and SoftLayer would like some more intense competition from the two major x86 server chip makers.
Over time, as memory-heavy and virtualization workloads have taken off, SoftLayer has had to add four-processor x86 servers to its fleet in addition to the single-socket and two-socket machines. To accommodate different memory, disk, and peripheral capacities (including GPU coprocessors in some of its data centers announced a year ago), the company uses 1U, 2U, and 4U enclosures from Super Micro.
And, for certain workloads, SoftLayer installs the Twin2 hyperscale multi-node machines or the Micro Cloud tray servers instead of traditional rackers. "On the back-end, we try to limit the SKUs as much as possible to simplify inventory control," says Fleitman. "We standardize on a small number of motherboards and chassis." Super Micro manufactures the boxes, and SoftLayer engineers call in tweaks to the local factories (in California, Amsterdam, or Taiwan, depending on the data center's location).
SoftLayer and Super Micro have been at this for a number of years, and they have the server capacity planning and manufacturing down to a science. "We have enough data to predict what our demand and inventory is going to be like," says Fleitman. "On top of that, we take shipment on a monthly or quarterly basis, depending on need. And because of our runbooks, we are able to take inventory very quickly from our supplier and get it operational."
It also helps that SoftLayer is buying servers that Super Micro already makes, stocks, and sells and little bumps in demand can be satisfied quickly. With custom boxes, Super Micro might have to spin up a custom assembly line, and that takes both time and money.
We don't make techies take vintage boxen home
SoftLayer has a target lifespan of four years for a server, and it sits in the same data center, in the same spot in the rack, for that lifetime. If a customer wants to keep a server and not have to reload their software on a new box after that four-year term, SoftLayer is fine with that. And as such, the average lifespan of a server is around six years right now. Eventually, when a data center is retrofitted, a vintage box is targeted for replacement and SoftLayer works with a number used equipment dealers to get rid of boxes. And no: Day and Fleitman don't make programmers and techies take home old machines to skirt around this process.
For performance reasons – given the customer base of seismic processing, media, gaming, and other workloads that are not friendly to hypervisors – a lot of customers at SoftLayer still like bare metal machines.
"We have customers who want a single OS on bare metal, usually a flavor of Linux or Windows," explains Day. "We have other customers who want the bare metal experience, but they want a hypervisor on board and they want to manage their own virtualization. And then we have our own virtual machine cloud, where we are managing the hardware and the hypervisor and the customer is just obtaining virtual machines from us. Without revealing exact percentages, the split is in the above order, with the largest number of customers wanting bare metal and the smallest wanting cloud."
Don't expect for SoftLayer to get out on the front end of the move towards ARM servers, even though the company is keeping a keen eye on it.
"We are really looking to react to customer demand on this one," says Day. "What we think is that it will be a price per unit of compute decision for customers. The market may respond to inexpensive ARM-based servers, and we think it is going to be a pricing decision. There has been some window shopping, but there has not been any serious deals on the table.
"We are not seeing the demand yet at all. We have seen the server industry come out with ARM servers that would work perfectly within a SoftLayer data center – we can rack 'em and stack 'em just like we can any other server – but we really need to wait on customer demand for those. We don't have the luxury of just slipping in an ARM-based server and then not telling the customer."
Although that might be a funny test for someone to pull. Like, Google or Facebook, for instance. It will be funny to see if any of us can tell when it happens – or if it has already happened.
Facebook's DIY data center
Just as SoftLayer is keeping an eye on the ARM server front, it is also watching the Facebook-sponsored Open Compute Project, which has open sourced server, rack, power distribution, and data center designs. Rival Rackspace Hosting has decided to go with modified Open Compute machinery in its data centers, and says that this, as well as creating and using the open source OpenStack cloud controller, is the edge that will be needed to compete against Google and Microsoft in the heavenly infrastructure biz.
SoftLayer agrees there can be advantages to the Open Compute approach.
"We definitely like the way that Open Compute is going, and as data center operators, we definitely like anything that uses less power," says Day. "Right now we see some risk in Open Compute relating to the supply chain, because you are going with a fairly customized server instead of being able to get servers through channels and many different distributors and manufacturers.
"We are flexible enough that if we had to switch to Dell or Hewlett-Packard tomorrow, they would fit fine inside of our operation in terms of the physical aspects of the machines and managing them. We haven't rolled Open Compute out, so I don't want to speak too much about what we might do in the future, but there is certainly a step function both in operations and data center layout when it comes to adopting Open Compute."
The far more likely possibility is that Super Micro adopts some Open Compute designs and SoftLayer snaps these up where appropriate. ®