Original URL: http://www.theregister.co.uk/2010/03/02/ibm_ex5_nehalem_ex_preview/

IBM sneak peeks Nehalem EX iron

Goes modular with homegrown chipset

By Timothy Prickett Morgan

Posted in HPC, 2nd March 2010 05:26 GMT

At the CeBIT monster IT trade show in Hannover, Germany today, IBM will preview its forthcoming System x and BladeCenter servers based on the eight-core "Nehalem-EX" Xeon processors. While the Nehalem-EX chips have been pitched for the upper end of the x64 range, IBM is taking a different approach with its initial Nehalem-EX boxes and building modular rack and blade boxes with a fairly modest socket count.

With the Nehalem-EX chips expected before the end of this month - and their six-core baby brothers, the "Westmere-EP" chips - also due around the middle of the month, Big Blue is trying to get a jump on the PR gun like Cisco Systems did two weeks ahead of the quad-core Nehalem-EP Xeon 5500 launch last March. While IBM no doubt will have machines that support both processors, as will all x64 server makers who matter, the Westmere-EPs (which will be called the Xeon 5600s) are less of a big deal in that they snap right into existing two-socket Xeon 5500 platforms.

The big jump architecturally for two-socket machines will come with the "Sandy Bridge" Xeons in 2011. These will require a revamping of sockets, chipsets, I/O, and a slew of related technologies.

With the Nehalem-EX processors, the moment of big changes is now, since these processors will finally bring QuickPath Interconnect to Xeon boxes with four or more processor sockets. And as IBM is doing as it deploys the Nehalem-EX chips using its homegrown eX5 chipset, these big chips are also going to be deployed in smaller blade and rack boxes with some unique memory and processing expansion that is not possible using the Xeon 5500s or 5600s.

Silicon Graphics is taking a similar approach with its Altix UV shared memory blade-based supercomputer, which implements Nehalem-EX processors on two-socket blades and takes the unused connectivity ports on the chip that would be used to make four-socket SMP boards to hook the blades into SGI's own NUMAlink 5 supercomputer interconnect. As previously reported, Bull is working on a chipset called Fame D to create four-socket Nehalem-EX system boards for a future server design called Mesca, and like the current IBM eX4 and Power chipsets used in the Power 570 class of machines, it allows for up to four chassis to be lashed together in a single, cache coherent, shared memory, SMP box.

(IBM got the goodies to glue multiple nodes together, be they based on Xeon, Itanium, or Power chips, into an SMP setup thanks to its $810m acquisition of Sequent Computer Systems back in the summer of 1999. The eX5 chipset is the fifth generation of Sequent-inspired chips).

According to Tom Bradicich, vice president of systems technology at IBM's System and Technology Group, rather than start with big boxes with lots of sockets, memory, and I/O, IBM wanted to rethink the "PC server" with the Nehalem-EX processors. To that end, the eX5 chipset is going to not just allow for scalability at the socket level for processors, but do so in a more granular way.

Perhaps more importantly, given that servers are more memory bound than CPU bound these days, IBM is going to use the eX5 chipset and the considerably larger memory bandwidth and capacity of the Nehalem-EX chips to create two-socket and four-socket servers that can have additional memory slots - and lots of them - to be added to them without having to add sockets. Memory and sockets on the eX5 machines can scale independently, or together.

IBM boasts that the System x and BladeCenter machines that will come out this year are the fruits of a five-year, $800m effort. When they are launched later this month, it says that the eX5 systems will have six times the main memory of other x64-based servers, slash storage costs by 97 per cent (presumably through the option of flash memory), have 30 times the database performance (again, a mix of lots of cores, main memory, and flash disk), and 90 per cent better performance per watt.

While Bradicich was not at liberty to divulge all the details of the forthcoming Nehalem-EX and Westmere-EP machines, since Intel has not announced the processors yet, he could go through some of the features of the eX5 chipset and systems that Big Blue believes will help the company set itself off from the pack of x64 server makers. "We will offer the opportunity for companies to buy less IT equipment," he said.

That's something that perhaps only an IBM Fellow like Bradicich can get away with saying, since it implies that IBM won't make as much money. But IBM wants to make money dealing with x64 server sprawl, says Bradicich, and that means "letting customers only buy what they need and getting them to use what they buy."

The counterpunch

The eX5 chipset has a counterpunch to the memory expansion capability of selected models of the "California" Unified Computing System B-Series blade servers from Cisco and their C-Series rack variants. Cisco's own memory expansion ASIC works in conjunction with the on-chip memory controllers of the Xeon 5500 processors and the related Intel chipsets to allow a rack or blade to support a maximum of 384 GB instead of the 144 GB limit set by the Intel electronics. (That's a factor of 2.7 times the memory expansion).

IBM has come up with its own memory expansion technology for the Nehalem-EX boxes and the eX5 chipset, called Max5. With this feature of the chipset, an add-on chassis that just has DDR3 memory DIMMs can be linked to a rack server. Ora similar blade form factor with extra memory slots can be linked to a blade server in much the same way that multiple nodes in high-end System x and Power systems can be lashed together to make SMPs. These memory expansion units are basically server motherboards with no server sockets, and the memory is available over fast QPI links to the servers they are linked to.

IBM is planning to launch two-socket and four-socket Nehalem-EX rack servers and a two-socket blade. The names, feeds, and speeds of these machines were not divulged. But Bradicich did say that for the rack servers, the Max5 memory expansion feature would come in a 1U box with 32 DIMM slots and the Max5 blade expansion feature would have 24 DIMM slots. The two-socket rack machine will have 32 native memory slots, the two socket blade server will have 16 native slots, and the four-socket socket rack machine will have 64 slots.

So you are talking about 256 GB of relatively cheap memory on a two socket blade server (using 4 GB DIMMs that cost maybe $250) and even 512 GB for somewhat pricey 8 GB DIMMs (which sell for about $1,000). Of course, heaven only knows what IBM will want to charge for the Max5 features and the cable to link them into the SMP cluster. As for capacity, IBM can put 768 GB into a four-socket box using 8 GB DIMMs, and 1.5 TB if customers want to go crazy with 16 GB DIMMs. "We think this is more than enough for this machine," Bradicich says with a laugh.

The eX5 chipset and server design is also more granular than the prior eX4 products. With these, IBM had Xeon 7400 processors and four sockets and their main memory were the basic building blocks of a machine that scaled up to 16 sockets. With the eX5, IBM is using a two-socket or four-socket mobo as the basic building block.

For now, all IBM says for sure is that the chipset will allow for two two-socket rack or blade servers that are matched in terms of processors and form factors to be linked together as an SMP using what it calls a FlexNode feature. IBM has been offering two-way to four-way upgrades in its LS23/LS43 Opteron-based blade servers (using the HyperTransport bus) for two years now. This is similar, but SMP is not just limited to blades. IBM is also allowing two four-socket rack servers to be linked together into an eight-way.

Initially, it looks like the FlexNode feature is just supported with two machines, and only one Max5 memory expansion unit can be plugged into a FlexNode setup. The interesting bit is that IBM's Systems Director software will apparently allow for a FlexNode configuration to be merged and broken on the fly, as workloads dictate. IBM's System x box was the star of the Nehalem-EX preview from Intel last May.

As for the scalability for processor sockets and memory in the eX5 chipset, Bradicich would not say anything about what larger Nehalem-EX machines Big Blue is cooking up. But the architecture almost certainly scales to 16 sockets, as the previous ones did, and maybe a bit further unless IBM is afraid of compete with its Power7 big iron due later this year, which will scale to 32 sockets using those eight-core Power7 chips. What Bradicich would confirm is that the eX5 chipset would not support Advanced Micro Devices' Opteron processors and did not want to talk about IBM's plans for supporting the future "Magny-Cours" 12-core Opteron 6100s, also due out this month. ®