Original URL: http://www.theregister.co.uk/2010/04/01/ibm_xeon_7500_servers/

IBM goes elephant with Nehalem-EX iron

Massive memory for racks and blades

By Timothy Prickett Morgan

Posted in Servers, 1st April 2010 16:53 GMT

With so much of its money and profits coming from big Power and mainframe servers, you can bet that IBM is not exactly enthusiastic about the advent of the eight-core "Nehalem-EX" Xeon 7500 processors from Intel and their ability to link up to eight sockets together in a single system image. But IBM can't let other server makers own this space either, so it had to make some tough choices.

While RISC, Itanium, and mainframe servers still have plenty of RAS advantages, the scalability advantages are becoming less of a factor, and that means all big Opteron and Xeon servers will put both direct and indirect economic and technical pressure on all IT shops that currently use proprietary midrange and mainframe machines as well as RISC/Itanium servers. But IBM has, at least at first, put Intel's Xeon 7500s and their crimped HPC variants, the Xeon 6500s, to use in a way that minimizes the damage to its high-end server products.

Big Blue is using its own eX5 chipset, which no doubt can scale to 16 sockets and maybe more, to relegate the Nehalem-EX processors to two-socket and four-socket servers where customers will pay a premium for fat main memories rather than building bigger boxes from the chips. That gives the new four-socket Power 750 and eight-socket Power 770 and 780 servers announced in early February using IBM's eight-core Power7 chips a chance to sell.

It also gives IBM something interesting to sell against two-socket servers based on the new six-core "Westmere-EP" Xeon 5600s from Intel and the twelve-core "Magny-Cours" Opteron 6100s from Advanced Micro Devices from rivals Hewlett-Packard, Dell, and Fujitsu. The Xeon 5600s took the field in the x64 server war two weeks ago and the Opteron 6100s rolled out to face them and the Xeon 7500s earlier this week.

BM dropped the Xeon 5600s in its existing BladeCenter blades and System x racks and towers that support the Xeon 5500s, since this is just a processor upgrade. Big Blue has not said squat about its plans for the Opteron 6100s, which have a new socket and a new chipset and are therefore not compatible with IBM's existing Opteron products.

IBM lifted the veil on the ex5 chipset a bit during the CeBIT trade show in Germany back in early March, getting the kind of preferential treatment that Intel reserved for server wannabe Cisco Systems ahead of the Nehalem-EP Xeon 5500 launch a year ago. The server racket is a bit like the third grade, with everyone wanting to be the teacher's pet, but only Steve Jobs can give teacher an Apple.

This week, IBM rolled out the specs of the three initial machines making use of the eX5 chipset and Intel's Xeon 6500 and 7500 processors. We now know that these machines, which didn't have names a month ago, are the BladeCenter HX5 blade server and the System x3850 X5 and x3950 X5 rack servers.

The BladeCenter HX5 is just like IBM's other blade servers in terms of its form factor. All of IBM's blade servers have been the same full 7U height since it shipped its original blades back in 2002, and the basic chassis still holds 14 single-wide (30mm) blades. The HX5 is a two-socket, single-width blade server and taking a trick out of the playbook that IBM created originally for its Opteron-based LS22 and LS42 blades, you can snap two HX5 blade servers back-to-back and turn them into a four-socket blade.

You can't do this with the Xeon 5600s or earlier Opterons in the 2000 series, and you won't be able to do this with the future Opteron 4100s either. But the Xeon 6500 and 7500 chips and the Opteron 8000 and 6100 series have built-in electronics that allows for this SMP expansion. This is now called a FlexNode upgrade.

IBM eX5 Nehalem-EX Servers

IBM's eX5 family shot: two linked BladeCenter HX5 blades on top, the prodigal System x3690 X5 in the middle, and the shipping System x3850 X5 on the bottom

The HX5 blade cannot support the top-end eight-core Xeon 7500 parts, which have a 130 watt thermal design point, but it has been certified to support the eight-core L7555, which runs at 1.86 GHz, has 24 MB of L3 cache, and is rated at 95 watts. For the same 95 watts, IBM also says customers can use the four-core E7520, which runs at 1.86 GHz as well, has 18 MB of cache, and is considerably less expensive.

If you can handle 105 watts per socket for your HX5 blade, then you can have the four-core E6510 running at 1.73 GHz with 12 MB of cache; the six-core E7530, running at 1.86 GHz with 12 MB of cache; or the six-core E7540, running at 2 GHz with 18 MB of cache.

Memory and more memory

The HX5 blade has room for two 50 GB solid state disks (one comes standard), and 16 DDR3 memory slots and IBM is supporting very low profile memory due to the skinniness of the blades, but is only selling standard voltage memory. (Maybe the Xeon 7500s do not have support for low-voltage memory like the Xeon 5600s just got? That probably comes with the Xeon 7600s).

IBM is supporting 4 GB and 8 GB memory sticks in the machine, which means memory tops out at 128 GB per two-socket blade. In its preview, IBM said that it had a memory blade add-on called Max5 that would allow an extra 24 memory slots for a two-socket blade as well, which adds another 192 GB of memory potentially.

IBM said back then than only one Max5 memory extender would be permitted on either a two-socket or four-socket blade, but this turns out to be not true. You can put one Max5 on each two-socket blade and then plug all four blades together into a big SMP server with four sockets and 80 memory slots, for a maximum of 640 GB of memory. All in about 2.5U of equivalent rack space.

Some HPC shops are going to love this. Four of these blades could be the new fat node in a cluster. Unless they get a look at the System x3850 X5 and x3950 X5 4U rack-mounted machines, or the System x3690 X5, a 2U rack server that was not announced yet but IBM said was in the works back in March.

The System x3850 X5 machine is the basic four-socket server node that has been at the heart of IBM's high-end x64 range for many years. IBM's homegrown eX chipsets allowed for one, two, four, and sometimes eight of these nodes to be linked together into a NUMA system. The x3850 X5 will eventually scale from one to four chassis, which is a 16-socket box potentially using top-end eight-core parts, for a maximum of 128 cores.

For now, it looks like IBM is only selling single-node x3850 X5 machines, with the two-node scalability kit coming on June 15 that turns it into a System x3950 X5. The Max5 memory extender for this box, which IBM was talking about back at CeBIT, is not yet available either. And there's no word on when the three- and four-node scalability kits for larger x3950 X5 machines will arrive.

The x3850 X5 has four processor sockets and IBM is supporting all eight variants of the Xeon 7500s in the machine, from the six-core, low-voltage 1.86 GHz L7545 all the way up to the eight-core, 130 watt 2.26 GHz X7560. IBM warns that the E7520 and E7530 parts, which are four- and six-core parts, cannot be used in eight-socket and larger configurations. IBM is supporting 1 GB and 2 GB DDR3 memory sticks running at 1.33 GHz and 4 GB and 8 GB sticks running at 1.07 GHz. IBM is also one of the first vendors out the door with 16 GB memory sticks, which the x3850 X5 supports as well.

This memory is plunked onto system memory cards with Intel's Scalable Memory Buffers, which bear some resemblance to the buggered memory cards IBM has used in Power Systems for years. With 16 GB DIMMs, which no one can afford right now, IBM can cram 1 TB of memory into a 4U chassis or 2 TB in an eight-socket box in two 4U chassis. The Max5 memory extender will add another 32 memory slots to the box, meaning customers can use less capacious and less expensive 4 GB and 8 GB memory sticks to put a still hefty 384 GB or 768 GB of main memory on a four-socket or eight-socket box, respectively. (That's provided IBM allows one Max5 memory expansion unit for each node).

The x3850 X5 has five PCI-Express 2.0 slots (one x16, five x8, and one x4), and redundant 1,975 watt power supplies. It has eight hot-swap, 2.5-inch SAS disk bays that support disks of up to 500 GB of capacity. If you want to go SSD, then you can jam 16 of the 50 GB SSDs into the bays.

IBM has certified Microsoft's Windows Server 2008, Red Hat's Enterprise Linux 5, Novell's SUSE Linux Enterprise 11 operating systems on the Nehalem-EX iron, with VMware's ESX Server 4.0 and ESXi 4.0 hypervisors also being certified on the machines.

In a base configuration with a single four-core 1.86 GHz E7520 processor and 8 GB of memory, the BladeCenter HX5 blade costs $4,629. With two of the six-core 2 GHz E7540 processors and 64 GB of memory, the HX5 costs $15,095. In a base configuration with two of the quad-core 1.86 GHz E7520 processors and 4 GB of main memory, the x3850 X5 costs $9,049 (there's no disks in that configuration). A machine with two of the eight-core 2.26 GHz X7560 chips and 16 GB of memory jacks the price up to $21,705.

In many ways, the two-socket x3690 X5 machine, which has not been formally announced yet, is more interesting. Like the HX5 and the x3850 X5, the x3690 X5 offers the FlexNode upgrade capability, so the 2U box can have another one slapped next to it and use IBM's eX5 chipset to convert it to a four-way server.

The x3690 will have 32 of its own memory slots, and another 32 can be added using the Max5 memory expansion module, for a stunning 1 TB of memory (using 16 GB DIMMs) for a two-socket box that tops out at 16 cores. Even with 8 GB memory sticks, the x3690 X5 can wield 512 GB of memory, which again makes it a perfect fat node for certain supercomputing workloads. The x3690 X5 may come in a 2U form factor, but it has enough room for 16 2.5-inch, front-mounted disks. ®