IBM is rejiggering its hybrid mainframe-blade "system of systems" that was announced last July and that began shipping in the fall.

Big Blue is also providing details of the hypervisors, networks, and operating systems that will be supported on the hybrid configuration, which marries a System zEnterprise 196 mainframe with blade chassis equipped with Power and Xeon processors.

The zEnterprise 196 mainframe is based on the company's z11 processor, which has four CISC cores running at 5.2GHz implemented in a 45 nanometer copper/silicon-on-insulator process, and which weighs in at 1.4 billion transistors including L1, L2, and L2 caches; integer, floating point, and decimal units; and a slew of other features. (You can see El Reg's in-depth coverage of the z11 chip here and a discussion of the zEnterprise 196 system there.)

Six of the four-core processors are implemented on a System zEnterprise 196 system board, which IBM calls a processor book. Up to four books and 3TB of DDR3 main memory (including a RAID-like memory striping that Big Blue calls RAIM) can be added to a system for a total of around 50,000 aggregate usable MIPS.

The system can have up to 80 cores dedicated to running z/OS, z/VM, z/VSE, and z/TPF workloads, with the remaining engines used as co-processors to handle I/O and to offload Java, database, or Linux processing. If customers want more co-processors for running Linux, they can use some of the 80 engines that way, too, but specialty engines come out of the pool of 16 engines.

IBM engineers set up a zBX blade chassis for a System zEnterprise 196 mainframe.

In an effort to push HP, Dell, Oracle, and Fujitsu x64 and RISC/Unix servers out of data centers that have IBM mainframes at their core, Big Blue took a page out of its AS/400 playbook (and one that Unisys also uses to create hybrid ClearPath mainframes) and has coupled Power and Xeon blade servers with the zEnterprise 196 mainframes.

By doing so, the IT executives in control of mainframes can make a land grab and try to wrest control of x64 and RISC iron from the executives in charge of those silos. The idea is to keep the mainframe at the center of the data center and to extend the mainframe's legendary security and good (but arcane) systems management to these other platforms. But to get the benefits of this hybrid architecture, you have to use IBM Power or Xeon servers.

When the System zEnterprise 196 machine and its zBX blades were announced last July, IBM had two different models of the zBX. The zBX Model 001 was a special version of the zBX that allowed the device to attach to older System z10 mainframes, while the zBX model 002 had top-of-rack networking and integrated switching to link the newer zEnterprise 196 to the blade servers in the zBX complex.

The zBX put two chassis in a rack, and four racks could be linked to a single zEnterprise 196 for a total of 112 blade servers to act as co-processors for the mainframe. A zEnterprise "ensemble" is a logical cluster of machines that has up to eight of these zEnterprise-zBX nodes. The BladeCenter chassis can, by the way, also be equipped with IBM's Smart Analytics Optimizer and DataPower appliances.

The first big change, which IBM quietly announced last October, is that the zBX Model 001 has gone the way of all flesh. So if you want to create a hybrid mainframe-Power-Xeon cluster and manage it all from Big Blue's Unified Resource Manager software, you need to upgrade to the newest mainframe – which not much of a surprise, when you think about it.

The second big change that's coming with the zBX, according to Doris Conti, director of System z marketing at IBM, is that Microsoft's Windows operating system will be supported on the Xeon blade servers inside the zBX complex. IBM has hosted over 300 workshops with mainframe customers discussing the new hybrid system, and customers were not exactly happy that IBM was restricting Linux to Xeon blades and not supporting Windows.

"We heard the feedback and we very much intend to deliver Windows support on zBX," says Conti.

Since last November, IBM has been shipping the zBX Model 002 chassis with entry-level PS701 blade servers – single-socket blades with four-core Power7 chips on them. IBM has not been precise about when it would ship Xeon-based blades, and only committed last summer to putting Linux on those Xeon blades sometime in the first half of 2011.

You may be wondering why Windows and Linux support on the Xeon blades in the zBX didn't ship back in November with the Power-AIX blades. Jeffrey Frey, the IBM Fellow and System z architect who designed the zEnterprise 196-zBX hybrid, says that the Xeon blades are coming later because IBM's Power-AIX customers were the ones Big Blue felt would take to the hybrid computing model first. (IBM is also fixated on preserving its market share in the Unix racket against resurgent Oracle and HP.) The plan now is to get Linux support on Xeon blades out the door this year, and then add Windows support as soon as possible.

Frey said that IBM was not sure how deeply it would have to get into the operating system or hypervisor code to manage AIX, Linux, or Windows when it started the zBX. So AIX, which IBM has the source code for, and Linux, which is open source, were the easiest places to start. IBM didn't want to get involved with Windows until it knew what it might need from Microsoft in the way of cooperation. "As it turns out, there is very little of that," Frey explained to El Reg, referring to the need to get into Windows code to make the OS work on the hybrid system.

Frey also let the cat out of the bag on what hypervisor IBM is using on the blades. IBM's own Processor Resource/System Manage (PR/SM) type 1 hypervisor and its related z/VM operating system (which can function as a type 2 hypervisor) are used to dice and slice the zEnterprise 196. The company's own PowerVM hypervisor is used on the Power7 blades to carve them up into logical slices and to virtualize I/O on the blades. IBM has chosen a variant of Red Hat's Enterprise Virtualization (RHEV), the commercial-grade implementation of the KVM hypervisor for x64 iron, for the Xeon blades; this tweaked version is known as RHEV-Blue, predictably, and is made to cooperate with IBM's mainframe firmware. Power VM will support AIX 5.3, 6.1, and 7.1, and Conti says that if mainframe shops want to run the IBM i 7.1 operating system (formerly known as OS/400) on Power blades, IBM will consider it, says Conti. As for Linux on Power, Frey says there will be a need for it, but that Windows on Xeon blades is more important to get to market given the installed base of machines at mainframe shops.

It's the network, stupid

It is pretty easy to draw some pretty pictures and to network blade servers to rack servers and put some switches between them to link them together. Data centers have been doing this for decades, after all.

The secret sauce to the zEnterprise 196-zBX hybrid is not just networking, but the way that it is networked, says Frey. With the setup IBM has created, the hypervisors on the Power and Xeon blades are treated just like system firmware on the mainframe, just like the PR/SM hypervisor. So PR/SM, PowerVM, and RHEV are all treated like firmware on the mainframe and are all linked back to the mainframe and to the Unified Resource Manager tool on the mainframe by a point-to-point Gigabit Ethernet network that's implemented in a switch that has long since been buried in the bulk power hub in the System z mainframe.

This switch hooks into the Advanced Management Module in the BladeCenter chassis, and the Unified Resource Management tool uses SNMP interfaces to manage the BladeCenter hardware and has hooks into the PowerVM and KVM hypervisors to manage virtual machine partitions on the Power and Xeon blades.

Here's what the management networks look like schematically:

As the schematic shows, this internal network is used to manage the deployment, performance monitoring, availability, security, and energy management aspects of the virtualized servers in the hybrid box.

Here's a more detailed view of the internal and external networks in the zEnterprise-zBX ensemble:

Network B in the picture is the Intra-Node Management Network, and it has redundant switches from Blade Network Technologies (recently acquired by IBM) in the BladeCenter chassis that link back into the Ethernet switch in the System z frame. This network does not have any encryption on it because it is not connected to the outside world.

In the above chart, FSP is short for field service processor, which implements the Network A, the Private System Control Network. The SE in the chart is short for Support Element, and it is a mirrored set of laptops that lets mainframe admins get access to the mainframe firmware. The HMC is short for Hardware Management Controller, and this is used to manage multiple mainframe systems. (In the Power line, the HMC manages PowerVM hypervisors, and this is not the same device despite having the same name.)

The old mainframe HMCs were essentially stateless, according to Frey, but with the zEnterprise-zBX hybrid the HMCs are more stateful, having knowledge about the conditions on the nodes and ensembles in the clusters. In fact, the Unified Resource Manager software runs piecemeal on the HMC, the SE, and in other elements like the AMM and hypervisors.

Network C in the schematic is a customer-accessible network called the Intra-Ensemble Data Network, a flat Layer 2 network that links all of the virtualized servers in the mainframe, Power, and Xeon servers together, and is based on top-of-rack 10 Gigabit Ethernet switches from Juniper Networks. This network does not have any Fibre Channel over Ethernet (FCoE) support for converging server and storage networks, but IBM is prepping such support for the next generation of the System z-BladeCenter hybrid.

Network D is an external management network that links the HMC to the SEs in multiple System z mainframes in an ensemble, and Network E links the server slices to the outside world where they can get orders to process work.

Now, I know what you are thinking: 'Why didn't IBM use InfiniBand instead 10 GE switches for the Intra-Ensemble Data Network and external network?' InfiniBand has lower latency, higher bandwidth, and also offers Remote Direct Memory Access (RDMA) support, which is where the low latency comes from. RDMA lets a server reach out over InfiniBand and go directly into the main memory of another server node linked to it by InfiniBand, rather than having to pass requests for data up through the TCP/IP and OS stacks, over the network, down into another machine's stack and back again.

Frey was a bit cagey about this, but said that using InfiniBand for the C and E networks would not have obviated the need for 10 Gigabit Ethernet, so it was just easier to pick one network and ride it. And with 10 Gigabit Ethernet supporting the RDMA over Converged Ethernet (RoCE) protocol, which was announced in April 2010, IBM can do RDMA in a future release of the product, should it decide to.

Frey would not comment on future plans, but did say that IBM was interested in RDMA, not InfiniBand, per se. (IBM's PureScale Power-DB2 database cluster uses RDMA with Voltaire InfiniBand switches to tightly couple database nodes together and let them quickly share data.)

Frey did confirm that a future zEnterprise-zBX system would have link aggregation, allowing for more than 10GB of bandwidth to be pumped down into any single blade. He added that IBM was also working on ways to converge storage into the cluster and do a better job of virtualizing that storage, possibly including adding IBM's SAN Volume Controller for thin provisioning, copy services, and other features. Blades will also be able to link to mainframe file systems in the next release.

IBM has not said how many System zEnterprise 196 customers it has, and it has not said how many have opted for the zBX hybrid. Conti reminded us that insurance giant Swiss Re was the first buyer of the new mainframe, buying two fully loaded zEnterprise 196 machines last September, each with 20,590 MIPS dedicated to z/OS workloads, ten Linux engines, and extra specialty engines for boosting Java performance, called System z Application Assist Processors, or zAAPs. Swiss Re is also going to implement the zBX blades at some point.

Conti said that IBM had "a bunch" of zBX shipments in the fourth quarter, without being precise about the number,and added that Big Blue expected the zBX sale to take a while. Most customers will upgrade to the new mainframes first, then consider the benefits of integrated blade servers being managed by the mainframe. ®