Hot Chips The systems business is largely dominated by x86-based machinery these days, but Big Blue's mainframe unit is hanging in there after five decades and is still a bit of a mint.

You put in $1bn every two years for hardware development, as IBM has done with the new System zEnterprise EC12, and you take out $7bn in ridiculously profitable hardware sales over the next two years, and probably as much revenue from monthly systems software at even higher profit margins over those same two years.

You can laugh all you want about how stodgy mainframes are, but IBM CEO Ginni Rometty is laughing all the way to the bank, as are IBM's shareholders, who benefit from billions of dollars in dividends and share buybacks that are paid in part by the mainframe biz.

Rometty is probably breathing a sigh of relief as well, with the prior z11 mainframe cycle running out of gas as she took the helm in January. She will get to close out her first year running IBM with what will presumably be a big bump in revenues now that the "zNext" or z12 machines (depending on what code-name you want to use) are coming to market earlier than expected.

The word on the street was that IBM would announce the zEnterprise EC12, the high end of the z12 product line, sometime in late September or early October with shipments ramping through the fourth quarter. But, according to sources inside IBM, the chips and systems were ready early and so Big Blue decided to get the machines out the door on the same day that the company is talking up the feeds and speeds of the z12 processor at the Hot Chips conference in Silicon Valley.

IBM told El Reg about the zEnterprise EC12 mainframe ahead of the Hot Chips presentation and did not want to spoil the show by giving out all the gory details of the z12 chip. But Jeff Frey, an IBM Fellow and and the CTO for the System z division who designed the z11 and z12 systems, gave El Reg some of the basic stats of the chip and talked about the new system that is built around it.

The z12 chip is implemented in a 32 nanometer high-K metal gate process and fabbed by IBM itself at its East Fishkill, New York foundry. The z11 chips used a 45 nanometer process, and the shift to 32 nanometers has allowed IBM to get twice the transistor density for logic and on-chip cache memory.

The z12 chip was designed to scale up to 6GHz, which is roughly twice as fast as the clock speed on your average Xeon or Opteron processor, but is coming out at 5.5GHz initially. The z11 had four cores on a single die running at a top speed of 5.2GHz, but the process shrink is allowing IBM to boost the core count by 50 percent to six cores while also cranking up the clocks a bit.

IBM is doing other things in the z12 chip to boost single-threaded application performance as a batch engine, which is critical for customers. This includes boosting the sizes of on-chip caches and putting together a second generation out-of-order execution pipeline, which debuted with the z11 chips.

The L2 instruction and data caches have been boosted to 2MB, up from 1.5MB with the z11 chips, and L3 embedded DRAM cache is now 48MB, double that of the z11. On the outboard L4 cache controller and SMP hub that glues processors on a system board together (IBM calls them books) and multiple books to each other, the L4 cache has also been boosted 384MB, double that of the zEnterprise 196 machine announced two years ago.

The z12 chips also have more relative branch execution units to help speed up the threads, and mainframe compilers can now issue prefetch directives to the engines to help applications do a better job of getting instructions and data lined up to push more work through the z12 chip.

When you add it all up, the single-engine performance of a z12 chip is about 25 per cent higher than on the z11 that preceded it. IBM has not released official MIPS ratings (a throwback to the days when IBM actually counted the millions of instructions per second that a machine could process) for the z12 engines, but given that the top-end core in a z11 processor delivered 1,200 MIPS, that puts the z12 core at around 1,600 MIPS.

IBM's System zEC12 mainframe

"We are hitting the limits of physics in many cases," says Frey, "but what you see here with the zEnterprise EC12 is IBM not giving up on single-thread performance."

IBM has also added more processors to the System z complex and done other tricks to boost the SMP scalability with the zEnterprise EC12 behemoth.

Interestingly, the zEnterprise EC12 is the first IBM system to support transactional memory, which is an overlay atop standard DDR3 main memory that can reduce software locking across multiple execution units.

IBM calls it Transaction Execution Facility, and the idea is to have the processors and the SMP hub controller do "opportunistic locking" of main memory blocks instead of software locking as applications run. You do the transactions fast and then see if you needed to lock after the fact, and most of the time you don't.

If you did hit a locking condition, you go back and redo the work. The upshot is that the overall throughput of the machine can increase even if there is a huge stall every once in a while. We'll try to get more information on this at Hot Chips.

This transactional memory will help boost the performance on multithreaded applications where there can be contention, such as with DB2 databases and Java Virtual machines. Frey says that the combination of higher clocks, bigger caches, and transactional memory can boost multithreaded performance on an zEC12 box by as much as 45 per cent compared to a z196 machine.

Also helping DB2 and Java performance in the zEnterprise EC12 is support for 2GB page frames, which helps DB2 buffer pools and Java heaps take bigger bites of memory and do less swapping. Page frames topped out at 1MB with the prior zEnterprise 196 box.

Get your motor running

Like other chip makers, IBM cannot get every one of its chips to come out of the East Fishkill fab to work perfectly at all speeds and with all cores working. And as it has done with prior mainframes, IBM is using a mix of core counts in the z12 multichip module (MCM) processor complexes that package up multiple z12 chips into a block that is then welded to a zEnterprise EC12 book.

In this case, depending on the model and performance required, IBM uses a mix of z12 processors with four, five, or six active cores. A z12 MCM has five physical processors in it, for a total of 30 cores, and a zEnterprise EC12 system has four processor books, for a total of 120 cores maximum in a single system image.

IBM has these cores, which it calls processor units, or PUs, do a number of different tasks. They can be configured as Central Processors, or CPs, that are designated to run mainframe operating systems, including z/OS, z/VM, z/VSE, and z/TPF. By default, some are configured as System Assist Processors, or SAPs, and they are delegated to handling I/O processing so the CPs running workloads are not burdened with such work.

You can configure the cores as Internal Coupling Facilities, or ICFs, which run IBM's Parallel Sysplex system coupling software, which is used to lash together multiple physical mainframes in a shared cluster for making larger database and transaction processing complexes. (If you wanted to, you could turn a whole zEnterprise EC12 machine into a giant Parallel Sysplex switch, with 101 active CPs dedicated as ICFs.

These mainframe engines can also, of course, be designated as Integrated Facilities for Linux, of IFLs, to run Red Hat or SUSE Linux, or as z Application Acceleration Processors (zAAPs) for running Java or z Integrated Information Processors (zIIPs) for offloading DB2 database work. IBM also keeps one core in each machine in emergency reserve and has another two designated spares. This is one way the mainframe keeps its five nines of availability on a single system.

IBM charges for mainframe engines based on the work they will do, so a CP is the most expensive one since it will run z/OS, z/VM, z/VSE, and z/TPF. The IFLs are considerably less expensive (somewhere around a quarter of the price of CP) since IBM wants to have mainframe shops deploy Linux on their big iron rather than on x86 machinery.

zAAPs and zIIPs cannot run Java or DB2 routines any faster than CPs, but they offer the ability to offload Java and DB2 work from the CPs on engines that cost substantially less than a CP. This is how IBM has cut mainframe prices over time.

In effect, if you are running a mainframe operating system, your engine price is more or less frozen at levels set in the early 2000s. With no alternative to run your apps, the only way to cut costs is to keep moving to faster iron and use offload processing to zIIPs, zAAPs, and IFLs as much as possible.

There are five models of the System zEnterprise EC12 machine, as shown below:

System zEC12 mainframe models

Comparing the top-end models of the zEnterprise 196 and EC12 machines, the z12-based box has a total of 101 available CPs for customers to configure as they see fit, up from 80 on the z11-based machines. Both lines had five different models, but the new box has 161 unique capacity settings (set by clock speeds, activated cores, and other golden screwdriver tricks) compared to 125 different MIPS ratings for the z11-based systems.

The overall potential system capacity of the zEnterprise EC12 machine, as gauged in MIPS, is about 50 per cent greater than on the prior zEnterprise 196 server. That should work out to around 75,000 aggregate MIPS for the big bad zEC12 machine compared to 50,000 MIPS for the biggest z196 box from two years ago.

So, to recap, single engine performance is up by 25 per cent and system capacity is up by 50 per cent.

The one thing that has not changed between the two generations of System z mainframes is main memory capacity, which stands at a maximum of 3TB.

IBM is obviously still supporting its Redundant Arrays of Independent Memory (RAIM) data protection on that main memory. The RAIM function is implemented in the on-chip memory controllers and borrows the striping and parity protection ideas of RAID 5 disk arrays and adds it to memory.

RAIM made its debut two years ago on the zEnterprise 196 systems, and is another high availability feature that makes mainframes different from other systems and presumably worth a premium. All of that 3TB can be allocated to a single system image if you want, and a logical partition (LPAR) virtual machine on the zEC12 can span up to 1TB of virtual memory.

The new mainframes sport PCI Express 3.0 peripheral slots, the second machines on the market to do so (Intel was the first to support PCI Express 3.0 with its on-chip controllers in the Xeon E5 family launched in stages earlier this year.) The machines have up to 48 PCI Express 3.0 slots. They also support 6Gb/sec SAS links to internal disks.

As with the previous System z mainframes, the new zEnterprise EC12 box comes in variants that are water-cooled or air-cooled, with the water-cooled option being more efficient at taking away the prodigious heat from those high-clocking z12 engines and all of the components wrapped around them.

IBM is also now taking power and cooling into the machine from above if customers want to do that in addition to the networking wiring that it allowed with previous mainframes, thus allowing for the zEC12 machine to be put on a concrete slab floor instead of a raised floor like old-fashioned glass house.

The power and cooling profile of the zEC12 is the same as the z196, according to Frey, and the rack is essentially the same size but is a few inches deeper thanks to the funky new doors Big Blue has put on the rack.

Making mainframes safe and sound

Security, resiliency, and high availability are the three mantras of the IBM mainframe for the past several decades, and it shows.

The z12 processors have on-chip encryption, just as did the prior z11 processors. But the zEC12 machine is also getting a new outboard security coprocessor card, called the Crypto Express4S.

This coprocessor sports firmware that meets the European Union's PKCS#11 standard for encryption and also supports the EMV payment security standard espoused by American Express. This card is tamper resistant and has higher performance, but exactly how much was not clear ahead of its announcement today.

In most modern systems, all of the various hardware and system software components in the box are chattering all the time, streaming up alerts to system administrators. With a new capability called zAware, IBM is moving up into the software stack to help administrators figure out when mainframe applications are getting wonky - and where.

"We've done a great job diagnosing hardware failures, but zAware takes it up another level in the system," says Frey, adding that in most mainframe shops today, operators are intentionally suppressing about 85 percent of alerts because no human being can cope with the onslaught of data.

The zAware software resides partially in firmware and partially in its own logical partition and is based on a project code-named "Melody" out of IBM Research. zAware takes all of the operational and message traffic on the system, chews on it, and does pattern matching and other heuristics to figure out if a mainframe is running properly.

zAware samples all the operational data generated by the mainframe and its software stack ever two minutes and does a quick analysis over a 10 minute interval to see if anything peculiar is going on. It then compares this data to a 90 day rolling average to check it against a baseline for what the mainframe's behavior should be across the hardware and the software.

zAware is designed to look for anomalies operators might miss and is trained to find rare or infrequent messages as well as detecting an unusual number of normal messages or messages that are issued out of context that might indicate a problem. zAware is a priced feature on the mainframe, but no word yet as to what that price will be.

Still a system of systems

The mainframe has been pitched as a "system of systems" since the zEnterprise 196 launch two years ago, linking outboard blade servers based on Power7 and Xeon processors to the mainframe complex on a secure network that the outside world cannot see but mainframe operating systems and internal networking can so they can be used as coprocessors for those mainframes.

This is called a System z blade extension, or zBX for short, and the basic feeds and speeds have not changed here. There is a new zBX Model 003, however, which is required for the zEC12 machine; it has new firmware and updates to the supported hypervisors on the PS701 and HX5 blade servers plus some improvements to the link aggregation on the networking between the mainframe and the top-of-rack switches in the zBX racks.

IBM supports up to 112 PS701 blades and up to 56 HX5 blades in four zBX racks for offloading work to Windows, Linux, or AIX servers. Each rack can have up to two blade enclosures and has the necessary switches to link servers to each other and the mainframe as well as room for local storage for the Power and Xeon blades.

The zEnterprise EC12 mainframe supports the current z/OS V1.13 release as well as the earlier V1.12 release; if you pay for lifecycle extension services, IBM will let you run z/OS V1.10 or V1.11 on the box. You can run z/VM V5.4, V6.1, or V6.2 on the machines if you patch them with hardware updates, and ditto for z/VSE V4.3 and V5.1 as well as z/TPF V1.1.

Red Hat Enterprise Linux 5 and 6 and SUSE Linux Enterprise Server 10 and 11 can run on the IFLs. On the zBX blades, you can run AIX 5.3, 6.1, or 7.1 on the Power7-based blades. RHEL 5 and 6 and SLES 11 are supported on the Xeon blades as is Microsoft Windows Server 2008 R2 (with Datacenter Edition being recommended).

All models of the zEnterprise EC12 mainframe will be available on September 19, and it is likely that IBM will ship enough machines too goose what might have otherwise been a dismal third quarter for mainframes, and for servers in general with the Power7+ processors for its Power Systems boxes not yet launched.

IBM is offering upgrades from the prior System z10 Enterprise Class and zEnterprise 196 machines into the zEnterprise EC12 box; these upgrades will be available on September 19 as well. Model conversions and upgrades within the zEC12 family and upgrades will not be available until December, so pick your initial machine carefully.

IBM has been vague about mainframe pricing since its Consent Decree with the US government was rescinded more than a decade ago, and all IBM is saying today is that the entry price of the zEC12 is consistent with the entry price of the z196 it replaces. That doesn't really tell anyone much. We'll try to get a better handle on pricing for a future story. ®