Original URL: http://www.theregister.co.uk/2012/10/01/fujitsu_oracle_athena_chip_server/

Fujitsu, Oracle pair up on future 'Athena' Sparc64 chips

But Larry may not resell servers that use them

By Timothy Prickett Morgan

Posted in Servers, 1st October 2012 20:55 GMT

OpenWorld 2012 During Sunday's opening keynote at Oracle's OpenWorld 2012 San Francisco shindig, longtime server and operating system partner Fujitsu talked about a future "Athena" Sparc64 processor and related systems using it that will run Oracle's Solaris Unix variant and sport special instructions for speeding up Oracle databases.

Fujitsu senior vice president Noriyuki Toyoki never came out and said that Oracle would rebrand the Athena systems under development by Fujitsu, as Oracle currently does with the "Jupiter" Sparc64-VII+ systems – machines based on those processors are sold under the Sparc Enterprise M family by both Oracle and Fujitsu alike.

Those systems are designed as midrange and high-end servers with relatively large shared memories and high single-thread performance. And the Sparc Enterprise M machines are getting a little long in the tooth, with no substantial upgrades in the past several years, aside from some relatively minor clock-speed bumps.

Toyoki was being coy and did not say that the "Athena" chip was one and the same as the sixteen-core Sparc64-X behemoth that the company was showing off at the Hot Chips 24 conference at the end of August. But there seems very little doubt that the Athena and Sparc64-X processors are one and the same device.

The reason is quite simple. Fujitsu can't afford to have two different kinds of Sparc processors, as it has so aptly demonstrated by forking its own Sparc64 processors with the eight-core "Venus" Sparc64-VIIIfx and then the sixteen-core Sparc64-IXfx processors used in its supercomputer line.

The Venus chip is the one that's used in the 10.51 petaflops K supercomputer at the Riken Lab in Kobe, Japan, and its follow-on is used in the PrimeHPC line of commercial machines that Fujitsu has been peddling to customers for the past year. These chips – or tweaked variants of them with higher clock speeds appropriate to enterprise workloads where thermals are not such a big issue and SMP ports instead of controllers reaching out to the "Tofu" 6D mesh/torus interconnect used on Fujitsu supercomputers – are not available in Sparc Enterprise M machines.

A system board using Fujitsu's 'Athena' processor

A system board using Fujitsu's "Athena" processor

The Sparc64-X chip is meant to bring the lines back together again, uniting commercial and HPC servers, and giving both sets of customers reasons to upgrade to new systems.

Toyoki didn't provide a lot of details on the forthcoming Athena chips and the systems that will use them, but he did say enough to demonstrate (as did Takumi Maruyama, who is in charge of processor development within Fujitsu's Enterprise Server business unit, during Hot Chips) that Fujitsu is keen on supporting both HPC and traditional enterprise computing workloads, and that in fact, the boundaries between the two are getting harder and harder to see.

Toyoki said that the traditional means of scaling-up computers to do more work – increasing the clock speed in processors in a shared-memory system or going distributed and scaling-up server nodes in a more loosely coupled cluster – were not going to work in a world where "big data" meets "business data" and correlations between the business and the outside world need to be sifted and correlated at real-time speeds.

"We need to develop a completely new concept," Toyoki said.

Zoom shot of the Fujitsu 'Athena' Sparc64 system board

The Fujitsu 'Athena' Sparc64 system board (click to enlarge)

Toyoki glossed over a lot of the details, but he did throw up some photographs of the prototype Athena system boards and give some basic feeds and speeds. He also quickly showed a system node interconnect scheme that El Reg managed to snap – but the picture was blurry. There's enough info, however, to make some guesses about how this Athena system will work.

First, Toyoki said that the Athena system design has its roots in the K super, and like that machine will have a system board that has four processor sockets as a basic building block. Each processor socket will be able to support up to 512GB of main memory, which is a respectable but by no means exorbitant amount for an SMP or NUMA server these days.

If the Athena processor is indeed the same thing as the Sparc64-X chip (and again, it almost has to be because Fujitsu is not rich enough to do processor development for two unique kinds of chips, just like Sun Microsystems and now Oracle aren't, either), then the on-chip interconnect can implement a NUMA link across four sockets gluelessly, meaning you do not need an external chipset.

But even with sixteen cores in a socket running at 3GHz, as the Athena chip will, a four-socket box will not have the kind of performance to tackle the big-data jobs that Fujitsu is chasing with the machines. And so the plan is to use an unnamed interconnect (perhaps a variant of the Tofu interconnect used in the K super) to lash together up to sixteen of those Athena system nodes. Now, this is a real big-iron system, topping out at 64 sockets with 1,024 cores and up to 32TB of main memory in a shared memory system.

How does the Athena stack up to other Sparcs?

For comparison, a top-end Sparc Enterprise M9000 has 64 sockets, which can be crammed with four-core Sparc64-VII+ processors running at the same 3GHz clock speed that is the top-level target for the Sparc64-X chip. The M9000 server tops out at 256GB across its eight system boards, however, yielding a maximum of only 2TB of main memory. And that is older DDR2 main memory.

By the way, this M9000 box does not support Solaris 11, according to Oracle's spec sheet, which is hard to believe. But Fujitsu says it is supporting Solaris 11 in data sheet for the M9000. Go figure.

If you want to go the distributed route and not spend big bucks on a monolithic SMP server, you can use the Real Application Clusters (RAC) extensions to the Oracle database and throw it on a Sparc SuperCluster using the T4-4 servers. You can do that today.

A single rack of the SuperCluster has four T4-4 servers, which are four-socket boxes. Each machine has four Sparc T4 processors, which have eight cores with eight threads per core running at 3GHz, with 1TB of main memory. That's 128 cores with 4TB of memory per rack, and across the eight racks of a SuperCluster (or Exadata for that matter), you would have 1,024 cores and 32TB of main memory. That's the same as the future Athena system, albeit with three very big caveats: a T4 core is not an Athena core, a T4 thread is definitely not an Athena thread, and a cluster is not a shared memory system.

When Oracle moves to the Sparc T5 processor, which Oracle was showing off at Hot Chips 24 and which is expected to ship sometime before the end of this year, it will be able to double up a SuperCluster to a staggering 2,048 cores and perhaps as much as 64TB of memory, if memory capacity doubles along with the core count.

Schematic of the Athena server interconnect

Schematic of the Athena server interconnect

How far performance can scale on the Athena boxes depends in large part on the socket-to-socket interconnect that Fujitsu has cooked up, and which, the more you look at it, looks less and less like Tofu.

According to the schematic above, Fujitsu is using a distributed crossbar interconnect, with two crossbar ports on each four-socket Athena board, which are multipathed into the on-board glueless interconnect to cut down on latencies. Four crossbar switch boards, each with a pair of crossbar ports like each server system board has, are at the heart of the system.

If you look really carefully at the Athena system interconnect diagram, you'll see that only one of the two crossbar ports on the system boards is actually used in the sixteen-way NUMA server, and that the used port only has six links. Maybe Fujitsu's artists just got lazy and didn't want to draw all of the links on the diagram because it would look like something out of a Spirograph.

That said, if for some reason those lines are representative of the way the crossbar interconnect works, then half of the crossbar switches and half the ports are not even being used at all, and that means the Athena system can scale to 32 nodes and 128 processors, or a whopping 2,048 cores in a single system image.

The Sun and Oracle roadmaps, last updated at OpenWorld 2011, did not have the future M4 processors scaling that far – but roadmaps change. Either way, with 64 or 128 Sparc64-X processors, this would make a formidable NUMA server or a fat memory node in a very large supercomputer cluster.

According to Toyoki, memory capacity and latency across a shared-memory system are, in fact, two of the things that Fujitsu very much kept in mind it designed the Athena processor and system.

The Athena system boards will have liquid loop cooling, which was designed for the K super and now the PrimeHPC commercial supers, to cool the processors and their main memory, allowing for the memory to be slammed right up against the processor sockets and thereby reducing the length of wires and therefore signal latencies between the CPU and its memory banks.

While neither Fujitsu nor Oracle were very clear about how the future Athena systems would come to market, Toyoki was very clear when Andy Mendelsohn, senior vice president of database server technologies at Oracle, took the stage with him to talk about some of the Oracle-specific features that the Athena machines will support.

For one, the Logical Domain (LDom) hypervisor, rebranded as Oracle VM for Sparc and thus far only available on Sparc T series chips, is going to finally make its way to Fujitsu boxes, providing true logical partitioning.

Fujitsu plans to support LDom partitions with the Athena systems

Fujitsu plans to support LDom partitions with the Athena systems

Toyoki also explained that the Athena chip would have direct decimal math execution units as well as special number-format conversions that are required inside the Oracle database, which converts from decimal to binary, does an addition, and converts it back to decimal format to store the answer in the database. These and other "software on chip" functions, as Fujitsu calls its on-chip function accelerators, are expected to significantly boost the performance of the Oracle database.

Oracle cofounder and CEO Larry Ellison took the stage to introduce the new Exadata X3 machines and the Oracle infrastructure cloud right after Toyoki, and he did not say Oracle would be peddling Athena machines, either. But Ellison seemed to be enthusiastic about the machines just the same.

"I am going to make a promise to you," Ellison said. "By this time next year, that Sparc microprocessor will run the Oracle database faster than anything on the planet."

While Fujitsu had been very careful not to put out a launch date for the Sparc64-X or Athena chips and the systems that use them, Larry just couldn't help himself. Now, it would be nice if Oracle customers would be told that the Athena chip is the Sparc64-X chip and that Oracle was going to resell and support them when Fujitsu gets them ready for market.

In the meantime, we'll gather up what information we can find on the forthcoming Fujitsu boxes and go through some early benchmark tests Fujitsu disclosed. Perhaps Oracle will say more about its future Sparc plans on Wednesday when John Fowler, executive vice president of systems, gives his keynote. ®