Original URL: http://www.theregister.co.uk/2008/12/11/realtime_linux_tests/

Red Hat and Novell duke it out in real time

There's money in them thar milliseconds

By Timothy Prickett Morgan

Posted in Operating Systems, 11th December 2008 23:58 GMT

When it comes to processing financial transactions, money can be won or lost in milliseconds. That's why high throughput, low latency, and consistent latency for transactions are the name of the game. Financial institutions are fanatical about their market data and trading systems, and Linux distros want to cash in on that.

The two key commercial Linux distributors, Red Hat and Novell, have both announced real-time variants of their respective distros: Red Hat Enterprise MRG (pronounced "merge" and short for messaging, real-time, and grid) and SUSE Linux Enterprise Real Time (or SLERT for short). Novell got into the real-time game first of these two companies, with the first SLERT 10 release launched in September 2006 with partner and long-time real-time operating system provider Concurrent Computer.

The third release of SLERT 10, which came out in November 2007, was gutted and a real-time patch set called CONFIG_PREEMPT_RT replaced the code Novell and Concurrent originally cooked up, and Red Hat's Enterprise MRG variant of RHEL is also based on this same patch set. (Concurrent, Red Hat, Novell, IBM, Silicon Graphics, and a few others have contributed to the real-time kernel patches, and Red Hatter Ingo Molnar manages this open source project).

Red Hat threw its hat into the real-time ring in December 2007 and started rolling out the real-time and messaging parts of the RHEL variant in June of this year. Both SLERT and Enterprise MRG only run on 32-bit x86 or 64-bit x64 processors architectures. Their regular Linuxes also run on IBM mainframes and Power servers as well as various Itanium-based servers.

Both Novell and Red Hat want to prove that these real-time variants of their respective Linuxes provide low latency at high throughput and therefore justify the extra licensing fees that both companies charge for the products. And to that end, both companies have asked the Securities Technology Analysis Center, which runs a financial application benchmarking center in New York and which also sells a bunch of tools to help financial institutions measure and tune the performance of their own applications, to put these Linuxes through the paces and demonstrate the performance benefits compared to normal Linux distros.

And to make comparisons easier, both Novell and Red Hat asked STAC to test the real-time Linuxes running the Reuters Market Data System at the 6.3 release level. This Reuters application gathers up financial information from myriad sources and then distributes it out to traders, brokers, and other financial end users who are relying on this information to give them an edge. Reuters also provides a benchmarking suite for prospective customers, and this is what STAC used in the tests.

The specific test was for what Reuters calls point-to-point server (P2PS), which is what you do when you want to stream a high volume of market data to dedicated desktops on your network.

Red Hat put out its performance numbers first, setting up Enterprise MRG 1 on six of IBM's HS21 two-socket blade servers. Each blade was equipped with two quad-core Xeon E5440 processors, which run at 3 GHz, and 16 GB of main memory. The blades had two Gigabit Ethernet ports and two T320 10 Gigabit Ethernet dual-port adapters from Chelsio Communications. The IBM BladeCenter chassis used in the test had a 10 Gigabit switch from Blade Network Technologies in the box. Red Hat did two runs, changing the size of messages streamed to the desktops (what is called the maximum transmission unit), running the test at 1500 bytes and then 9000 bytes.

Low mean latency

According to the STAC report, this setup tested had the lowest mean latency of any RMDS setup tested yet, with less than 1 millisecond of end-to-end infrastructure latency at the 700,000 update per second throughput level. (This was on an RMDS benchmark test that was optimized to reduce latency, not one that was meant to show maximum throughput).

On the PSPS Producer 50/50 fanout test - which is the extreme throughput workload in the Reuters test suite that assumes most users have common data feeds that have to be updated all at once from the backend systems Enterprise MRG was able to cope with 110,000 inbound updates per second on the RMDS workload and juggle 5.56 million outbound updates per second. With the 9000 byte MTU limit, the six blades were able to handle 140,000 inbound updates per second and push 7.07 million outbound updates per second on the workload.

A few days later, Novell announced its own results on the RMDS tests, also done by STAC. And instead of just kicking out a single number to compare it to other Linux and Unix platforms also tested using the RMDS benchmarks, Novell went one better and did four iterations of the test: two using SLERT 10 and two using the regular SLES 10. Each was configured first with Gigabit Ethernet links and then InfiniBand links. This allows prospective customers to see the benefit of using SLERT over SLES and also the effect of faster networking.

Novell ran its RMDS tests at the STAC lab on Hewlett-Packard BladeSystem blade servers, specifically on four BL460c blades, which are two-socket machines using quad-core Xeon X5450 processors running at 3 GHz, each configured with 16 GB of main memory. Each blade had two integrated Gigabit Ethernet ports and also had a dual-port 4x InfiniBand mezzanine adapter. The machines were configured first with SLES 10 SP2 and then with SLERT 10 SP2 Update 3, in each case tested with Gigabit and then InfiniBand interconnect.

With the Gigabit Ethernet links between devices running the RMDS code, SLES 10 was able to process 200,000 updates per second with under 1 millisecond of mean latency, but switching to SLERT 10 pushed it up to 500,000 updates per second. Moving from Gigabit Ethernet to InfiniBand interconnect, the SLES 10 setup was able to process 600,000 updates per second while keeping the mean latency under 1 millisecond for RDMS transactions. Adding SLERT to the InfiniBand setup did not boost performance of low-latency transactions by much, though, with it rising to only 750,000 updates per second.

On the P2PS Producer 50/50 fanout test, however, InfiniBand had a dramatic effect, with SLERT 10 being able to push 10.1 million updates per second outbound and SLES 10 hitting 9.34 million updates per second. On the Gigabit Ethernet network, both SLES 10 and SLERT 10 topped out at 1.3 million updates per second. The two takeaways from the much smarter Novell tests are that InfiniBand can make a huge difference in raw throughput on the RMDS workload, and that SLERT plus InfiniBand managed to edge out Red Hat's Enterprise MRG in the latency-optimized portions of the test.

Incidentally, the Novell-HP setup made use of Voltaire's InfiniBand adapters as well as its messaging accelerator software, which was announced in June of this year and which was designed specifically to improve the performance of InfiniBand networks in situations where it is being used in multicasting applications. Voltaire claims that this software, which runs in conjunction with Linux, can reduce latency on such applications by as much as 50 per cent. ®