Crazy Canucks heat their lab with muahaha-capable server
Supermicro 2028GR-TRT quad GPU box
Review Playing with cool technology is not why either of us chose systems administration as a career. Nor was it something we expected as part of our employment. That said, it is impossible to deny the attraction cool technology has. It is fitting, then, that on systems administrator day we get the chance to publish a review on the kind of hardware that lured us into systems administration in the first place.
Supermicro has once again graced our lab with another impressive server, this time in the 2028GR-TRT 2U Superserver. This isn't a schnazzy new form factor. It's not a blade system or an ultra-small, low-power widget that tries to be different.
The 2028GR-TRT is a brute force beast of a system that makes no excuses and takes no prisoners. It is nothing less than a super computer in a can; the maximum amount of compute capacity you can cram in to 2U of rack space.
The 2028GR-TRT is a juggernaut, and its – quite literally – howling performance makes some deeply repressed, primeval part of our personalities break out in face-splitting grins. The same parts of us that love fast cars, fighter jets, great big honking guns and enormous missiles blasting things into space.
Perhaps it is best explained by the following video. (Warning: very, very loud. Under no circumstances watch this video with headphones on.)
You can forget taking calls while in the vicinity running tests on this system!
Now, volume is not much of a consideration when you're talking about systems intended to live in a noisy data center, but when it's just over in the next room, the volume is certainly hard to ignore. The airflow out the back is such that we've repeatedly jokes about installing wind turbines behind this Superserver to regenerate some electricity.
A look at the hardware
The video above is the initial boot spin up of the fans once all four nVidia Tesla K40 cards have been installed.
Earlier this year we reviewed some Supermicro Twin servers, the 2028TP-DC0FR and 2028TP-DC1R units. Only the one of these units – the 2028TP-DC1R – could handle GPUs; it handled one per node, with two nodes in 2U.
Unlike the 2028TP-DC0FR and the 2028TP-DC1R, the 2028GR-TRT is not a twin server. There's only one node in the chassis. One node is enough.
The 2028GR-TRT on in our lab sports a pair of Intel Xeon E5-2697 v3 processors at 2.6 GHz. It has 16 DIMM slots with 16GB DDR4 at 2133MHz each for a total of 256GB. Oh, and of course, the aforementioned four Nvidia K40 Tesla GPUs which each have 12GB of GDDR5 RAM and 2880 CUDA cores.
One thread... two thread... that's a lot of threads
For all intents and purposes the 2028GR-TRT is a supercomputer node – and is easily the most ridiculous system we've played with to date, earning the in-house designation of "muahahaha-capable".
Supermicro has a knack for building versatile and solid proprietary motherboards for their chassis designs. The 2028GR-TRT uses the X10DRG-HT motherboard, purpose-built for GPU servers. With the addition of a pair of PCI-E riser cards, you can connect up to six GPUs to the motherboard, provided you have the right chassis for it. This particular chassis only supports the four GPUs.
There's no shortage of SATA ports with 10 on the board; it's got a pair of 10GBase-T ports and an additional dedicated IPMI LAN port, so unless you find yourself needing a 40GbE connection you won't need to add any NICs. If you do need to add some additional networking oomph, there is a PCI-E x8 low profile slot.
There isn't any inward-facing USB ports to use a small flash key as a boot drive on this motherboard. Instead, Supermicro has done one better with their SuperDOM (Disk on Module) flash memory module. SuperDOM does have distinct advantages over a generic USB port. It connects directly into a SATA port – and on this generation of motherboard, the two yellow ports have extra pins for power. (For older motherboards, a small 5V cable is needed.)
SuperDOM modules come in 16, 32 and 64GB flavors, consume 1-2 watts each and have read/write speeds comparable to a mid-high performance SSD. Unfortunately, we didn't receive the modules in time for this edition of the review, but we felt it's an interesting feature worth mentioning.
In order to provide enough storage oomph to drive the system we equipped the unit with 6 Intel 520 series 480GB SSDs for workload data. It runs a pair of 15K SAS drives for the operating system.
The CPR-2021-3M2 PSU's efficiency
If you're planning on building a supercomputer, data center or even just a rack full of compute nodes, efficiency is always a consideration. The 2028GR-TRT comes with a redundant pair of CPR-2021-3M2 2000W power supplies, certified at the 80 PLUS Platinum rating. With an average efficiency of 95.30%, there's not much to complain about here.