How to build a BONKERS 7.5TB, 10GbE test lab for under £60,000

Now, the real fun and games starts: Networking

Networking has proven to be the most troublesome aspect of my test lab assembly. Most of my networking nerd friends are thoroughly indoctrinated Cisco believers. If you corner them, they will admit that something called "Juniper" can probably do what needs be done, but you need the better part of a bottle of scotch to get them to even admit that Arista even exists. They aren't much help when price is a consideration.

If your corporate network is run by these types of folks you may end up deploying whatever the company standard switch is into your test lab. Just as likely you wouldn't be allowed to touch the thing if you did buy a company-standard switch as that would fall under the oversight of the networking team. While I can't help you resolve internal corporate politics, I have done some legwork into finding solid 10GbE switches for the rest of us.

First on my list is the D-Link DGS-3420-28TC; with 24 10/100/1000Mbps ports and four SFP+ 10GbE ports it is the entry-level to the 10GbE world. With a little bit of hunting, you can find this switch for roughly $1,500, more commonly about $2,000. It is a solid switch if your 10GbE port needs are four or fewer; it is also a great switch to bridge the 1GbE and 10GbE segments of your network. $375 per 10GbE switch port is almost reasonable, and a solid consideration if you are only putting a few 10GbE compute nodes into play.

More switches and virtualisation nodes

If you need versatility, try the 24 SFP+ 10GbE Dell PowerConnect 8132F. With the lower support option, the Dell switch costs $8,069; $336 per port, and very reasonable. If your organisation requires full enterprise support, even for test lab setups, then the cost jumps to $12,880. At this price the Dell is an eye-watering $536 per 10GbE switch port. For the money you get a switch that supports more enterprise features than the D-Link and has an expansion module. You can fill the expansion module with either four additional 10-GBase-T or SFP+ ports, two 40Gbit QSFP+ ports or an additional eight SFP+ ports via the QSFP+ module and breakout cables. The expansion modules cost extra.

The workhorse switch for me is the 24 SFP+ 10GbE Supermicro SSE-X24S. It supports [PDF] the standards I want in a test lab – or indeed in a production – switch and can be found for as little as $7,500. This puts 24 ports of 10GbE at your disposal for $312.5 per 10GbE switch port. It has no uplink ports, expansion modules or easy connectivity to traditional 1GbE networks; it is best for those looking to keep their entire network segment 10GbE.

These are, of course, prices for those who don't have any sort of volume deals with the vendors in question. You can drive down the cost of the network a small amount with a little effort.

If you are not using the Fat Twin for your compute nodes, then a dual-port Intel X520 10GbE card will run you roughly $425. Fat Twin users can snag a Supermicro AOC-CTG-I2S for $430. It is Micro-LP, so depending on the Fat Twin it will leave your other LP PCI-E slot open for additional goodies; an important considering for a test lab node.

10GbE nose to tail

Our storage node is $5,500 and delivers a mix of slow and fast storage. We can get 8 Fat Twin compute nodes for $21,000, but outfitting it with 10GbE cards and buying the requisite SFP+ direct attach cables will drive the total cost to $25, 000. The switch is an additional $7,500. The total bill for our enterprise test lab – without an operating system on our compute nodes, mind you – is $38,000.

Of a 24-port switch, this configuration would have 18 ports occupied. That's just enough room in my test lab to hook up my Persephone 3 and two Eris 3 systems. Seeing as how the storage server has to be up 24/7, it is here that I house long-term test lab VMs. The domain controller, DNS/DHCP server, various layers of CentOS router/firewall VMs and an administrative VM used to RDP into the lab all live here.

This works well for me; the test lab is 10GbE throughout and lives on its own subnet. A CentOS VM running on the file server acts as a router bridging the rest of the network and the test lab. Because the file server sports both 1GbE and 10GbE NICs, I can RDP into the test lab network through one of the file server's 1GbE ports, saving the cost of a 10GbE port on my production network to bridge the two networks. More importantly, it allows me to put all sorts of firewalls and intrusion detection systems between the two to keep any of the creepy crawlies that may leak into the test lab from breaking out into the production network.

Final considerations

There are a lot of tweaks you can make to this configuration. Rather than relying on software raid in my storage node, an additional $800 would buy an LSI MegaRAID SAS9280-16i4e. Not only does it do all the commonly requested RAID levels, it could use the eight Hyper-X drives as a block-level cache array to front end a whole lot of spinning rust. Your files server instantly becomes bulk automated tiered storage. Once I get my hands on one, I'll review it.

Unless you rely a great deal on open source, the hardware may well be the least of your concerns. You aren't going to cover eight nodes of compute with a Technet license. Eight Server Standard licences will run you at least $6,000, with eight Server Datacenter moving just over $30,000. If you choose Microsoft's hypervisor and management tools to power your test lab, the System Center Datacenter management tools – and let's face it, you'll be running enough VMs on those kind of systems to need Datacenter – will run almost another $30,000.

Going the VMware route (using acceleration kits to provide the first 6 licences and the vSphere server, then adding 10 CPU licences and the mandatory minimum support) will range from about $23,700 for Standard to just over $70,000 for Enterprise Plus. That's before you add any Windows licences you may or may not need.

Bringing it all together, that's 64 Sandy Bridge generation Xeon cores at 2.4Ghz with 1TB of RAM, 6TB of slow storage, 1.5TB of fast storage, for $38,000. Add VMware Standard and unlimited Windows Server instances all interconnected with dual 10GbE networking for a little over $90,000. Not a bad test lab at all. ®