VirtenSys PCIe cloud switch arrives
Are PCIe SANs in prospect?
SaaS data loss: The problem you didn’t know you had
Virtensys has delivered pre-production units of its server I/O virtualising VMX5000 switch to potential server and storage OEM partners. Its use enables the sharing of I/O adapters and storage among connected servers using PCIe connections.
The idea is to use PCIe cabling and extend a server's PCIe bus outside the server to a switch used by many servers. Server storage (direct-attach storage or DAS) is attached to the switch for shared use instead of to each server for its individual use. I/O adapters such as Ethernet NICS and Fibre Channel HBAs are also attached to the switch for shared use.
Sharing of devices in this way is, VirtenSys claims, a way to increase device utilisation, lower device counts, reduce electricity needs and lower both capital and operating expense (Capex and Opex).
Virtensys argues that use of its switch can reduce rack and blade server management costs by more than 60 per cent, power and cooling costs by up to 30 per cent and equipment costs by as much as 50 per cent. It also claims that its technology enables networking resources to be dynamically allocated on demand as connected servers talk to virtual I/O adapters and DAS rather than physical adapters and storage. The switch product supports both physical and virtualised servers and comes either with (VMX-500 LSR) or without (VMX-5000 LS) RAIDed SAS and SATA hard disk drives, known as virtual DAS.
Timed for release just before VMware's Cannes VMworld show, the company says its virtual I/O is a natural fit with VMware's Virtual Data Center Operating System. VMware VP for infrastructure alliances, Shekar Ayyar, said: “We are... working with VirtenSys in an engineering collaboration to explore integration and optimisation options for our respective products. VirtenSys solutions support VMware’s vision of self-managing, self-healing virtual datacenter environments.”
The VMX-5000 switch can also be used in XenServer and Hyper-V environments. Its use is completely transparent, Virtensys says, to a server's operating system, and the product can be slotted in to a server infrastructure with no change at all required to the server's software. In an ominous threat to volume sales of I/O adapters VirtenSys says the number of I/O components in a server infrastructure can be reduced by up to 90 per cent.
VMX-5000 switches attach directly to multiple physical servers and support Intel and Neterion 10GE NICs and QLogic Fibre Channel HBAs. This device and interface portfolio should be expanded and both SAS and SATA I/O adapter support should be ready in a few weeks. There is a faint prospect here that storage array manufacturers could, if they are using X86/PCIe-based controllers, extend their PCIe bus to the switch and so help produce a complete PCIe 'cloud' embracing servers, networking interfaces, and shared storage.
What that could mean for iSCSI and Fibre Channel is death by PCIe. What we are talking about here is, in effect, a PCIe SAN (Storage Area Network). Such products are already emerging.
What Dell, IBM, HP could do with this technology is easy to see. For example we might envisage an HP BladeSystem with PrioLiant servers, ProCurve switches and EVA storage all linked across a PCIe cloud. Within this cloud there would be no Fibre Channel or iSCSI and LAN connectivity would occur at the PCIe cloud edge.
On the Dell front we could also see the emergence of Dell server and Dell storage PCIe cloud lego building blocks (rendering the InfiniBand links between servers and Xsigo switch and the switch itself redundant), as we can with IBM.
The VMX switch is available in rack-mount and blade server form factors with production units ready in six to nine months from VirtenSys' OEM partners. These partners will be able to offer rack-mount and blade server products at lower cost, both Capex and Opex, to customers looking to save cost and electricity.
If the technology works then it's feasible that it will become just another part of the industry standard commodity server infrastructure building block set. ®
COMMENTS
Further reading
from IDT and PLX - some PCIe switch chips with a prospect of multi-root architectures. Maybe pre-IOV, but the docs attached give a thorough technical background, answer many of my basic questions.
http://www.idt.com/products/getDoc.cfm?docID=18639469
http://www.idt.com/products/getDoc.cfm?docID=18688297
http://www.plxtech.com/products/expresslane/switches.asp
Interestingly for me, Pericom has not much to offer in that vein... Unsurprisingly, neither has Intel. That LSI paper on IOV mentioned before makes me wonder what LSI has up its sleeve.
PCI-e IOV - multiple root complexes can't talk to each other
Ahh well. So you can put your NIC in an external expansion box, rather than into the server itself. But unless it's a very special NIC that supports IOV, you can only use it from one root complex (= from only one host computer). The IOV standard simply says that the external multi-root PCI-e switch can carry traffic for multiple PCI-e bus trees that don't know about each other (like VLAN's). Each bus tree is "owned" by a particular "OS partition" running on the host computer. At least the part of the bus tree carried by the external switch runs virtualized on the switch, though I guess IO virtualization at PC server chipset level is already in the works too.
Any peripherial board that is to be shared by multiple PCI-e trees must have special "virtualization" support, to be able to keep track of several simultaneous PCI-e conversations with multiple root complexes. Not so very sexy...
I bet the HPC folks would appreciate much more if the external PCI-e switch could cater for direct root-to-root data transfers - for HPC networking purposes. Imagine DMA from one root complex to another root complex (memory to memory). This doesn't necessarily mean that a standard IP networking stack would be involved - perhaps as a slow fallback solution or for management purposes. Rather, I could fancy some dedicated low-latency mailboxing framework. It would really get the multi-root PCI-e monster fairly close to a NUMA, except that we're still speaking distinct "OS partitions" in the IOV lingo. The way I understand PCI-e IOV, such direct transfers are impossible. Maybe via an intermediate "virtual NIC" or some other peripherial along those lines (call it a DMA engine with some dedicated memory of its own) implemented within the external PCI-e switch.
The sort of bandwidth available from PCI-e at least makes very good sense for direct RAID storage attachment. Perhaps not via an additional intermediate storage bus technology (that could be useful as a slow lane for some wider-area SAN interconnects).
Love it!
I just love this PCIe cloud idea.
PCIe 3 is due to be ratified later this year which means each 1x channel will handle 1GBs so with a 32x channel that some serious bandwidth available compared to FC & iSCSI
Hot swap is going to be a problem - but I guess the switches might be able to handle this.
All they need to do now is standardise on 2 x 16x backplane chassis slots...Now we might actually start getting some interoperability in the datacentre. Racks of NICs, racks of graphic cards.... do the same with a HyperTransport backplane & now we can have racks of CPUs as well. At last proper hardware virtualisation!
IT infrastructure monitoring strategies
Agentless Backup is Not a Myth
Top 10 SIEM implementer’s checklist
Steps to Take Before Choosing a Business Continuity Partner
Enabling efficient data center monitoring
