Original URL: http://www.theregister.co.uk/2010/08/03/dcb_trill/
The DCB world is not enough
TRILL needed too
As Fibre Channel (FC) SAN storage gets emulated on Ethernet as FCoE, it's becoming clear that enhanced Ethernet is not enough; we need TRILL as well before full FCoE becomes a reality.
To make FCoE (Fibre Channel over Ethernet) practicable we need FC-using applications on servers and FC-accessed storage arrays to believe they are still using FC and not FC emulated on Ethernet. FCoE in other words has to behave like physical FC.
It's well understood that emulated FC must not drop frames and that frames should arrive quickly and not be delayed as they cross the network. Either condition would break FC, which was designed to have sole occupancy of the network it ran on.
Ethernet is a multi-user network with, for example, LAN and iSCSI traffic running on it. Adding FCoE to the mix means more data types crossing the network. The network's traffic load is generally increasing anyway because of the onrush of server visualisation, the rise of multi-core servers, and the steady increase in the number and capacity of the storage arrays the apps running in the virtual machines on multi-core servers access.
This data centre Ethernet is going to be congested and also going to have many, many switches sitting there between the servers, client PCs on the LAN, storage arrays and wide area network ports. The wish is to virtualise this network so that its resources can be more effectively used and its internal configuration altered without reconfiguring applications in servers and the other end points and vice vera.
Against this background how is end-to-end FCoE going to happen. There are two overview aspects to this we need to consider. The first is to look at how Ethernet can be made to perform like Fibre Channel. The second is to see how this Ethernet can provide high availability, load-balancing and high bandwidth as a basic feature so resource-hungry higher-level networking features don't have to be used.
Data Centre Bridging
The IEEE is developing Date Centre Ethernet (DCE) to provide a lossless and low-latency transmission of data, such as FCOE frames, across the network, traffic flow control in other words. There is a Data Centre Bridging task group working on this and looking at four aspects of the problem.
- Congestion Notification for congestion management - the 802.1Qau project.
- A priority-based flow control mechanism to ensure no data loss in congested networks, referenced by 802.1Qbb.
- Enhanced Transmission Selection (ETS) to assign network bandwidth to classes of traffic. The 802.1Qaz project refers to this.
- A discovery and capability exchange protocol to tell neighbours in a DCE network their capabilities and configuration to enable consistent configuration across the network. This is IEEE 802.1AB.
An IEEE 802.3x flow control standard uses a Pause frame to tell a sender to not send any more messages for a specific time. A Pause frame time value of 0 tells the sender it to resume transmitting at once. Ethernet provides an ability to set priority bits in an Ethernet message header. It is anticipated that the 802.1Qbb workgroup will combine 802.1Q prioritisation with the 802.3x Pause mechanism to provide priority-based flow control.
The idea would be to give FCoE traffic a high enough priority so it would sail through a congested Ethernet network while other traffic was delayed. If all this works well then this enhanced Ethernet could transmit FCoE traffic as well as physical Fibre Channel could transmit FC frames.
But DCE would still be limited in that it could not easily scale up as much as, or as easily as, the onrushing stampede of virtualised and multi-core servers demands. That's where TRILL comes in.
When several Ethernet switches, operating as layer 2 devices in in terms of the OSI 7-layer network reference model, are interconnected by inter-switch links ( ISLs), the IEEE 802.1D Rapid Spanning Tree (RSTP) protocol or a similar one is used to workout the best paths through the network and avoid things like endless loops. Spanning Tree allows only one link to be active between two Ethernet nodes, so this is achieved using the concept of active and inactive ports, with the former needed to construct a path through the network. Active bridging ports are put into a forwarding state, and un-needed ones into a blocked state.
A blocked link cannot be used for data transport and such ports are present but inactive, thereby literally wasting network bandwidth. This limits the overall bandwidth of a network and means reconfiguration when a port or switch fails takes time. It also limits the size of layer 2 Ethernet domains, and data centres will often have several of then delimited by layer 3 boundaries. We can't be having that; spanning tree approaches are inadequate.
What's needed is a routing protocol in layer 2, then we could get rid of spanning tree limitations, but routing is a layer 3 activity. The developing IEEE Transparent Interconnection of Lots of Links (TRILL) standard should solve this. It also enables multiple access paths at layer 2, which will increase available network bandwidth.
It uses the concept of a routing bridge (pdf) (RBridge) running an IS-IS (Intermediate System - Intermediate System) routing protocol, a link state routing protocol which operates by sending link state information through a network of routers or switches. Each router can then construct a representation of the network’s topology.
DCE plus TRILL
A DCE network, supporting DCB, built with switches supporting TRILL will mean that FCoE traffic can traverse the network, making multiple hops, without losing frames due to them being dropped as a result of congestion, or without taking too long a time, thus breaking the FC link, because the FCoE frames have a high enough priority be unaffected by the congestion. Such a network will also be able to have more than one link between switches, and these links can be simultaneously active, thus increasing effective bandwidth and availability.
A DCE network lacking TRILL can have multi-hop FCoE but it won't be routable and will be limited in scale and reconfigurability.
If the network needs more capacity then more switches and links are added. The IEEE is expected to, or hoped to, finalise DCE and TRILL next year. For now both Brocade and Cisco, the leading FCoE promoters, have announced products that will have a superset of TRILL functionality and, hopefully, support TRILL itself when it is standardised. These are Brocade One and FabricPath.
Combining Ethernet and Fibre Channel intricasies
Storage people considering FCoE will need to be really very well-educated about the intricacies of Ethernet indeed - the text above is simply the most basic overview imaginable - and how it will need to change to properly and effectively support FCoE. You will need to understand how different suppliers' products intercept the emerging standards.
It won't be practicable to have separate storage networking and Ethernet disciplines in your shop; the two have to converge; if they don't then your FCoE implementation will be done by Ethernet people that don't understand Fibre Channel or Fibre Channel people ignorant of Ethernet. You might as well draft your resignation letter now. ®