Moving back to Layer 2
Expert Clinic Ethernet needs to become a lean, mean networking machine, and the way to do that, we're told, is to flatten the fabric, avoid layer the processing and keep as much as possible down in Layer 2. Is this real? What does it mean for performance, network management and open Ethernet standards?
Three experts tell you how they see it. FIrst up is network architect Greg Ferro, who says server virtualisation has change the nature of network traffic. Layer flattening operational risks exist and need fixing, but Ethernet performance increases are welcome while this is, hopefully, taking place.
Greg Ferro - Network Architect and Senior Engineer/Designer
For the past decade, applications have been welded to their servers —installation, commissioning and the odd upgrade, perhaps an occasional move to a new server and then the final stage lifecycle was power down and disposal. Servers were connected to their Ethernet and FibreChannel ports, configured to function and then largely ignored. The arrival of workable virtualisation started a new IT Infrastructure phase where the logical/physical connection between operating systems and their servers is changed.
For Ethernet, this dynamic capability is a performance problem, serious security issue and operational nightmare all at the same time.
The performance problem is at two levels. The first is described as the North-South/East-West design problem. In the past, all traffic flowed up and down a tree-like network, metaphorically North-South from edge to core and back again. This was partially driven by the use of Spanning Tree, but also by the 80/20 rule where 80 per cent of server data left the data centre by crossing the core to the egress point.
The use of virtualisation changed the 80/20 rule by needing East-West bandwidth between servers for VM migrations and access to Storage Arrays in the local network and the root of those bandwidth trees. This is more fully described here.
The second performance problem is that the Ethernet switch silicon performance needed to vastly improve. Most vendors have boosted performance by adapting techniques from storage fabric silicon ( which needed to over-engineered to cope with the deficiencies of the FC protocol).
But operational integrity issues remain stubbornly unresolved. Ethernet remains vulnerable to loops since the Ethernet protocol has no loop prevention. And clustering technologies such as Microsoft’s so-called Network Load Balancing & Exchange clustering, Oracle WebLogic and many others have used nasty network hacks that damage services in large Layer 2 (L2) domains. There is a technique known as Undirected Unicast, which floods all Ethernet frames to every server in the VLAN and creates a Denial of Service-like traffic profile within a single VLAN.
And the shared Ethernet medium that allows servers to move from VM Server to VM Server without changing the IP address, means that all servers must share the same “operational network” and this creates “fate sharing” problems when there are hardware and software failures. A single faulty network adapter (admittedly rare) could cause an outage of hundreds of servers, and there is no protection against this.
But operational integrity issues remain stubbornly unresolved
But these risks haven’t changed in the last twenty years or even stopped us from successfully deploying networks and large data centres. So while we wait for finalisation of network standards that will address these challenges, most companies are feasting on the the performance increases while the new features of Ethernet Fabrics are delivering the goods today.
Greg Ferro describes himself as Human Infrastructure for Cisco and Data Networking. He works freelance as a Network Architect and Senior Engineer/Designer, mostly in the United Kingdom and previously in Asia Pacific region. He is currently focussing on Data Centre, Security and Application Networking technologies and spending a lot of time pondering design models, building operational excellence and creating business outcomes.
Next up is Simon Pamplin from Brocade, and he sees flattening to layer 2 as beneficial, increasing network utilisation, improving management, and helping scalability.
Simon Pamplin - Director Pre-Sales, UK and Ireland, at Brocade
Compared to classic hierarchical Ethernet architectures, Ethernet fabrics provide higher levels of performance, utilisation, availability, and simplicity while stripping cost from the network. They flatten Layer 2 network design to help organisations on the journey to a cloud-optimised environment, making network design and management simple:
Classic Ethernet networks are hierarchical with three or more tiers. Traffic has to move up and down a logical tree to flow between server racks, adding latency and creating congestion on Inter-Switch Links (ISLs). Spanning Tree Protocol (STP) prevents loops by allowing only one active path, or ISL, between any two switches.
Ethernet fabrics prevent loops without using STP. Flatter networks include self-aggregating ISL connections between switches, eliminating manual configuration of Link Aggregation Group (LAG) ports while providing non-disruptive, scalable bandwidth within the fabric. Ethernet fabrics support any network topology (tree, ring, mesh, or core/edge) and avoid bottlenecks on ISLs as traffic volume grows, since all ISLs are active.
Classic Ethernet switches require configuration of each switch port and server virtualisation requires multiple virtual machines to be configured on each switch port. Ethernet fabrics have distributed intelligence, which allows common configuration parameters to be shared by all switch ports in the fabric. In the case of virtual machine migration, the network policies for that virtual machine are known at every switch port so migration does not require any changes to network configuration.
When a link is added or removed in the fabric, traffic on other links continues to flow non-disruptively
Classic Ethernet allows only one path between switches. Ethernet fabrics overcome this. When a new switch connects to the fabric, no manual configuration is required for the inter-switch links. The switch joins the fabric and learns about all the other switches in the fabric and the devices connected to the fabric. Should a link in a trunk go off-line, traffic on the remaining links is not affected and incoming frames are automatically distributed on the remaining links without disruption to the devices sending them.
Classic Ethernet uses STP to define a loop-free path, forming a logical hierarchical switch tree - this lowers utilisation. When a new link is added or removed, the entire network halts all traffic for tens of seconds to minutes while it configures a new loop-free tree. Ethernet fabrics do not use STP to remove loops. They use link state routing with equal-cost multi-path routes, so when a link is added or removed in the fabric, traffic on other links continues to flow non-disruptively. Link resiliency is assured and full utilisation of all links between switches is automatic.
Classic Ethernet switches require management. Each switch has to be configured and each port has to be configured for protocols (STP, RSTP, MSTP, LAG, and so on), VLANs, network policies, QoS, and security. Ethernet fabrics share configuration information among all switches in the fabric. When a new switch joins the fabric, it automatically receives common information about devices, network policies, security, and QoS. This simplifies network configuration, reduces mistakes, and reduces operating cost.
Simon Pamplin has over 18 years experience in the IT industry, both as customer and vendor. He has specialised in technical pre-sales and spent the last eight years at Brocade.
Our third expert is Andrew Buss from Freeform Dynamics, who thinks similarly to Greg Ferro; classic up-and-down the tree Ethernet is ill-suited to virtualised server-to-virtualised server and server-to-networked storage array communications. These need a flatter Ethernet to become efficient.
Andrew Buss, Service Director, Freeform Dynamics
The concept of an IT service used to be defined as an individual, often silo'd, application running on dedicated hardware. With static infrastructure, traffic was fairly predictable, moving between applications on the server and client. As a result, the Ethernet-based data network tended to be optimised around this type of traffic flow and so the network evolved into hierarchies resembling a tree.
This architecture is well suited to these predictable traffic patterns between dedicated server and client. However, performance tends to suffer when traffic has to move between trees in the network – as happens when servers in different network segments need to communicate with each other.
At the same time, as Ethernet networks greatly expanded in size, switching based on Layer 2 became difficult to scale as it was broadcast-based. Protocols such as Spanning Tree helped, but ultimately the management, reliability and traffic overhead led to limited impact. Techniques such as VLAN segmentation also assisted, but introduced even more complexity, such as the need to route traffic between VLANs. This resulted in the emergence of Layer 3 Ethernet switches that work natively with IP to enable routing between VLANs in hardware.
Meanwhile, storage was moving out of the server, from internal direct-attached disks to a networked pool of storage. This SAN (Storage Area Network) had to support many servers connecting to many storage assets. To be effective, any node on the storage network must be able to communicate effectively with any other node on the network.
This led to the implementation of a different type of network architecture to traditional Ethernet – a fabric network designed for the needs of storage systems, where the network has a consistent state that is optimised to support point-to-point communications between any two nodes on the network and which has single step lookups to forward traffic.
Ethernet networks - as traditionally implemented - have arguably become too complex to support dynamic services
Over that last few years there has been a growing shift in how IT services are delivered. The old, static approaches are progressively giving way to a service-centric approach that is defined by the needs of the business. The service is delivered using a set of building blocks that can be assembled to provide the service efficiently and effectively. A large enabler of this has been the emergence of virtualisation to allow service instances to be spun up or down anywhere on the network as required.
As a result network traffic between client and server is becoming less predictable; perhaps more importantly critical inter-server communications is becoming more widely used while virtual machine traffic is exploding. On top of this storage traffic is ballooning and in a growing number of cases also moving onto Ethernet.
Ethernet networks - as traditionally implemented - have arguably become too complex to support dynamic services and as a result of this the underlying Ethernet network architecture needs to evolve to become more fabric-like in its characteristics and architecture, at least for certain areas of the data centre network where there is a lot of server and storage traffic.
A big step required in order to move to a fabric is to “flatten” the network and remove the “trees” that make point-to-point communications difficult. The ultimate goal would be to have a completely flat fabric with no tiers whose main purpose is to aggregate traffic and transport it between network segments. However, in most cases this will not be cost-effective to implement, or realistic for organisations to adopt widely in the medium term.
The goal will be to try and collapse parts of the network by removing one or more of the network tiers. In effect this would provide “islands” of fabric. This can be achieved by deploying some of the latest Ethernet fabric switches without having to have a “roots and branch” overhaul of the entire network.
Another change with the move to a flat fabric network is that extended support for Layer 2 networking is needed. As we’ve seen, Layer 3 switching helps in situations where IP is the dominant protocol. However, the rise of storage traffic on Ethernet as well as the growing prevalence of virtual machines means that non-IP traffic is getting to be a lot more significant and needs to be catered for at Layer 2 across different network domains. This requires new equipment with support for enhanced Layer 2 protocols such as FabricPath to remove the need for Spanning Tree on large, segmented network implementations.
Whichever direction your data centre network is headed, the long term trend in IT is towards a more dynamic and converged IT infrastructure. Having a long term view on supporting non-IP traffic effectively over Ethernet should be high up on the list in order not to be locked into an architecture that is difficult to reconfigure or scale.
Andrew Buss is the Service Director for Freeform Dynamics. He is is responsible for managing the company’s services portfolio for both end user and vendor organisations, and has been an industry analyst since 2001.
Flatten away but stay open
Keeping Ethernet traffic as much as possible down in Layer 2 is seen as a good thing as it enables Ethernet to cope better with unpredictable traffic and cross-fabric traffic in an era of virtualised servers and networked storage (Pamplin). Moving to a Layer 2-centric network can be done without ripping and replacing the entire Ethernet (Buss) but keep the idea of open networking in mind and don't get trapped by network hacks (Ferro). ®
Inefficient use of Ethernet is because of poorly written software not because of flaws in the makeup of the OSI model. Anyone who has used Outlook over a WAN knows what I am talking about. 3K of traffic is generated to successfully deliver and confirm delivery of a 1K email. Plus all the other background crap MS crams up my network with.
MS is not the only manufacturer who needs to stop making software that turns my network into a Hen Party.
Friends and Foes
All the talk of auto configuring, self healing fabric is great but the Internet can not be viewed as friendly territory where every other switch and router is trusted. Just think back to the Chinese traffic grab a few years ago and consider the implications of a few forged hop counts that can cause traffic to be routed from New York to Washington DC via Beijing or Tehran. That trust mistake was made 40 years ago with the initial design and should not be repeated.
Once you acknowledge the trust issue all the automated solutions collapse and the hierarchical arrangement makes sense - rings of trust are more accurately branches of trust and you stay as far from the root as possible. Fabric only works within limited areas of trust and the best way to interconnect these zones will still be hierarchical.
L2MP is here
The replacement technology for STP is here already. It's broadly know as Layer 2 Multi Pathing (L2MP). There are two approaches but only one looks serious and that's called Transparent Redundant Interconnection of Lots of Links ( TRILL ) and you can find details on the IETF using your favourite search engine.
Cisco has a proprietary and pre-standards implementations of TRILL that they call FabricPath which is shipping today.