Introduction to Ethernet storage
NAS, NFS, pNFS, CIFS, iSCSI and FCoE
Deep dive El Reg has teamed up with the Storage Networking Industry Association (SNIA) for a series of deep dive articles. Each month, the SNIA will deliver a comprehensive introduction to basic storage networking concepts. This article looks at mainstream Ethernet-based storage.
Ethernet-based Storage – Standards, Deployment and Futures
This article, derived from a SNIA tutorial of the same name, describes the evolution of Ethernet, explores Ethernet Storage technologies today, and speculates on the impact of emerging technologies such as FCoE, pNFS and Cloud environments.
The past two decades have seen the emergence of Ethernet as a storage networking fabric, with Network Attached Storage (NAS) in the 90s, and iSCSI SANs in the past decade. To date, these technologies have been viewed as niche markets when compared with the incumbent storage networking technology (Fibre Channel SAN), despite the fact that each of the markets has consistently grown faster than the market as a whole.
However, we may be close to an inflexion point. According to the latest IDC Worldwide Storage Systems Tracker, Ethernet Storage (NAS plus iSCSI) grew to 51 per cent of the networked storage capacity shipped in 2010, and the FC SAN networked storage revenue share slipped to 55 per cent. Given a continuation of the more rapid growth rate of Ethernet Storage, and the emergence of new technologies, it seems reasonable to anticipate that Ethernet storage will soon become the revenue market share leader.
As the Internet has become the operations fabric for all organisations over the past 15 years, every enterprise has realised that a robust Ethernet infrastructure is a competitive requirement in business today. Consequently, every enterprise has invested in Ethernet networking expertise.
Most of us have witnessed the cycle by which Ethernet evolves, as it went from 10BASE-T to 100BASE-T to 1000BASE-T in our offices. Each new generation extended the capabilities of the previous generation, entering the market with high prices that declined as volumes picked up and the new infrastructure became pervasive, and enabled new applications. Gigabit Ethernet (GbitE) enabled two important new applications for corporate networks – Storage over Ethernet, and Voice over IP.
As the number of GbitE ports and the range of applications proliferated, it drove the requirement and demand for 10 GbitE. In June 2002, the standard for 10 GbitE (IEEE 802.3ae) was approved. 10 GbitE products have been available in the market since 2003 - initially providing switch-to-switch and switch-to-router connectivity.
Subsequently, we saw 10 GbitE options on high-end NAS and iSCSI storage systems. Most recently, in the past couple of years, we have seen the widespread availability of 10GbE ports on commodity server platforms – driven by standard high-speed I/O (PCIe), affordable pricing, and new TCP/IP offload innovations. Strong CPUs and large memory foot prints are also driving Server Virtualisation and subsequently the need for 10 Gigabit Ethernet connectivity.
And the cycle continues. The 10GbitE port-count ramp and the broadening range of Ethernet-based applications (virtual server and cloud environments) are driving the need for the next speed bump. The 40Gb/100Gb Ethernet specification (IEEE 802.3ba) was approved in June 2010, and 100Gb connectivity solutions for core routers and switches started to show up on vendor price lists this year.
Ethernet Storage Technologies
Network Attached Storage emerged in the 90s as a replacement for traditional Unix file servers. The enabling technology was NFS – an industry standard file-access protocol. Microsoft subsequently introduced its own file access protocol, CIFS/SMB, for Windows file serving.
NAS systems connected to standard Ethernet networks and provided much better price/performance than general-purpose servers for file serving workloads, significantly improved ease-of use, and provided built-in automated data management and protection capabilities (e.g., snapshots).
Although the original niche for NAS was file sharing applications and user “home directory” storage, ease-of-use drove NAS to become increasingly popular for enterprise applications that managed data through a file system interface. Eventually all the leading DBMS systems were re-engineered to support file access, and NAS became a popular storage solution.
This broader deployment accelerated with the ubiquity of 1Gigabit Ethernet, chip-set improvements and protocol stack improvements. The release of NFSv4 in 2003 brought significantly improved security, access control and replication and migration facilities
Today you can find significant NAS deployments in support of a very broad range of enterprise applications – particularly when operational flexibility, ease-of-use and low total cost of ownership are important considerations. IDC assessed NAS revenue to comprise 30 per cent of the networked storage market in 2010, with 46 per cent year-over-year growth.
iSCSI is a SAN protocol designed to transport SCSI block storage commands and data over Ethernet using TCP/IP. iSCSI was specifically designed to take advantage of all the ease-of-use, availability and guaranteed delivery mechanisms provided by TCP/IP, provide a seamless path from 1 GbitE to 10 GbitE and beyond, while adhering to the behaviour expected of a standard SCSI storage device. The spec was ratified in 2003.
Given that Fibre Channel SANs were already well entrenched when iSCSI solutions started to ship in 2004, their initial niche was storage consolidation for application servers in environments where simplicity, flexibility, and price/performance were the critical IT decision factors.
Most iSCSI deployments have occurred in departmental or regional data centres of large enterprises, or in the main data centre of medium and small enterprises. Most are “green field” SANs replacing direct-attached storage, particularly in Windows and Linux environments in which limited admin support, host attach costs, and infrastructure complexity have traditionally inhibited Fibre Channel SAN deployment.
In 2010, iSCSI systems revenues represented 15 per cent of the networked storage market, with 45 per cent year-over-year growth.
Virtual Server Environments
The advent of virtual servers have contributed to the high growth rates of NAS and iSCSI in the past couple of years, as companies look for a single network fabric to support virtual server and data mobility. NAS is the most popular storage protocol for VMWare, and iSCSI is a common choice when the guest O/S is Windows. Ethernet Storage is increasingly becoming the infrastructure of choice for large-scale virtual server environments, and a surprising number of enterprises today have “Ethernet-only data centres”, particularly among service providers.
Today’s virtual server environments are prototypes of tomorrow’s cloud environments. Most analysts agree that virtual server penetration is less than 50 per cent today. As that heads towards 100 per cent, this will drive the further adoption of Ethernet storage. In addition two new technologies are likely to accelerate this trend.
Fibre Channel over Ethernet and Convergence
FCoE is a new protocol that enables Fibre Channel commands to be transmitted natively over Ethernet using an optional transport mechanism instead of TCP/IP. It is designed to maintain backward compatibility with existing Fibre Channel endpoint infrastructure, and targets data centres wishing to transition to a "converged" (10 Gigabit) Ethernet network infrastructure, instead of separate storage and data communications networks.
FCoE, a standard developed by the INCITS T11 committee, depends on a new set of Ethernet capabilities, which have collectively become known as “Data Centre Bridging,” to provide a suitable transport environment. These capabilities (mostly Lossless Ethernet, congestion management and priority handling) were developed in the IEEE 802.1 committees.
FCoE is in the early deployment phase today, with more IT organisations assessing the technology in pilot projects than implementing it in production. But this is changing now as the pilots become deployments and references for other IT organisations.
Since the ratification of NFSv4, further advances have been made to the standard, notably NFSv4.1 (as described in RFC-5661, ratified in January 2010) that includes several new features such as parallel NFS (pNFS). pNFS represents a major step forward in NAS innovation: it distributes data across storage “clusters”, eliminates or reduces load- and capacity-balancing, and accelerates I/O.
pNFS and scale-out storage environments are expected to reach the market over the next 12-18 months, further expanding the market.
Ethernet is one of the few fundamental and truly ubiquitous technologies in IT today. Ethernet Storage has expanded from a collection of niches to become a significant percentage of the networked storage today. Today’s trends and emerging technologies will inevitably drive Ethernet Storage to become the preferred infrastructure for tomorrow’s IT environments.
This article was written by David Dale, SNIA Ethernet Storage Forum Chair, NetApp.
To view the SNIA tutorial visit www.snia.org/education/tutorials/2009/fall#networking.
About the Ethernet Storage Forum
The SNIA Ethernet Storage Forum (ESF) is responsible for driving the broad adoption of Ethernet-connected storage networking technologies and solutions. The ESF members represent leading hardware, software, data storage and data management vendors dedicated to providing the global IT community with vendor-neutral information, education about Ethernet storage solutions.
The ESF delivers educational content through industry and SNIA events around the globe, along with online, on demand webcasts. In addition, the ESF publishes articles and whitepapers covering the features and benefits, deployment scenarios and best practices for Ethernet-connected storage networking technologies and solutions.