Comment There is a battle going on behind the scenes over the location of storage's soul: the controller hardware and software. Oracle, Dell, EMC and VMware want it to be in the server, while NetApp and HDS want it to be in the array, an array operating with servers but distinct from them.

The picture is not as clear-cut as this on the surface – NetApp is working with Oracle for example – but this is my take on what is happening down in the development depths, among the strategists and engineers with multi-year product horizons.

The modern storage industry, the one shipping networked external storage arrays, has been built on two foundations. One is EMC's establishment of a market for third-party external, block-addressed storage arrays distinct from the server suppliers of the time: HP, IBM, Digital Equipment, etc.

The other was the invention and establishment of file-addressed network-attached storage (NAS or filers). NetApp is the single most effective proponent of that, although EMC grew to ship more filers than NetApp. EMC and NetApp represent the twin peaks of the external storage array.

A storage array comes in two flavours. It is either monolithic, with multiple controllers or engines and some fancy interconnect hardware to link these to the storage shelves – think Symmetrix, latterly VMAX – or modular. Modular arrays have two controllers linked – by simpler Fibre Channel or latterly SAS – to the storage shelves. NetApp's FAS arrays and EMC's CLARiiON are classic embodiments of this idea.

Applications in servers sent SCSI block requests or file access requests to these arrays, which presented themselves, logically, as a single pool of storage, separated into dedicated logical disks (LUNs) for the server apps, or sharable filestores.

Andy Bechtolsheim

This long-lived storage concept is now being discarded, and the first nail in its coffin came from Sun and the inventive Mr Andy Bechtolsheim.

Honeycomb upsets the storage hive

Bechtolsheim's idea was that co-locating servers and storage in the same overall enclosure would speed server apps dependent on lots of stored data. Thumper, a server-rich NAS device delivered as the X4500, was one result of this and Honeycomb another.

Neither set the world on fire but they did show the way to getting more data into servers faster. Then Oracle bought Sun in 2009 and suddenly Bechtolsheim's idea got a rocket boost from the Exadata product, a set of server resources running Oracle software with their own storage resources. This is setting the Oracle World on fire, with much encouragement from Oracle marketing because its own bunch of modular arrays was pretty second-rate.

Oracle Exadata database machine

What Sun invented and Oracle extended is the NoSAN server. EMC has seen this idea and responded by devising an opposite of this, the No-Server SAN, a kind of reverse engineering in its way.

EMC brings the servers to the array

EMC is trying to have it both ways. VMAX, VNX and Isilon arrays are going to be able to run application software in server engines in the array controller complex. There is a natural fit with VMware's ESX running the whole shebang and VMs being loaded to run storage controller software and applications that benefit from low-latency access to buckets of data. These array-located app servers use the array's own internal network or fabric, VMAX's Virtual Matrix for example, instead of the normal Ethernet or Fibre Channel fabric. This isn't SAN access as we know it.

EMC also has its Project Lightning to have its arrays manage the loading and running of flash caches in servers, PCIe-connected flash. That's a step on the road along which Dell appears to be further along. The Round Rock company is also going to build servers with flash, but this is a storage tier and not cache. This tier zero storage is logically part of the entire array controller-managed storage pool with automatic data movement.

The future may be not to have a storage networking protocol at all.

Now EMC may well have this in mind as well, with FAST VP shipping data to and from the server flash which is then not really a cache but tier zero too. However Dell's vision, as I understand it, is to move the storage into the servers and hook it up to the same PCIe gen 3 bus that the flash and server's DRAM hook into. Once again this means that the servers will not use traditional external storage links to access data. This again is not a SAN. What do these NoSAN ideas mean for external array vendors?

Storage array vendor conundrum

The mainstream external storage array vendors are facing two, not one, strategic challenges to their long-term hegemony. The nearer term one is the introduction of networked flash-only arrays from companies like Huawei, Nimbus, Pure Storage and others, which threaten to cream off their profitable fast array business based on 15,000rpm Fibre Channel disk drives. All primary data storage could migrate to flash-only arrays, leaving the EMC, Dell, HDS, HP, IBM and NetApp-style vendors to store the bulk data crumbs, the low-access rate, lower profit, online data repositories.

EMC VMAX array Virtual Matrix

The second threat though is the more important one because, after all, a networked external flash storage array is still a networked external storage array. Co-locating servers and storage with no traditional storage networking protocol connecting them is different. If servers absorb external storage, as is the Oracle Exadata plan, or external storage absorbs servers, as is the import of Dell and EMC's plans, then networked external storage is no longer needed: we have the NoSAN server.

There's a distinction to be made between converged systems with co-located servers and storage using storage networking protocols and converged systems that do not, the NoSAN servers.

A VCE Vblock, an HP VS3 and a NetApp Flexpod set-up all use networked external storage. An app-running VMAX has no storage networking protocol needed to link servers (app-executing VMAX engines) and storage (storage controller app-executing VMAX engines) as they all hook into the VMAX internal fabric. This is fundamentally an entirely different beast. For a start the server-storage link is proprietary and not open. You wouldn't be able to substitute a NetApp, HP or IBM array for the VMAX storage part of VMAX app-executing system. It would be like trying to put a BMW engine in a Mercedes car. Good luck to you and don't expect any sympathy for the grief heading your way.

Criticise Fibre Channel, FCoE and iSCSI all you like but they are open systems allowing you to switch suppliers of servers and storage either side of the storage networking link.

FCoE becomes irrelevant

Discussions about whether to move to Fibre Channel over Ethernet (FCoE) from Fibre Channel will be rendered irrelevant by co-located servers and storage communicating across some kind of I/O backplane or mesh network, or servers and storage talking across PCIe. FCoE is a transition from one instantiation of a legacy storage-server connection protocol to another.

The future may be not to have a storage networking protocol at all. Data will ship between server memory and flash to co-located array storage across PCIe buss links or an array's internal fabric.

What this means for stand-alone storage array products is that a razor thin end of a wedge is appearing in the door through which they sell their arrays. This wedge can be expected to fatten. If it does, then the external array business will be impacted. It may be that data growth will be such that array sales increase while NoSAN server storage sales grow as well. But there is a long-term threat here, looking at a five-year or 10-year timescale, in which the external array business declines.

Its only hope of rescue will be through the provision of external array links that match the latency and bandwidth of third-gen PCIe, or internal array fabrics such as is found inside VMAX.

A strategic battle has been joined and the existence of the external storage array business is being questioned. Yes, it's a razor thin wedge that has been slipped into the external array door, but wedges do what wedges do: they become thicker. If Dell and EMC pull their NoSAN server ideas off, then a world of hurt awaits external array vendors that have no effective response. ®

Related stories

• EMC denies big server biz plans ... but IS building servers (24 May 2012)

  https://www.theregister.com/2012/05/24/emc_building_embedded_servers/

• EMC VSPEX storage torpedo could sink FlexPods (10 April 2012)

  https://www.theregister.com/2012/04/10/emc_vspex_vs_flexpods/

• The God Box: Searching for the holy grail array (13 February 2012)

  https://www.theregister.com/2012/02/13/the_god_box/

• Pure Storage attacks EMC heartland (22 November 2011)

  https://www.theregister.com/2011/11/22/pure_storage_fc/

• SAP, EMC and VMware in 3-way cloud orgy (11 November 2011)

  https://www.theregister.com/2011/11/11/emc_sap_hana/

• Freak lightning strike sends app, storage servers back in time (9 November 2011)

  https://www.theregister.com/2011/11/09/emc_lightning_strike/

• Isilon on its separate scale-out reality (28 October 2011)

  https://www.theregister.com/2011/10/28/isilon_vs_netapp/

• Gelsinger pops lid on top-secret EMC Lightning code (26 October 2011)

  https://www.theregister.com/2011/10/26/emc_lightning_revealed_a_little/

• Dell building its own Exadata killer (4 October 2011)

  https://www.theregister.com/2011/10/04/dell_poweredge_12g_rna_networks/

• EMC Project Lightning flash cards promised 'this year' (4 October 2011)

  https://www.theregister.com/2011/10/04/project_lightning/

• EMC morphs storage arrays into servers (2 September 2011)

  https://www.theregister.com/2011/09/02/emc_app_servers/

• Why should storage arrays manage server flash? (26 July 2011)

  https://www.theregister.com/2011/07/26/storage_array_manage_server_flash_cache/

• Server flash cache deathmatch: Bring it (31 May 2011)

  https://www.theregister.com/2011/05/31/server_nand_memory_tier/