Enterprise storage: A history of paper, rust and flash silicon

The others became less profitable, fell into debt and collapsed or were taken over. At one time there were more than 200 disk drive makers. Now there are just three: Seagate, Toshiba and Western Digital/HGST.

Seagate's ST-412 disk driveE

The rise of disk drive arrays and their falling cost/GB meant that they could take over backup data storage duties, especially if repetitive data in backup data sets could be removed by deduplication technology. This was a devastating blow to the tape backup industry, leading to rapid format consolidation. Now there are effectively only two mainframe tape suppliers with proprietary data formats left, IBM and Oracle, and effectively just one server computer format, LTO. This is the Linear Tape Open format, now in its sixth LTO-6 generation. The LTO format is owned and developed by a three-member consortium; HP, IBM and Quantum.

While disks were loaded with more and more data and technology enabled them to be read and written to faster and faster, one huge problem remained. Disks could, and did, fail, causing all the stored data to be lost. The problem was avoided by making extra copies of data. The data was mathematically processed so that reduced amounts were needed, lowering data protection costs, with RAID technology - Redundant Arrays of Independent Drives. Different RAID schemes include RAID 0, 1, 2, 4, 5, 10, with each individual scheme optimised for data access speed, data protection surety, and capacity needs or combinations of these.

With RAID schemes the contents of a failed drive are rebuilt on another drive using the RAID data held on the surviving drives. But such RAID rebuilds took longer and longer as drive capacities grew.

HGST 7K500 3.5-in hard disk drive

As the number of drives in arrays grew to the hundreds and then past the thousand mark, it became necessary to protect against a second drive failure when recovery from a first drive failure was under way, and so the RAID 6 scheme was devised.

Disk drive growth rapidly increased the total amount of stored digital information and researchers reckoned digital storage capacity passed analogue capacity in 2002 and the rate of increase continues to accelerate.

IDC Digital Universe growth in Exabytes (IDC/EMC June 2011)

As the speed at which the disk ran was critical to how quickly data could be read or accessed, so the manufacturers pushed their technology to deliver ever faster rpms. Disk speeds continued to increase until they reached an effective limit of 15,000rpm. Any faster, and the disk platters risk flying apart under the centrifugal force and from vibration. Lower speed drives could store more data and may rotate at 5,400rpm. That meant that the time taken to read and write all the data to and from a drive went up as drive capacity increased, and this became more and more a limiting factor.

The problem is that computer processor speeds, impelled by Moore's Law, rose and rose and rose, leaving disk I/O speed far behind and utterly unable to catch up. A move to 2.5-inch drives meant that more could be put in a disk drive enclosure to increase the overall I/O rate, but not dramatically so.

This inability to keep up with host computer speeds provided the backdrop for what we are seeing now - the rise of solid-state NAND flash storage, with no moving heads, in fact no read/write heads at all.

NAND flash

Invented by Dr Fujio Masuoka at Toshiba around 1980, NAND flash is organised as cells in blocks which must be written to simultaneously. So it is not, strictly speaking, a random access device.

Schematic diagram of flash memory cell

One type of flash, NOR (named after the Not Or logic concept) is used in mobile phones and other devices as a form of read-only memory. NAND (Not And logic) is used for data storage in cameras, USB sticks, tablet computers, notebooks and a vast range of other devices needing tiny, low-power and persistent data storage with faster access than disk.

A semiconductor technology, with no mechanical parts, it is considerably smaller than disk at equivalent storage amounts. A 1TB USB stick fits in the palm of your hand and is much lighter than the equivalent 4-CD case stack sized 3.5-in hard disk drive.

A quick look at optical storage

CDs and DVDs have also been used for archiving data storage but their widespread use has been held back by their inherent disadvantages. They enjoy relatively low capacity and slow write-speeds compared with hard disk drives and tape is cheaper than both disk and CD/DVD optical for storing bulk data. Disk is also faster for writing and reading backup data. All of which means that optical disks are only found today in niche markets.

Meanwhile, there have been persistent attempts to develop holographic storage technology as a way of providing archival duration and high-capacity optical technology. They have all failed. So much so that it's now feasible to consider archival storage using very cheap flash memory, TLC flash.