Enterprise storage: A history of paper, rust and flash silicon

Has anything really changed since the punch card era?

Boost IT visibility and business value

The others became less profitable, fell into debt and collapsed or were taken over. At one time there were more than 200 disk drive makers. Now there are just three: Seagate, Toshiba and Western Digital/HGST.

Seagate ST-412 disk drive

Seagate's ST-412 disk driveE

The rise of disk drive arrays and their falling cost/GB meant that they could take over backup data storage duties, especially if repetitive data in backup data sets could be removed by deduplication technology. This was a devastating blow to the tape backup industry, leading to rapid format consolidation. Now there are effectively only two mainframe tape suppliers with proprietary data formats left, IBM and Oracle, and effectively just one server computer format, LTO. This is the Linear Tape Open format, now in its sixth LTO-6 generation. The LTO format is owned and developed by a three-member consortium; HP, IBM and Quantum.

While disks were loaded with more and more data and technology enabled them to be read and written to faster and faster, one huge problem remained. Disks could, and did, fail, causing all the stored data to be lost. The problem was avoided by making extra copies of data. The data was mathematically processed so that reduced amounts were needed, lowering data protection costs, with RAID technology - Redundant Arrays of Independent Drives. Different RAID schemes include RAID 0, 1, 2, 4, 5, 10, with each individual scheme optimised for data access speed, data protection surety, and capacity needs or combinations of these.

With RAID schemes the contents of a failed drive are rebuilt on another drive using the RAID data held on the surviving drives. But such RAID rebuilds took longer and longer as drive capacities grew.

hitahi t7k500 hard disk drive

HGST 7K500 3.5-in hard disk drive

As the number of drives in arrays grew to the hundreds and then past the thousand mark, it became necessary to protect against a second drive failure when recovery from a first drive failure was under way, and so the RAID 6 scheme was devised.

Disk drive growth rapidly increased the total amount of stored digital information and researchers reckoned digital storage capacity passed analogue capacity in 2002 and the rate of increase continues to accelerate.

IDC Digital Universe growth

IDC Digital Universe growth in Exabytes (IDC/EMC June 2011)

As the speed at which the disk ran was critical to how quickly data could be read or accessed, so the manufacturers pushed their technology to deliver ever faster rpms. Disk speeds continued to increase until they reached an effective limit of 15,000rpm. Any faster, and the disk platters risk flying apart under the centrifugal force and from vibration. Lower speed drives could store more data and may rotate at 5,400rpm. That meant that the time taken to read and write all the data to and from a drive went up as drive capacity increased, and this became more and more a limiting factor.

The problem is that computer processor speeds, impelled by Moore's Law, rose and rose and rose, leaving disk I/O speed far behind and utterly unable to catch up. A move to 2.5-inch drives meant that more could be put in a disk drive enclosure to increase the overall I/O rate, but not dramatically so.

This inability to keep up with host computer speeds provided the backdrop for what we are seeing now - the rise of solid-state NAND flash storage, with no moving heads, in fact no read/write heads at all.

NAND flash

Invented by Dr Fujio Masuoka at Toshiba around 1980, NAND flash is organised as cells in blocks which must be written to simultaneously. So it is not, strictly speaking, a random access device.

Flash Memory cell schematic

Schematic diagram of flash memory cell

One type of flash, NOR (named after the Not Or logic concept) is used in mobile phones and other devices as a form of read-only memory. NAND (Not And logic) is used for data storage in cameras, USB sticks, tablet computers, notebooks and a vast range of other devices needing tiny, low-power and persistent data storage with faster access than disk.

A semiconductor technology, with no mechanical parts, it is considerably smaller than disk at equivalent storage amounts. A 1TB USB stick fits in the palm of your hand and is much lighter than the equivalent 4-CD case stack sized 3.5-in hard disk drive.

A quick look at optical storage

CDs and DVDs have also been used for archiving data storage but their widespread use has been held back by their inherent disadvantages. They enjoy relatively low capacity and slow write-speeds compared with hard disk drives and tape is cheaper than both disk and CD/DVD optical for storing bulk data. Disk is also faster for writing and reading backup data. All of which means that optical disks are only found today in niche markets.

Meanwhile, there have been persistent attempts to develop holographic storage technology as a way of providing archival duration and high-capacity optical technology. They have all failed. So much so that it's now feasible to consider archival storage using very cheap flash memory, TLC flash.

The essential guide to IT transformation

Next page: All types of flash

More from The Register

next story
The Return of BSOD: Does ANYONE trust Microsoft patches?
Sysadmins, you're either fighting fires or seen as incompetents now
Microsoft: Azure isn't ready for biz-critical apps … yet
Microsoft will move its own IT to the cloud to avoid $200m server bill
Oracle reveals 32-core, 10 BEEELLION-transistor SPARC M7
New chip scales to 1024 cores, 8192 threads 64 TB RAM, at speeds over 3.6GHz
Docker kicks KVM's butt in IBM tests
Big Blue finds containers are speedy, but may not have much room to improve
US regulators OK sale of IBM's x86 server biz to Lenovo
Now all that remains is for gov't offices to ban the boxes
Gartner's Special Report: Should you believe the hype?
Enough hot air to carry a balloon to the Moon
Flash could be CHEAPER than SAS DISK? Come off it, NetApp
Stats analysis reckons we'll hit that point in just three years
Dell The Man shrieks: 'We've got a Bitcoin order, we've got a Bitcoin order'
$50k of PowerEdge servers? That'll be 85 coins in digi-dosh
prev story


5 things you didn’t know about cloud backup
IT departments are embracing cloud backup, but there’s a lot you need to know before choosing a service provider. Learn all the critical things you need to know.
Implementing global e-invoicing with guaranteed legal certainty
Explaining the role local tax compliance plays in successful supply chain management and e-business and how leading global brands are addressing this.
Build a business case: developing custom apps
Learn how to maximize the value of custom applications by accelerating and simplifying their development.
Rethinking backup and recovery in the modern data center
Combining intelligence, operational analytics, and automation to enable efficient, data-driven IT organizations using the HP ABR approach.
Next gen security for virtualised datacentres
Legacy security solutions are inefficient due to the architectural differences between physical and virtual environments.