RIP Netezza, IBM’s FPGA-powered data warehousing dream
Once mighty business killed by cloud computing
IBM is finally pulling the plug on Netezza, a family of data warehousing and analytics appliances that have been around for the past 15 years.
Most of the models have reached the end of support, and no new Netezza boxes will ever be made - IBM has effectively killed the brand, and salted the earth.
This sounds surprising, when you consider that IBM purchased the Delaware business in 2010 for $1.7bn.
"These appliances have served you well, but with advances in technology it is a perfect time to upgrade," the company said in a short statement on a page which has since been removed.
It all started so well: Netezza, established in 1999, designed a massively parallel system specifically to run complex analytics on very large data volumes. In its heyday, it was competing against the likes of Teradata, Oracle and HP. It had a great Wall Street debut. And yes, IBM paid megabucks for it.
"It was solving a real problem, the performance issue, and they were faster than the competition," Mathias Golombek, CTO at German database developer Exasol, told El Reg.
Netezza's products got their reputation for performance thanks to the use of field-programmable gate arrays (FPGAs) in a patented hardware acceleration process - long before these types of custom chips found mainstream popularity. FPGAs are usually programmed to do just one thing, but do it well. These FPGAs were fine-tuned to execute analytic queries.
After Netezza got acquired, it traded custom servers for standard IBM blades, making the overall product cheaper, but FPGAs remained a part of the design.
During its lifetime, the Netezza business produced a bunch of appliance models, including Skimmer (version N100-1), TwinFin (N1001 versions) and Striper (N2001 versions). Big Blue's PureData System for Analytics (N2002) has another year to go, and N3001 has another four years.
The problem with all of these boxes is they can't handle one important trend in IT – cloud computing.
"Netezza was a hardware-accelerated appliance, utilizing an integrated SPU with blade, drive and FPGA all in one place. While a clever bit of innovation that we loved and that has served thousands of customers well, it was only available in one infrastructure form factor," Christopher Reuter, sales leader at IBM, wrote in a blog post in 2018.
"When clients started asking for cloud and virtualized versions, IBM quickly realized that the integrated SPU was not able to be replicated for most clients. This custom hardware was too custom to be repeated by clients in their own data centers or by other cloud providers."
Reuter also said some of the custom parts for N1001 and N2002 systems were simply no longer available. IBM hadn't released new Netezza hardware since 2014.
"You cannot sell complex and specialized, expensive hardware to customers anymore, they just don't want it," Golombek said.
"I think even more important is the fact that the architecture was quite limited. They had a master node, for instance, so the parallelization is not very good. If you execute a sequence, you have a lot of intermediary steps written to disk, that also limits your concurrency."
The Big Blue is in the middle of a mammoth effort to shift customers onto IBM Integrated Analytics System (ISA), which is positioned as "the next generation Netezza-based appliance" but is actually very different: Netezza hardware was powered by FPGAs while the Integrated Analytics System relies on solid-state storage and memory for its performance boost.
ISA replaces the trusty Netezza software - a heavily modified PostgreSQL - with IBM's Docker-friendly Db2 Warehouse, and its BLU Acceleration in-memory technology. There's also added support for operational analytics and embedded Apache Spark, and IBM claims the new box is up to five times faster - but upgrading to it could be a pain for some.
Big Blue says when migrating the old Netezza workloads to IAS, "most customers will achieve up to 90-95 per cent compatibility" - a statement so full of ambiguity it's actually impressive.
IBM also offers IBM Db2 Warehouse on Cloud (Db2WoC) as a SaaS alternative, and again has said it can support "nearly all" Netezza workloads. "For the select few edge cases, we offer free tooling to help you make a smooth transition," reads the marketing copy for IAS.
The main advantage of the new Db2 Warehouse ecosystem is the fact that the on-premises appliance and the cloudy software can work together, and workloads can be shifted around depending on the needs of the business - and Netezza never made the transition to the cloud, grounded by its once innovative custom silicon. ®