Hadoop rigs are really supercomputers waiting for the right discipline
So says IBM, as it re-tunes its Spectrum HPC-wranglers for clustered workloads
IBM reckons the rigs assembled to run the likes of Hadoop and Apache Spark are really just supercomputers in disguise, so has tweaked some of its supercomputer management code to handle applications that sprawl across x86 fleets.
As explained to The Register by IBM's veep for software-defined infrastructure Bernie Sprang, apps resting on clusters need to optimise workloads across pools of compute and storage resources, and can benefit from templates that make it easier to deploy without dedicated hardware. That second point, Sprang says, is important because he's starting to see “cluster creep”, a phenomenon whereby different teams inside an organisation each cook up their own compute clusters that could perhaps be shared instead of hoarded.
Big Blue thinks workload deployment and scheduling tools can give organisations the chance to acquire and operate fewer clusters without running out of data-crunching grunt.
Enter the new IBM “Spectrum” tools, comprising just such a resource-sharing tool in the form of the “Conductor” product that can “templatise” clustered apps. There's also the “Spectrum LSF” tool for workload scheduling. IBM already offers Spectrum Storage, the filesystem formerly known as GPFS, to support large-scale workloads.
Big Blue's wrapped this new set of release up in software-defined jargon, but isn't going after mainstream workloads. It does think the Spectrum range is a fine idea for those contemplating cloudy or hybrid cloud analytics rigs, as it will happily span on-premises and public clouds.
Sprang likes to position the Spectrum suite as akin to a hypervisor, inasmuch as it sits between the server and the workload. But that's where the similarities end: this lot is aimed squarely at clustered apps and Big Blue hopes its high-end pedigree will interest those now wrestling with hyperscale workloads. ®
Sponsored: Beyond the Data Frontier