Huawei hugs open-sourcey Alluxio: Thanks for the memories

China giant + OS software in big data analytics acceleration scheme

Little girl embraces robot. Photo by Shutterstock

Huawei has announced a Big Data analytics acceleration scheme using its FusionStorage product and Alluxio open source software; which seems to be the canine genitalia du jour for speeding up lethargic analytics queries.

Alluxio is the renamed Tachyon Nexus, an Andreessen Horowitz-backed startup.

FusionStorage is Huawei's distributed software-defined storage system. The latest FusionStorage 6.0 supports distributed block, file, and object storage, with classification, encryption and deduplication.

Alluxio's software is a memory-centric, virtual distributed storage system. It functions as a ginormous local cache for a remote storage system, such as a set of HDFS nodes. It is based on a cluster of local nodes which are accessed by compute nodes running Big Data analytics jobs and queries.

The software provides:

  • Tiered storage (memory, flash, disk) with automatic data placement
  • Single namespace with transparent naming
  • Native S3, Google Cloud Storage, Open Stack Swift, Alibaba OSS, Microsoft Azure Blob store integrations
  • Fuse Connector, key-value interface
  • One command cluster deployment
  • Metrics reporting
Alluxio_1
Alluxio_2

Alluxio supports lots of different underlying (remote) storage systems include HDFS, Gluster, S3, OpenStack, GCS, NFS, OrangeFS, IBM Spectrum Scake, Ceph, Isilon and others.

Alluxio_3

Baidu ran Spark queries 30 times faster with Alluxio. Batch queries that took 15 minutes completed in less than 30 seconds and a 1000-user Alluxio cluster provided more than 50TB of RAM space.

Barclays accelerated Spark jobs from hours to seconds by interposing an Alluxio set-up between the query-running compute nodes and a Teradata data repository.

Sounds like hot stuff. We think that all this suits Huawei and its IoT/cloud strategy very well. Check out the Alluxio project at www.alluxio.org and run through an Alluxio slide deck here. ®




Biting the hand that feeds IT © 1998–2018