Flippin' ECK, ours is the 'official' Elasticsearch experience for Kubernetes – Elastic

Elastic presented Elasticsearch for Kubernetes at an event in London this week, as well as explaining why it has acquired Endgame, an endpoint security specialist.

The search company's core product is the Elastic Stack, including the ElasticSearch database, optimised for fast queries and analytics across a large volume of diverse data types, and Kibana for exploring and visualising that data. It is used for powering site or enterprise search, log analysis and security alerts, geographic search and more.

The Mountain View firm drew nearly 400 developers and IT administrators to its London event, with the two big topics being progress with Elastic Cloud on Kubernetes (ECK), and the company's new security based on its recent acquisition of Endgame.

There is also the matter of the Elastic's ongoing dispute with Amazon Web Services (AWS) over the AWS Elasticsearch product, forked from Elastic's open source code. Posters stuck to the stairway at the event proclaimed "The official Elasticsearch experience," likely a dig at Amazon's product.

ECK, which is currently in beta, supports Elasticsearch, Kibana (data visualisation) and APM (Application Performance Monitoring) Server. It can be installed on Red Hat OpenShift, Amazon EKS (Elastic Kubernetes Service but no relation), or GKE (Google Kubernetes Engine). Products currently on offer are Elasticsearch, Kibana (data visualisation) and APM (Application Performance Monitoring) Server – though others will follow. "Things like apps search, enterprise search, site search and many more will also be managed by ECK," said product manager Roy Zanbel at the event.

What added value do you get from K8s? It is the usual story: Elastic is convinced that this is the right direction but there are also some frustrations with the current state of play.

"It gives this dream of containerization and orchestration," veep of worldwide solutions architecture Steve Mayzak told the Reg. "The benefits will come with time but the K8s community needs to build in better support for stateful services. Here's the challenge I see with K8s adoption."

The company's Elastic Cloud Enterprise "has all this orchestration, and it's tried and true and running thousands of clusters in production already" – but does not use K8s. "It does use Docker under the covers, but it was built before K8s was really a thing," says Mayzak.

The Endgame acquisition

The company has just acquired Endgame and now sells a full security solution. Why? The reason is that Elastic is focused on being able to track and analyse large amounts of data, and in a security context that means the ability to monitor and store data from endpoint agents. "That data is typically not stored for a long time," says Mayzak. "If you go with another vendor you typically get seven days of retention, but the average dwell time for an intruder is 90-plus days. Those two things don't make sense … and the fact that Elastic is so fast allows our customers to react."

Endgame security was already partly built on Elasticsearch, but the company has more integration work to do. “We’re going to integrate it with our SIEM [Security Information and Event Management] user interface so the two are basically the same thing,” said Mayzak.

The company has also changed the pricing model, so that you do not license per endpoint, but pay for your server-side usage.

Open source and AWS Elasticsearch

What about open source? Elastic, along with other companies such as MongoDB and Redis, is among those companies which has tried to address what it sees as a problem with big cloud providers profiting from their code without paying licence fees. "Open source companies are trying to protect their future by changing the license, to make it so that a cloud provider can’t just use it without you getting some benefit," says Mayzak.

The Elastic License prohibits use as "any software as a service … or as part of an application services offering".

Originally, much of the Elasticsearch code and most of the source code was offered under the permissive Apache 2.0 license, enabling AWS to launch its Elasticsearch service based on that code but with its own modifications.

"If you look at Amazon with their AWS Elasticsearch thing, it's just a bare bones version. A comparison between AWS ES and Elastic Cloud is a night and day difference," Mayzak claims. That claim is backed up to some extent by posts like this one, where user Nick Price says he is "absolutely stunned at how poor Amazon's implementation is."

The Reg asked AWS for a comment on the post.

One of reasons Elastic described the AWS version as "bare bones" is that Elastic's software has not for a long time been 100 per cent open source. The X-pack for Elasticsearch, which offers a number of important features, was not available for AWS to fork.

In March 2019, AWS released its own "Open Distro for Elasticsearch", under the Apache 2.0 license, stating: "This is not a fork; we will continue to send our contributions and patches upstream." It sounds like a fork to us.

The Open Distro adds features covering security, monitoring, alerting and analysis – in other words, covering many of the requirements which could otherwise be met by X-Pack.

Amazon's Adrian Cockcroft, veep of cloud architecture strategy, posted about Amazon's rationale for creating the Open Distro, complaining about "open source maintainers … muddying the waters between the open source community and the proprietary code they create to monetize the open source."

He said that it was no longer clear which parts of ElasticSearch were open source and which were proprietary. The Open Distro is presented as "100 per cent open source", making it easy for users to implement self-managed deployments for free.

Is Elastic troubled by these developments? "It's not troubling in the sense that, it's open source, so they're free to do it,” says Mayzak. “As soon as you open source, you give up that right to make these decisions for others. But there's a potential for confusion, because they're using our name."

Along those lines, the company earlier this month filed a lawsuit (PDF) against AWS, alleging trademark infringement.

Data, data, data

What's next for Elastic, K8s aside? One of the key areas will be Internet of Things (IoT) and dealing with the vast amounts of data generated thereby. "The best back end for a lot of these use cases is Elasticsearch," he claims.

Mayzak sees opportunities in fields like automotive. "We're being used by an auto manufacturer, they put an Elasticsearch data collector in each of their vehicles." There is work to do though in the area of edge computing, since it makes no sense to send all the data to the cloud. One solution is to run Apache Kafka on the vehicle itself to process the stream of data.

It seems a safe bet that the amount of data the IT industry wants to analyse will continue to grow, which is potentially good news for Elastic if it succeeds in fending off competing solutions from the likes of AWS, Microsoft Azure and Google Cloud Platform. ®