Newvem fluffs up penny-pinching control freak for AWS cloud

Startup Newvem uncloaked from stealth last November with Cloud Care, a control freak that reaches into CloudWatch and other Amazon Web Services management APIs to help companies figure out what they are spending and where they might be wasting resources and money. The service has tracked over 125,000 EC2 instances over the past ten months, representing $200m in spending on Amazon's compute cloud, and – here's the kicker – has saved customers more than $80m over that time on their EC2 spending. Now Newvem is launching a simpler, companion version of Cloud Care, and better yet, this one is free.

The irony is, Amazon loves these guys – or so Newvem co-founder Zev Laderman tells El Reg. It is hard to believe, but Laderman says Amazon wants customers to use AWS services like EC2 efficiently and have a good experience. As is the case with a real server, however, running a virtual one efficiently is the responsibility of both the server maker and the customers.

"When you spend too much, this is not the fault of Amazon," says Laderman. "Amazon gives you the tools and you have to accept shared responsibility for the management model."

That's where companies like Newvem step in. Newvem sucks all of the performance and cost data out of services like EC2 through the Amazon API straw and pipes it into a real-time analytic engine that chews on data from thousands of customers and a number of EC2 instances that is several orders of magnitude higher.

Newvem was founded in 2010 by Laderman, who is CEO, and Ilan Naslavsky, who is CTO. Both had the same roles at Aduva, a company that had built a patch management system for Linux and Solaris servers called OnStage. Sun Microsystems bought OnStage in February 2006 and eventually transformed it into Sun Connection and mashed it up into Sun's xVM Ops Center control freak for physical and virtual servers. Bits of these tools live on inside the Oracle Enterprise Manager control freak.

The Cloud Care service from Newvem has been in beta since last April and went live in November. The simpler, freebie service that Newvem is launching this week is more suited to newbie EC2 users, but will also work as a loss leader to get customers dependent on the Newvem service early on, so that they will want to trade up to the paid product as they grow.

The Cloud Care service does not compete with the OpsWorks variant of the Chef tool from Opscode that Amazon Web Services fluffed up earlier this week. OpsWorks automates the deployment of applications onto the EC2 service using Chef and can also link them to the S3 storage service and other platform-level services such as the Relational Data Service (RDS) database service.

OpsWorks is literally a control freak, while Newvem Cloud Care is, by analogy, more of a mother-in-law, telling you what you are not doing right and how you ought to be saving more money. Cloud Care doesn't just mash up and chew on all of the data culled from the Amazon APIs; it also mixes in crowdsourced configuration information so that can it compare your workloads and virty server setups to peers running similar software with acceptable results, and thereby point out ways to help you save dough.

The OpsWorks and Cloud Care services are complementary, and given the increasing popularity of the Newvem service, it would not be surprising to see Amazon snap it up, too, to help EC2 customers become more frugal. It could then extend its data-munching and bean-counting approach to other AWS services – as contrary to logic as that might seem for an IT vendor.

Call it Bezos' Law: Cut price until it hurts to drive out competition and let Wall Street worry about profits, start your own space transport company and conquer the Moon. Several generations – human ones – later, Amazon is the world's biggest company store and it can finally lower the boom on us all. Or better still, do it gradually without any of us realizing it.

With the freebie version of Cloud Care, which is itself a cloud-based SaaS program running on AWS, Newvem is offering a kind of healthcare check for those small businesses that are spending under $5,000 a month on EC2 instances. Based on Newvem's sampling of its customers, these smaller AWS users represent about 75 per cent of the customer base. The top 1 per cent of the AWS base spends more than $1m a month on EC2, based on that sample.

With the latest update, Cloud Care can also do analysis on S3 storage, but this is not in the freebie version. Newvem has also shrank the snapshot window down from one hour to 15 minutes, which means it is chewing on data four times as often, and it is likely that it will continue to shorten that window as much as economically possible.

The freebie Cloud Care tool is meant to be a high-level tool. It will do regular checkups on your EC2 images and tell you if your costs are out of whack with your workloads or if you are taking operational risks with the way you have set them up. If you want to solve your own AWS problems and have the skills to do so, then you can get by with the Free Edition.

The Premium Edition of Cloud Care, which is not free, makes recommendations to help you solve issues. It also helps you decide if you should switch from on-demand to reserve or spot instances to save money without affecting your applications. In addition, the Premium Edition can also track S3 and EC2, and there will eventually be an Enterprise Edition of the Cloud Care service that will have the kitchen sink tossed in.

The big issue for SMB shops using EC2 capacity, says Laderman, is overcapacity. About 15 percent of customers overprovision their EC2 images, and about half of them have security issues they may not even be aware of.

Of course, any system that is running at absolute peak efficiency is by its very nature brittle when it comes to response to changes, so you can't blame these customers for being cautious. But in a cloud computing world where you can change the size of the virtual machine on the fly (and increasingly in hypervisors without taking the VM offline to do so), such overprovisioning is no doubt a waste of money. So long as Amazon has excess capacity to sell you so you can quickly expand, then you are fine. The situation is different if you are perfectly provisioned; you hit a workload spike and there is no more CPU for you to buy.

Newvem is still working out its pricing for various components of the Premium Edition, but generally speaking, the price is anywhere from a tenth of a penny to 3 pennies per usage hour for EC2 and S3 services, for those enterprise-class customers that are spending thousands to millions of dollars per month buying EC2 capacity and S3 capacity to match it. For smaller customers, Newvem is packaging up bundles of usage capacity in tiers and working out how well or poorly these map to typical customers.

Laderman says that the company could have just charged a flat percentage of a customer's monthly AWS spending, but that many of those large customers are growing at anywhere from 3X to 70X per year, with 10X being typical. With that kind of growth, customers would very quickly resent paying for the Newvem service. Instead, Newvem is trying to base its pricing on the number of hours of a given EC2 service, and so long as the company helps customers save money by right-sizing EC2 images, it can also make a buck. ®