Electric Cloud creating a Build storm
And forecast looks similar for European expansion
Interview Electric Cloud supplies software to speed up the Build process that's such an important part of modern "agile" software delivery. It uses a sophisticated approach to running the components of a Build in parallel.
And now the company is in Europe.
As its CEO Mike Maciag says: "Software build management and acceleration have been recognised as the next important areas for enterprises to address as part of their Application Lifecycle Management (ALM). We're seeing a steady, dramatic increase in demand across the globe, particularly in Europe."
So, Reg Developer interviewed Andrew Patterson, the European business director at Electric Cloud's new European headquarters in Oxford, to find out what his product was all about.
Reg Dev: ITIL is a successful framework for operational lifecycle management. How does Electric Cloud support ITIL (presumably Service Transition) specifically?
Patterson: I'll answer this in terms of compliance in general, as we don't often encounter ITIL with our customers. When looking at compliance with regards to external mandates or with standards frameworks (such as ITIL or CMMi), there's a common implication for build and release processes: traceability. You must have a documented, repeatable process. In the build world, that means that for every release you must know exactly what was built, when, and by whom, all the way back to the source code. This is inherently difficult when the build infrastructure is manual. Thus, tools like ElectricCommander are needed to both report on the process and to support repeatable, reusable procedures.
Reg Dev: I find it interesting that you don't often meet ITIL in your customers. Why do you think this is, and is familiarity with ITIL increasing?
Patterson: Adoption of process standards/frameworks really seems to vary. With the CMMi framework, for example, it seems to vary by industry. We have run into it within defence contractors, aerospace companies, and some ISVs, but rarely among enterprise IT teams. So that may be the case with ITIL as well. I suspect there may also be some geographical differences in adoption of ITIL. As a company, we began operations in the US and are now gaining momentum and additional customers in the UK and Europe, so may run into ITIL more frequently in the future.
Reg Dev: At the recent CMSG conference, Ryan Lloyd of MKS suggested that build/release was the missing piece in many automated ALM processes. Obviously, 'build' is essential somewhere, but as a vendor in this area, how do you see it integrated as a first-class part of the process?
Patterson: I'd go so far as to say that build and release represents the next wave of innovation and IT investment in ALM. It wasn't so long ago that developers used "homegrown" SCM approaches. I can't imagine anyone doing that now, given the wide variety of both commercial and open source solutions available with established track records.
The last 20 years have seen literally billions of dollars spent on tools and process improvement on the front-end tasks. But with few exceptions, little was invested to support build and release processes. So the bottleneck just shifted to the back end of ALM. That's where we come in.
Reg Dev: So, what is different now?
Patterson: It was difficult to address the back-end problems until two key technologies became available - the web, for access and sharing, and cheap rack-mounted servers for scalable computational power. The web has been available for about 10 years now, but rack-mounted servers just became cost-effective a few years ago, they are now dropping in cost rapidly. A lot of what we have done at Electric Cloud is to figure out how to harness these technologies for builds and other back-end tasks (for example, without our automated dependency management technology there is no reliable way to use server clusters to run builds in parallel).
Another reason builds are garnering attention right now is the move to agile development. One of the key principles of agile development is frequent integration and testing, both of which depend upon reliable and efficient back-end processes. Many organisations are trying to become more agile, but discover that they cannot do that without major improvements in their back-end processes ("you can't do agile with long builds").
For example, Intuit uses ElectricAccelerator to rebuild the QuickBooks software product automatically every 30 minutes all day long, every working day. This allows them to detect and fix errors very quickly, so that overnight production builds virtually never fail.
Reg Dev: So, how do you integrate with the different ALM toolsets that are available?
Patterson: ElectricCommander is our primary point of integration with the rest of the ALM tool chain. The natural handoff point between the front end of ALM and the back end build and release process is at SCM. When code is checked in, build and test cycles should commence in a seamless and automatic fashion. So we have out of the box integrations with tools like IBM Rational ClearCase, Perforce, AccuRev, etc, and are building out integrations with automated test solutions as well.
Reg Dev: And what is your position on Eclipse versus Microsoft's VSTS? Do you support open frameworks such as ALF?
Patterson: As a company we've not taken an either/or approach, though we've certainly observed the rise of Eclipse as more of a framework than is the case with Visual Studio. The growing ecosystem of Eclipse plug-ins is certainly testament to that.
Both of our keystone products support Visual Studio builds, and ElectricAccelerator distributes and accelerates native Visual Studio builds to deliver 8-20x speed improvements over a sequential build. What's more, our ElectricCommander build and release automation solution leverages the Eclipse BIRT reporting framework, which is rapidly becoming industry standard.
We're watching the Eclipse ALF project with great interest. It sounds like a promising way for best-of-breed ALM tools such as ours to interoperate in a standard fashion. But to be quite frank, we haven't seen a demand for this among our customers. We typically work with large enterprises, and so wide adoption of something like the ALF framework may be slow in coming.
Reg Dev: Now, at a slightly lower level, what about feedback of useful management information from the build process?
Patterson: For any team, the build-and-release process is where all the pieces come together - or not - and thus it can be a definitive measure of a project's overall health. The builds become the "heartbeat" of software development. So providing reporting and visibility into build processes (via dashboards, charts, reports, etc.) gives a leading indicator of progress. Even providing red and green lava lamps (red indicates the most recent build has failed) or a "stop light" chart on a big screen monitor gives everyone in the development organisation instant understanding of whether the project is on track or not.
With geographically distributed teams, this information becomes more critical. Historically, nightly builds were the norm, but with teams in multiple time zones, there is no night. Each team in each location has to have information such as "which was the last clean build", "from which branch they should be building", etc.
We've built a reporting engine into our ElectricCommander tool for just this purpose, again based on the Eclipse BIRT framework. We specifically chose this approach so teams could extend the reporting to extract the information most useful to them, and to also enable them to pull that data into the commercial reporting product they already use (whether something like Actuate or Crystal Reports, or even Microsoft Excel).
Reg Dev: Does your Build toolset enable increased transparency through from physical code to the business requirements?
There is more and more pressure to provide full traceability of code backward from the released application (or "gold master") all the way back to initial code creation, for example: What went into this release? What bugs are fixed? What packages are affected?). So providing a manifest or "bill of materials" is essential (and fairly impossible with a completely manual build infrastructure).
Reg Dev: OK, now at the nuts and bolts level, how do you actually make Build more efficient?
We tackle the build and release process at three levels – we automate it, accelerate it, and analyse it. Looking at each in turn:
- Automation: Most organisations have a homegrown solution for invoking and managing build and release tasks, comprised of scripts and servers and perhaps open source tools. The problems with a manual approach are numerous: they typically require manual intervention for ongoing maintenance, they don't scale to support multiple projects or teams, and they're inherently brittle. We've developed the ElectricCommander tool to address these issues at an enterprise level. ElectricCommander distributes individual jobs in parallel across multiple servers for faster throughput and efficient resource utilisation. Tasks can be run on demand, on a defined schedule, or continuously (i.e. a build is invoked whenever something is checked into a specified branch). It's based on a three tier architecture that can scale to manage any size project and any volume of tasks. It will also help teams respond nimbly to change, requiring minimal script changes to get started or make changes to the system on the fly. It also provides the reporting that manual approaches typically lack, to provide visibility into key trends and status of build and release tasks.
- Acceleration: This is where we hold several patents. The idea of distributing a build across multiple machines is deceptively simple. In reality, there are thousands of dependencies in a build of any reasonable size, so when you run processes in parallel a missing or undetected dependency will cause the build to fail or recompile in the wrong order. Our patented conflict detection and correction technology determines exactly which files were used to build every object file, library or executable, such that when build steps are run out of order, it automatically re-runs them in the correct order. So ElectricAccelerator produces a correct build, even when scaling to dozens of nodes in the cluster. With this technology, we typically see 8-12x speed improvements, with some customers realising as much as a 20x improvement.
- Analysis: In most organisations builds are black boxes. Unless it's a brand new project, no one is terribly sure what's happening in the build and they probably don't want to. In many organisations, being the "build master" is a punishment. We've shone a light on the build in ways that empower both developers and managers to really understand what's happening and take action to optimise, repair, or fine-tune the build. Our ElectricInsight tool mines the output of ElectricAccelerator to create a graphical representation of the build structure for performance analysis. You can literally, with a click, see for each job which files were used, and where dependencies may lie. Without this, doing this type of analysis would require pouring over thousands of pages of log files. At a higher level, our ElectricCommander tool features reports covering individual projects as well as cross-project views.
Reg Dev: I notice you mention 'several patents'. As an aside, this is a hot topic amongst developers. What protection does this really give you? Surely, a big company could break your patents and enmesh you with the slowest patent lawyers in the world – by the time you'd won, you'd have lost the business? And, aren't granted patents in the public domain where they can inspire genuine 'reverse engineers'?
Patterson: Patents are certainly a topic of some controversy. Many engineers worry that patents may impede the adoption of new technologies, particularly in fast-moving areas such as software. Others wonder how much protection patents really provide, and argue that the greatest barrier to entry for competitors is the amount of functionality in our products: it would take a competitor a long time to re-create what we've done, even without patent issues, and by then we will have implemented a lot of additional functionality. But the investment community believes that patents add value to the company, and we think they will also help us to defend against the most flagrant abuses of our intellectual property assets. Finally, they give us negotiating chips if other companies come to us with patent claims. The patent system definitely isn't perfect, but we still think it has value.
Reg Dev: Finally, does all this mean that Build/Release is now a finished technology? Or is there scope for improved features and even more efficiency? If so, please describe...
Patterson: ElectricAccelerator (for accelerating builds by distributing them across a cluster of machines) is the more mature technology, but we're always looking for ways to leverage this expertise. We've built significant knowledge around distributed processes and cluster computing – what are other processes we can accelerate or streamline using a distributed approach? We'll see.
On the other hand, the ElectricCommander product is an inherently extensible tool. At its core, it is a process automation framework. Build is one process among many. We've already developed integrations with SCM tools, testing tools, and are looking at the next wave of integrations with defect tracking tools and solutions at the deployment end. The more the build stage is tied into the overall development lifecycle at a process level, the more overall efficiencies developers will realise. That's been a challenge historically – ALM has traditionally been comprised of a series of disconnected tools and people. Now that vendors such as ourselves are bringing automation to the build and release cycle, we in the larger development tool industry have the technology and process knowledge to deliver on the promise of integrated ALM now. ®