Squeezing in little Quake between builds? Not any more: Facebook Bucks up Java compile tool

Citing the social network's need for speed, Facebook senior software engineer Jonathan Keljo says the company's developer tools group has revisited how its Buck open-source build tool compiles Java code and made it faster still.

As such a new version of the software was released today for programmers to fetch and play with: it's under the source_only ABI generation mode.

Developed in 2012 and open sourced in 2013, Buck was created to hasten Android build times and has since been adapted to work with iOS.

"The tools at that time available for Android were slowing us down," said Keljo in a phone interview with The Register. "So Buck was created to try to fix that."

Compiling code can take time, sometimes so much so that Keljo in a draft blog post provided to The Register likened slow build cycles to an "exhausting, soul-killing fountain of frustration."

Asked if we were talking about idling away hours waiting for the Java compiler, Keljo said the time savings would be measured in seconds. "If the builds were in hours, there would be pitchforks and torches," he said, noting that build times interrupt the edit-run-compile cycle.

Given that web devs tune sites to shave seconds from page load times for fear of losing online visitors to boredom, Facebook's fanaticism for fast is perhaps more understandable.

And while Facebook's gains may be but seconds due to its longstanding focus on optimization, less sophisticated users of Java could recoup minutes or hours, maybe.

Translated

To save its Android apps team and others crafting Java programs from waiting while source code to get translated into JAR files, Keljo and colleagues have made several changes recently to the way Buck works.

Facebook's Android app became so large that something had to give. So the developer tools team modified Buck's Exopackage feature, which obviates the need to reload the app completely every time there's a code change.

The Exopackage creates an app file (.apk) with minimal Java code and puts remaining code into different .dex files, so they can be moved to the mobile device where the Android app is running only once they've been changed.

The latest improvement to Exopackage involves support for sideloading Android resources and assets. So as was previously possible with binaries, Exopackage now transfers images and the like only when altered.

"Buck encourages you to break up your app into smallish modules..and to describe to Buck how those are related," said Keljo.

Buck optimizes builds in part through build rules supplied by the developer, which lead to smarter decisions about compilation. It relies on stub JARs, stripped down versions of JAR files that it uses for more efficient code crunching.

This process has been improved through the addition of Java class usage tracking, which evaluates Java classes in modules and recompiles only modules that depend on the changed class.

Keljo said this has cut down build rules by 35 per cent. He didn't have a time metric for the change, but said, "The fastest build is the one you don't have to do."

Buck also improves the Java build pipeline through a limited form of parallelism.

Dependencies

While some of the compilation process has to happen in a serial fashion, Buck has enough information through its dependency rules to figure out which rules don't have dependencies. It can then build those rules into JAR files at the same time it processes other rules.

"It turns out that the Java compiler part-way though its job, it actually has all the information you need to create the stub JAR," said Keljo. "We're parallelizing it, even though the rules seem like they can't be parallelized because they depend one another."

Finally, Buck incorporates further optimization through informed guessing, based on conventions like package names being usually lower case and class names starting with a capital letter.

Previously, before building a rule into a JAR file, the build tool would check the stub JARs file for its dependences and then the stub JARs summarizing those dependences, until it had a complete graph of connected code.

But many of those graph connections aren't necessary because the Java interface for a module can often be inferred from the source of the module.

"What we realized is that if you don't have to build the dependencies before you can build any rules, you can crunch it down," said Keljo.

When Facebook's Instagram app for Android was built using these improvements, the build time was reduced by 30 per cent.

And now every developer of Android apps can savor moments stolen by Buck. ®