Exploring AWS CodeGuru: New automated code review has smart features – but Java-only
Do the sums before adopting
Analysis AWS's automated code review and profiling tool CodeGuru drew substantial interest when it was announced at the vendor's re:Invent shindig in Las Vegas this month.
Developers like the idea of automated code review and even small amounts of time saved or bugs averted are of considerable value. Modern editors like Eclipse or Visual Studio Code already provide coding assistance with common errors, auto completion and auto formatting; if you try coding in Notepad without this kind of help you soon discover how useful it is.
CodeGuru, needless to say, promises the magic of machine learning, based on "hundreds of thousands of internal projects" as well as "over 10,000 open source projects on GitHub."
There is also CodeGuru Profiler, bundled with it but really a separate tool, which helps to identify expensive lines of code. Profiling is nothing new, but it can help identify inefficiencies that yield big improvements when fixed. In a presentation at re:Invent, attendees were told of examples where the number of instances required to run an application reduced by 75 per cent after optimising a thread contention issue, or by between 40 per cent and 67 per cent in another case.
CodeGuru is in preview and we took a quick look. There are a number of limitations. Both the code review and the profiler only work on Java applications. The code review only supports either GitHub or AWS CodeCommit repositories, and the profiler requires an application hosted on Amazon EC2, or containerised on ECS (Elastic Container Service) or Fargate.
El Reg gives it a spin
Getting started with Reviewer is a matter of signing into AWS, heading to the CodeGuru dashboard, and "associating" a GitHub repository. This worked fine for us when signing into GitHub as the repository owner, but the documentation recommends creating a dedicated CodeGuru user on GitHub, since the reviews are posted as comments by the account used for association, which can be confusing.
We tried doing this by setting up a separate GitHub user as collaborator on the project, but then the repository association failed. It probably works OK with a GitHub organisation, which is the more common scenario in business. What was annoying, though, was that we could not remove the failed association ("Disassociate" did not work), nor add it again, so we had to resort to creating a new repository to test CodeGuru.
CodeGuru is triggered by pull requests, which is when a developer asks for their code to be merged into the repository. We used a simple Spring Boot Java project and started making pull requests. Unfortunately we could not persuade CodeGuru to come up with any recommendations. According to the documentation it provides recommendation on AWS best practice, concurrency, resource leaks, sensitive information leaks, and common coding best practice. We tried some obviously buggy code:
MyClass c = new MyClass(); c = null; return c.GetMessage();
And when that passed unnoticed, a simple race condition, but while CodeGuru said it had analysed the code, there were no recommendations.
Well, it is a preview and we are not alone; another user reported the same issue in the AWS forums. Support can no doubt work it out. You can see what is meant to happen in the presentation here. Presuming it works, though, how much does it cost? The price is $0.75 per 100 lines of code, which could work out to be expensive. In Amazon's pricing example, 50 developers making 600 pull requests of 200 lines of code each comes to $900 a month. However, if a substantial pull request was reworked a few times, the amount of code could easily exceed this estimate.
The pricing for the profiler is $0.005 per sampling hour up to a maximum of 36,000 hours per month, so the maximum charge is $180 per application. While this may soon pay for itself after finding a needed optimisation, it may be something you want to turn off once an application is running sweetly. You would also want to be convinced that the CodeGuru profiler was finding things that other more cost-effective profilers miss. The AWS claim is that the profiler easily identifies the lines of code that need to be optimised, and that it can run in production "without impact to the application."
The profiler has three components. There is an agent which you add to your application, a profiler service, and a recommendation engine. The results are presented as interactive visualisations. There are three types of visualisation: an overview, a "Hotspots" visualisation showing the functions which consume the most time, and an "Inspect" visualisation that shows code grouped with the functions it calls, and the functions by which it is called.
The good news is that AWS CodeGuru is offered with a free 90-day trial. That is long enough both to profile some existing applications, and to try the CodeGuru Reviewer in anger to see if it is delivering enough value to justify its cost. Otherwise, these are not cheap services, and we suggest careful evaluation. ®