Azure-hosted AI for finding code defects emitted – but does it work?

How many defects has it found? Never mind that, check out the architecture flow chart

Got Tips? 9 Reg comments
A new Code Defect AI project claims to find bugs in your code when you commit to a repository
A new Code Defect AI project claims to find bugs in your code when you commit to a repository

Altran – in association with Microsoft – has pushed out an open source project to find code defects via AI whenever you commit code.

It is easy to see why Microsoft is keen. This is a self-managed service where you download the code and purchase Azure resources on which to host it. The project from Altran – a consulting and software engineering firm that's part of Capgemini – uses two Azure Web Apps (one for the UI and one for the API); an Azure virtual network, VPN gateway and virtual network security group; Azure container registry; Azure MariaDB database service; and a container for scheduling the prediction code. Not to mention, it also includes Azure Application Insights for that all-important traffic analysis.

The UI web app is to be configured with two P1v2 Linux plans, each setting you back around £61 per month, and the AI web app with two P2V2 Linux plans at around £123.00 each per month. The MariaDB database is 4 vCores at £221 per month – so you are paying £600 per month before adding in storage costs, virtual network gateway and so on. The code and installation instructions are on GitHub here. The guide explains everything you need to know about deploying the application, but nothing much about how to use it.

The way the system appears to work is that a bug detection model is trained by inspecting open source GitHub projects, including the history of reported bugs in the codebase. Then, say the docs rather vaguely, "Once we have a model that has acceptable value of precision and recall, selected model is deployed for prediction on new commits." Code that seems likely to be buggy is flagged for further investigation and testing.

According to Altran, "Code Defect AI relies on various ML techniques including random decision forests, support vector machines, multilayer perceptron (MLP) and logistic regression. Historical data is extracted, pre-processed and labelled to train the algorithm and curate a reliable decision model. Developers are given a confidence score that predicts whether the code is compliant or presents the risk of containing bugs."

Code Defect AI supports integration with third-party analysis tools and can itself help identify bugs in a given program code. Additionally, the Code Defect AI tool allows developers to assess which features in the code have higher weightage in terms of bug prediction, i.e., if there are two features in the software that play a role in the assessment of a probable bug, which feature will take precedence.

The "solution" page looks glossy but The Reg noticed a surprising omission in the presentation. Normally such things have enthusiastic testimonies from happy users. We did find a video in which Altran VP Software Engineering and Cybersecurity Khalid Sebti said that "customers who have used our code defect solution predictions are able to identify over 20 per cent of the defects before the testing process begins."

But how?

Where is the evidence, though? We asked Altran, which first sent us a document entitled "PowerPoint Presentation" assuring us that this was "an AI powered solution that identified bugs" but with no data to back up the claim. We asked again and were referred to the code on GitHub and in particular the "flowchart of the architecture." There are several such flowcharts, one which has "storage and data lake Azure" stuck on the side for good measure. We did learn from the aforementioned video that this is merely phase one of the project. In phase two, "the solution expands to other Microsoft AI/ML technologies including ML Server and SQL Server." The flow chart for this includes the magic word Kubernetes for extra goodness, as well as both SQL Server and Cosmos DB.

Phase two of the project is expanded to use increasing numbers of Azure services

Phase two of the project is expanded to use increasing numbers of Azure services

The question perhaps is whether it makes sense to train your own AI model on defects found in open source GitHub projects when it might be better to subscribe to a service that has done its own model training at scale. On its security blog, Microsoft posted last month that "since 2001 Microsoft has collected 13 million work items and bugs. We used that data to develop a process and machine learning model that correctly distinguishes between security and non-security bugs 99 per cent of the time and accurately identifies the critical, high priority security bugs 97 per cent of the time."

At Microsoft, the post explained, "47,000 developers generate nearly 30 thousand bugs a month," which strikes us as good going. The methodology for this is also promised to be open-sourced on GitHub "in the coming months" though it does not appear to be related to the Altran project.

AWS trained its CodeGuru profiler on "hundreds of thousands of internal projects, as well as over 10,000 open source projects in GitHub." This is the kind of scale which, like Microsoft's security project, may yield interesting results.

Is this an open source project hoping to attract the support of a sufficient collaboration community to sustain momentum? It looks unlikely; some code has been dumped in GitHub but there is no evidence yet of any community around it. Think of it more as a sample showing how you can code-host an AI project on Azure. In that respect it may be useful, but the "better code faster with Azure AI," that the PowerPoint promises? We are sceptical but await customer reports with interest. ®

Updated at 08:55 on 20 May to add

Altran has provided us with more information on how its AI defect finder works. The company told us that the AI does not look much at the source code itself, but rather at the commit metadata, “the number of files in the check-in, code complexity, density of the check-in, bug history of the file, history of the developer, experience of the developer in the particular module/file etc.”

Training of the model is done only on the project being examined; the reference to open source projects is only because “the published solution is for a GitHub project,” the company said.

It has an example for us. “One of the projects where we used this solution started flagging a larger number of code commits as carrying a higher risk on Tuesdays. We initially doubted the ML model. However, we then discovered that the development team had a practice of doing weekly builds on Tuesdays. Developers were able to commit the code that was not fully tested before the weekly build,“ we were told.

The example bears out the claim that the solution “solves issues in software validation and verification that are not addressed by traditional source code analysis and test automation tools,” though we might add, do you really need a complex cloud-hosted AI system to tell you that code committed by developers under deadline pressure and skimping on testing has more bugs? Fixing defects is expensive though, so any system which improves the process has value. The project is part of a more extensive Altran testing system called ATLAS.

Sponsored: Practical tips for Office 365 tenant-to-tenant migration

SUBSCRIBE TO OUR WEEKLY TECH NEWSLETTER


Biting the hand that feeds IT © 1998–2020