How to steal the mind of an AI: Machine-learning models vulnerable to reverse engineering

Amazon, Baidu, Facebook, Google and Microsoft, among other technology companies, have been investing heavily in artificial intelligence and related disciplines like machine learning because they see the technology enabling services that become a source of revenue.

Consultancy Accenture earlier this week quantified this enthusiasm, predicting that AI "could double annual economic growth rates by 2035 by changing the nature of work and spawning a new relationship between man and machine" and by boosting labor productivity by 40 per cent.

Certainly things could work out well for Accenture, which a day later announced a partnership with Google to help companies deploy Google technology like machine learning. It's as if the global services firm has a stake in the future it foresees.

But the machine learning algorithms underpinning this harmonious union of people and circuits aren't secure.

In a paper [PDF] presented in August at the 25th Annual Usenix Security Symposium, researchers at École Polytechnique Fédérale de Lausanne, Cornell University, and The University of North Carolina at Chapel Hill showed that machine learning models can be stolen and that basic security measures don't really mitigate attacks.

Machine learning models may, for example, accept image data and return predictions about what's in the image.

Taking advantage of the fact that machine learning models allow input and may return predictions with percentages indicating confidence of correctness, the researchers demonstrate "simple, efficient attacks that extract target ML models with near-perfect fidelity for popular model classes including logistic regression, neural networks, and decision trees."

That's a polite way of saying such models can be reverse engineered. The researchers tested their attack successfully on BigML and Amazon Machine Learning, both of which were told of the findings in February.

In an email, Cornell Tech computer science professor Ari Juels, a coauthor of the paper, suggested mitigating these attacks could prove challenging. "Effective countermeasures to model extraction could well be possible, but this remains an open research question," he said.

"My suspicion is that solutions that don't degrade functionality will need to involve greater model complexity."

Many machine learning models in recent years have been released as open source software, because the companies developing them want users to improve the code and to deploy the models on their cloud infrastructure.

But there are also plenty of machine learning models that depend upon confidentiality, like fraud detection algorithms. And given that firms like BigML and Google allow model creators to charge fees to others using their models, the incentive to keep models from public view goes beyond the tech companies pushing an AI agenda.

The potential motivation for machine learning attacks goes beyond fraud enablement and fee avoidance. Model extraction can facilitate privacy violations, the researchers point out, by making it easier to identify images of people used to train a facial recognition system, for example.

At the same time, there's a case to be made for supporting efforts that reveal undisclosed algorithms, whether they involve machine learning, computer vision, or other AI-oriented disciplines. Lack of transparency into automated systems leaves people unable to determine whether, for example, the rejection of an employment application by an applicant tracking system followed from lack of qualification, encoded bias, or the software's inability to read italicized text [PDF] in a resume.

"Machine learning models certainly deserve greater scrutiny to ensure algorithmic fairness," Juels observed, pointing to a recent paper he and fellow researchers published on the subject. "Such scrutiny is best performed by the developers of the model, though, as advocated for in our paper, rather than through what amounts to an attack on the model."

It may turn out that much of the predicted AI-driven economic growth arises from litigation to expose or shield the algorithms and machine learning models that make decisions about us. ®