Emergent Tech

Artificial Intelligence

Another AI attack, this time against 'black box' machine learning

The difference between George Clooney and Dustin Hoffman? Just a couple of pixels

By Richard Chirgwin

17 SHARE

Would you like to join the merry band of researchers breaking machine learning models? A trio of German researchers has published a tool designed to make it easier to craft adversarial models when you're attacking a “black box”.

Unlike adversarial models that attack AIs “from the inside”, attacks developed for black boxes could be used against closed system like autonomous cars, security (facial recognition, for example), or speech recognition (Alexa or Cortana).

The tool, called Foolbox, is currently under review for presentation at next year's International Conference on Learning Representations (kicking off at the end of April).

Wieland Brendel, Jonas Rauber and Matthias Bethge of the Eberhard Karls University Tubingen, Germany explained at arXiv that Foolbox is a “decision-based” attack called a boundary attack which “starts from a large adversarial perturbation and then seeks to reduce the perturbation while staying adversarial”.

Foolbox tested against Clarifai's black-box AI

“Its basic operating principle – starting from a large perturbation and successively reducing it – inverts the logic of essentially all previous adversarial attacks. Besides being surprisingly simple, the boundary attack is also extremely flexible”, they wrote.

For example, “transfer-based attacks” have to be tested against the same training data as the models they're attacking, and need “cumbersome substitute models”.

Gradient-based attacks, the paper claimed, also need detailed knowledge about the target model, while score-based attacks need access to the target model's confidence scores.

The boundary attack, the paper said, only needs to see the final decision of a machine learning model – the class label it applies to an input, for example, or in a speech recognition model, the transcribed sentence.

Foolbox tested against logos in the Clarifai black box

The researchers tested their attack using the Clarifai API, tricking it into mis-identifying celebrities and missing prominent logos. ®

Sign up to our NewsletterGet IT in your inbox daily

17 Comments

More from The Register

Artificial intelligence is good for at least one thing – making hardware important again

Red Hat Summit Latest compute craze turns the tide on system trends

Artificial intelligence? yawns DDN. That's just the new HPC, isn't it?

We already do bigger, faster arrays – now we're scaling up

Artificial intelligence will eradicate channel drudgery, says Lenovo boss

Canalys Channels Forum 2016 Any intelligence would be a start, grumble partners

Google ramping up AI in China, Nvidia's Titan V, Intel's hip-hop misstep

Roundup And more in your machine-learning news summary

MEPs in 'urgent' call for new laws on artificial intelligence and robotics

Liability issues with self-driving cars is key concern

Google Cloud boss promises 'security built into every layer of the system' at UK shindig

Google Cloud Next Hopes to lure new cloud-sniffers with location lockdown feature

IT bosses worried about network security reckon AI Jesus can save them, says Oracle survey

Of course Big Red finds another thing needing more automation

Reg readers cluster in pub to ponder artificial intelligence

Reg Events And hear why a mic-dropping, beer hurling AI would be a real advance

What's holding you back from Google Cloud? Oh, OK... it was hoping you'd say 'lack of hardware security modules'

Like AWS and Azure, GCP now hosts secrets inside HSMs

Nvidia adds nine nifty AI supercomputing containers to the cloud

Now you can splash out on tons of GPUs if you really need to