Better filters won't cure this: YouTube's kids nightmare

What has been seen?

child watches tablet in the dark

Comment For the "smartest guys in the room", Google often seems to be the last to know what’s going on in its own front room. And something very strange indeed is going on over at YouTube.

The artist James Bridle – who created the witty "self-driving car trap" (spoiler: it's a chalk circle) – has been investigating the outer reaches of kids' content. He describes how channels attracting millions of subscribers are producing very strange things indeed.

These range from the "decidedly off" to the nasty. The BBC has reported on the latter. The most disturbing may not be the obvious but predictable 4chan-derived pirate nasties, but Bridle describes them as "things which aren't overtly disturbing to adults, just incredibly dark and weird". Basement content factories are churning out imbecilic content that looks like this:

"This is not a simple matter of 'won't somebody think of the children' hand-wringing. Obviously this content is inappropriate, obviously there are bad actors out there, obviously some of these videos should be removed. Obviously too this raises questions of fair use, appropriation, free speech and so on. But reports which simply understand the problem through this lens fail to fully grasp the mechanisms being deployed, and thus are incapable of thinking its implications in totality, and responding accordingly."

We find a phenomenon so extensive that it cannot be easily dismissed as the product of a specific culture, or trolls.

For what Bridle is describing, he explains, is a disturbing new kind of "art", mass-produced only because a huge ad-supported platform such as YouTube rewards its creation. The "such as" caveat is redundant here, since no other video platform has scale. It has to be YouTube. Just as no other social network (at least outside China or Russia) has the scale of Facebook, so for fake news and political ads, it has to be Facebook. This is as much a characteristic of the internet in 2017 as fake news or spam.

As Bridle explains, thinking of "better filters" as a solution doesn't even begin to solve the problem.

An explanation of the production side gives us a clue. "Online kids' content is one of the few alternative ways of making money from 3D animation because the aesthetic standards are lower and independent production can profit through scale. It uses existing and easily available content (such as character models and motion-capture libraries) and it can be repeated and revised endlessly and mostly meaninglessly because the algorithms don't discriminate — and neither do the kids."

But the giant indiscriminate advertising engine is what incentivises it.

YouTube's Good Baby Toys channel

YouTube's "Good Baby Toys" channel. Note the number of views.

A sweatshop can produce endless huge amounts of material and doesn't care if the millions of "viewers" are robots or children. ("This is content production in the age of algorithmic discovery – even if you're a human, you have to end up impersonating the machine," Bridle notes.)

"It's not about trolls, but about a kind of violence inherent in the combination of digital systems and capitalist incentives. It's down to that level of the metal. And right now, right here, YouTube and Google are complicit in that system... The system is complicit in the abuse.. The architecture they have built to extract the maximum revenue from online video is being hacked by persons unknown to abuse children, perhaps not even deliberately, but at a massive scale."

We asked Google yesterday about guidelines for acceptable content for children on YouTube – but they have not responded.

Years ago, your reporter attempted to pacify a toddler by showing a clip of In The Night Garden from YouTube, only to find a pre-roll lavish advertisement for the Church of Scientology. Several more children's videos were being used to promote the cult: a use of BBC Worldwide material that the licensee perhaps did not anticipate. But that seems almost innocent now. ®

Sponsored: Minds Mastering Machines - Call for papers now open

Biting the hand that feeds IT © 1998–2018