Yet another job menaced by AI! Uh, wait, it says here... Dance Dance Revolution designers

Dance Dance Revolution – one of the few computer games that causes players to break into a sweat – is getting a boost from artificial intelligence.

The game is frequently spotted in arcades. Players stand on a metallic square pad that has up, down, left and right arrow buttons, and you basically follow on-screen prompts to stomp on the correct sequence of arrows, in time to music. Each edition of Dance Dance Revolution comes with a set of tracks to complete. It's played like this:

Youtube Video

That's all well and good until you get bored of dancing to the same music over and over. But forget all that – now you can use machine learning to create dance patterns for any song or track you want.

Dance Dance Convolution, described in a paper on arXiv this week, was developed by a group of researchers from the University of California, San Diego. It's an AI system that can be used to generate a step chart for any desired song.

Dance Dance Revolution (DDR) first emerged in Japan in 1998, and the mania quickly spread to Europe and America. DDR is serious business. The latest world championship was won by Chris Chike in February, an expert rhythmic button basher who also excels at Guitar Hero – a similar type of game played on a guitar-shaped controller. Chike has since announced a brief break from the sport after a loss of sensation on the tip of his right index finger from repetitive strain injury.

Hardcore DDR dancers like Chike – and perhaps even you – have most likely exhausted the limited track list in the game during practice. Luckily, there are open-source tools like StepMania that allow fans to craft their own step charts.

Gamers can distribute these homemade levels but it's difficult to do due to music copyright: players need both a level's pattern data and a copy of the track. Designers may be forbidden from handing out copies of the music, leaving people unable to play the levels. Now gamers can take their favorite smash hits and use Dance Dance Convolution to generate decent step charts from the tracks, all by themselves.

This bypasses levels designed by fans and other developers, and the need to source the music to play them. Just take your own music and make your own patterns using AI. There is already software out there to automagically spit out steps from a given music file, but the point of Dance Dance Convolution is that it is a superior designer – almost as good as human designers.

Here's how Dance Dance Convolution works. First, features such as pitch and rhythm have to be extracted from an audio file by converting it into a spectrogram and representing it as a tensor (a mathematical form similar to vectors) before it can be processed by an algorithm.

Next, a “step placement” algorithm made up of a convolutional and recurrent neural network processes the raw audio and splits it into 10ms slices. It calculates the probability that a step should be placed at a given time, explained Zachary Lipton, coauthor of the paper and researcher at UC San Diego.

Finally, a “step selection” algorithm takes the list of step times previously computed and maps them as a dance move in the DDR game.

DDR developer veteran gives it the thumbs up

The system was trained on two different datasets. One was created by a DDR fan who goes by the name Fraxtil, and it contains 90 songs with five charts per song for each of five levels of difficulty. The second dataset has several authors and consists of 133 songs with one chart per difficulty, except for 13 songs that lack charts for the highest difficulty.

Both datasets gave the researchers 35 hours of annotated music and 350,000 steps. The dance patterns tend to mirror musical structure, and the higher-difficulty charts are born from more complex musical patterns.

“I’ve seen some attempts at autogenerating step charts over the years, and this is by far the most successful iteration. I’m most impressed by the variety of instruments it can detect and how well it chooses which ones to emphasize.

“That being said – even ignoring any rhythm or patterning errors – it’s pretty easy to tell that its output is synthetic. There’s a lot of creativity involved in step charting, mainly selective use of repetition and contrast, that the AI either can’t learn or can’t apply effectively,” Fraxtil told The Register.

Youtube Video

Making good step charts isn’t a trivial pursuit. Fraxtil has been producing step patterns for the game for eight years and is only pleased with the last four years of their work. “It’s all very subjective, but as you play other step artists’ charts you start to learn what works on the dance pad and what doesn’t.”

It’s not all fun and games, and there is academic merit behind the study. The Dance Dance Convolution project started when Chris Donahue, coauthor of the paper and researcher in the music department of UC San Diego, wanted to learn more about music information retrieval (MIR).

“MIR is crucial if computers are to continue to engage and assist humans in both the consumption and production of music. Music is an exceedingly complex phenomenon and attempting to extract high-level information directly from audio necessitates that a computer reconcile issues not just in signal processing but also human acoustic perception and social contexts.

“I wanted to do research in MIR but found it difficult to acquire the audio associated with most datasets. One day it occurred to me to use the countless gigabytes of DDR data I had sitting on an old hard drive to train a neural network to perform rhythm extraction. At some point I thought, why not also try to generate the arrow sequences to go along with the rhythms?” Donahue told The Register.

Fancy a go?

The full source code will be released upon official publication of the research, Lipton said. But if you fancy a boogie to your favorite song, here’s a link to an online version of Dance Dance Convolution. ®