Finish Him! Kicking Mortal Kombat’s Kano With AI

Today’s complex video games are both a practical and fun research environment for training machine learning bots. This summer the “OpenAI Five” bot team faced top human players of the multiplayer online battle arena game Dota2 in a highly anticipated showdown at the International Dota2 Championships in Vancouver. Although it was a loss for OpenAI Five, it was also a win for AI.

Now, a Google engineer has used AI to enable humans to throw virtual punches and kicks at the tough guys in popular fighting video game Mortal Kombat3. Instead of jabbing at a controller, the user fights via a web camera, with the AI rendering their air strikes to an onscreen clone in real time. You punch and kick, and Kano goes down.

The project, which has garnered positive feedback on Reddit and Twitter, is the brainchild of Minko Gechev, an engineer with Google’s Angular team. As a main feature of this work is using the camera as a controller for a small JavaScript clone, Gechev used TensorFlow.js to train an image classification model.

The accuracy of a deep learning model is largely determined the quality of the training data, and so Gechev began by collected high-quality videos and pictures of volunteer “fighters” to use as training data. He defined three motion categories: punches, kicks, and others. Data augmentation then provided a rich set of images.

Rather than having the algorithm analyse a sequence of frames, Gechev designed a supervised deep learning model to identify fighting stance and strikes from single frames in the images captured by the user’s web camera.

Gechev first used a convolution neural network (CNN) to build the classification model used to sort the images. CNN architecture is renown for image recognition, object detection, and classification, and his trained model reached 92 percent accuracy. He wondered however whether the model could differentiate for example between a front kick, a back kick and a roundhouse kick, and so sent the CNN output for each frame to a Recurrent Neural Network model, which discovered dependencies between individual frames to recognize what actions they encode.

It’s not the first time this particular video game has been targeted by AI researchers. The Israel Institute of Technology attempted to train an AI bot to master Mortal Kombat using their novel “Retro Learning Environment” (RLE), which can run games on the Super Nintendo Entertainment System (SNES), Sega Genesis and a number of other gaming consoles.

It seems that iconic video games such as Mortal Kombat, Super Mario and Dota2 will continue to attract AI researchers seeking accessible methods for bringing their models to a better understanding of the real world.