A Horse Made of Pixels and Other Stories

The topic of this piece was innocuous when suggested, but my usual research rabbit hole proffered up an unexpected bit of joy: the chance to slip in (and maybe drone on like a madman about) a reference to my favourite video game, Red Dead Redemption 2. But hold your horses (pun intended); this piece is about the juxtaposition of the many elements that make up the moving image, and if a game is being mentioned in this regard, I must begin with Prince of Persia.

Back in the ancient days, occasionally known as the 1980s, this game received rave reviews on its release, though it only became a hit once it was ported to other computers. As Mechner later put it, it was “saved by the ports”. While the storyline was enchanting in itself, it’s the protagonist’s mechanics I tip my hat to, because they were surreally real. His shuffle, walk, run, jump, hanging on and sword-fighting movements felt very human, and added to the appeal of the experience. The odd thing is, after Prince of Persia I basically stopped playing video games until RDR2. And while there’s no doubt that video games made unbelievable leaps between PoP and RDR2, both are valid examples of what this article explores.

In any case, I won’t wax eloquent about RDR2 itself; there are hundreds of excellent YouTube videos that do just that. Instead, I want to mention just how oddly immersive this “game” is, to the point where you begin to genuinely feel for the fates not just of your playable characters (Arthur Morgan and John Marston), but of others like Lenny, Sean, Mrs. Downes and Hosea. Perhaps the most impressive feat is caring about your horse. Yes, your clearly make-believe, CGI-based equine companion, whose (spoiler alert) death scene has crushed the spirit of many a player. Even typing these sentences gives me pause, because in an increasingly digital-first world we strive to experience what’s real (the shorthand, aka lazy, expression these days is “touch grass”, I believe), and what I’ve just typed runs counter to that struggle. Collectively, we’re somewhere between Neo and Cypher at this point.

Anyway, where was I? Ah yes. There’s a connection between these two games, separated by almost three decades.

To create the unusually fluid movements of the original Prince of Persia, Jordan Mechner videotaped his 15-year-old brother David running, jumping and climbing in their high-school parking lot in Chappaqua, New York. That was in 1985, four years before the game came out. Mechner then traced individual frames from the footage and digitised them, a technique known as rotoscoping. The tapes and development materials are now preserved by The Strong National Museum of Play.

Red Dead Redemption 2 used a far more sophisticated approach. Rockstar recorded its actors at its own motion-capture hangar in Bethpage, New York, over a total of 2,200 days, alongside character animation, computer-generated environments and real-time rendering. Roger Clark’s performance as Arthur Morgan became a digital character who moves, interacts and responds within a world the player is free to explore. Dan Houser was so protective of that illusion that he avoided Clark on set, so he wouldn’t hear his real voice. The two techniques aren’t the same. Rotoscoping uses photographed movement as a reference to draw from; performance capture records an actor’s movement and applies it to a digital character. In both cases, animators still shaped the result. What connects the games is that each starts with recorded human movement and turns it into something that isn’t simply a recording of a human being.

Film has been doing the same thing from the other end. Jon Favreau’s The Lion King (2019) looks like wildlife photography, albeit with animals who converse and occasionally sing. Yet apart from one shot, every image in it was made on a computer. Favreau later revealed that the opening sunrise was the only real footage, photographed in Africa and slipped in among 1,490 rendered shots to see whether anyone would notice (I did not). Nor were the animals motion-captured: animators at MPC keyframed every performance, studying how real animals move and behave.

The production borrowed its methods from live-action filmmaking. Using virtual-reality headsets and game-engine technology, Favreau and his cinematography team could walk around digital locations, place cameras and compose shots much as they would on a physical set. Asked whether the result was live-action or animation, Favreau said: “it’s neither, really.” Looking at the finished film alone, you wouldn’t know how it was made, but, as an aside, Dan Houser would argue that’s the point. Games, he told GQ, are “still magical”, and players gain something by not knowing how they’re made.

Filmmakers have been blurring this line for a long time. In 1952, Norman McLaren of the National Film Board of Canada released Neighbours, a short about two men whose quarrel over a flower escalates into violence. McLaren used pixillation, photographing live actors one frame at a time rather than filming them continuously. By repositioning them between exposures, he created movements no real body could make. The people were real, but their movement was animated. No computer was involved, and the film went on to win the Oscar for Documentary Short Subject. An animated film, honoured as a documentary.

Loving Vincent (2017), directed by Dorota Kobiela and Hugh Welchman, also started with live actors, then buried them under oil paint. The cast was filmed on green screen and on sets built in Van Gogh’s style, and 125 painter-animators then repainted that footage in oils, frame by frame: about 65,000 frames in all. The photographed performances are still underneath; what you see is paint, and it got to me more than I expected.

Spider-Man: Across the Spider-Verse (2023) does the opposite of hiding its joins. The film moves across multiple universes, each with its own visual identity, drawing on comic-book printing, collage, hand-drawn animation, painting and 3D computer graphics to give different worlds and characters their own appearance and movement. Take Hobie Brown, better known as Spider-Punk. His look comes from zines, photocopies and patchwork collage, and his movement follows suit, because different parts of him run at different frame rates. According to Sony Pictures Imageworks’ head of character animation Alan Hawkins, Hobie’s jacket was animated on 4s, his body on 3s and his guitar sometimes on 6s, which meant building multiple rigs, each with its own setting. The studio also built a pipeline tool, StepSets, so that stepped animation wouldn’t break the work of downstream departments; it was used in every shot of the film.

For this guild’s readers, the detail worth knowing is Mumbattan, the city of Pavitr Prabhakar, or Spider-Man India. The effects artists working on it were given the classic Indrajal Comics as the inspiration behind its visuals, and had to work out what fire, dust and explosions would look like in that world. Throughout the film, hand-drawn 2D effects and 3D simulation were passed back and forth between departments until the look was right. Most films hide the joins between techniques. This one shows them, even when that made the work harder.

The same year The Lion King came out, Favreau ran the experiment in reverse. For the first season of The Mandalorian (2019), which he created, Industrial Light & Magic worked with Epic Games and others to build StageCraft: a stage surrounded by LED walls displaying computer-generated environments, rendered in real time in Unreal Engine. The environments could be adjusted during filming and lit the actors and sets, reflections included. More than half of the season was shot this way, letting the filmmakers make decisions on set that would otherwise have waited for post-production. On The Lion King, the camera crew went into the computer. On The Mandalorian, the computer came onto the stage.

All of that is about making images. Games also change who decides what you see. In a film, the director and editor choose the angles and the order of events; in a game, players influence where they go, what they look at and, sometimes, how things unfold. RDR2 has carefully directed cinematic sequences, but long stretches of exploration outside them, so its world and characters have to work beyond the demands of any planned shot.

Alejandro G. Iñárritu’s Carne y Arena (2017) is neither a film nor a game. Developed with ILMxLAB and based on the true accounts of Mexican and Central American migrants, it puts visitors, alone and barefoot, in a dark room covered in sand, wearing a VR headset, for a six-and-a-half-minute border crossing. The migrants and border guards play themselves, their performances recorded through motion capture. There are no objectives or choices that change the outcome; your movement only changes what you see. Iñárritu called it “part theater, part documentary and part fiction”, and didn’t consider it cinema at all.

The one time I tried a VR headset (for a Star Wars game in a mall in 2019), it blew my mind and got me going “Holy crap” like a stuck record for 20 minutes.

Machinima runs the traffic the other way: a game used to make something you only watch. One of its best-known examples is Red vs. Blue, the comedy series Rooster Teeth launched in 2003 using footage from Halo. Co-founder Burnie Burns described manipulating the game’s characters like puppets and recording them in real time. Dialogue, editing and sound effects turned those recordings into scripted episodes, distributed online. The game supplied the characters, sets and real-time rendering; the finished series wasn’t something anyone could play.

In May 2026, curators at the British Film Institute described online video as the third age of the moving image, after cinema and television. Their account runs from animated GIFs and Flash animation through machinima, livestreams, vlogs and video essays to today’s short-form feeds. One detail stands out: early GIFs and Flash animations relied on short loops partly because bandwidth was limited, and the repetition soon became an aesthetic in its own right. A technical limit turned into a style.

None of this means animation, cinema, visual effects, stop motion and gaming have become indistinguishable. The techniques remain different, and the people who practise them have highly specialised skills. A stop-motion animator doesn’t automatically understand digital compositing, and a filmmaker working with virtual production still needs to understand cinematography. The overlap happens in the work, where contributions from different disciplines combine into something no single production label quite describes. Call The Lion King animation, RDR2 a game, Red vs. Blue machinima and Carne y Arena a VR installation, and you’d be right every time, just never completely.

Which brings me back to Prince of Persia. In 1985, a teenager running around a parking lot provided the movement for a character in a computer game. Three decades later, actors, animators, programmers and digital artists built the extraordinarily detailed world of Red Dead Redemption 2. Almost everything about how the two were made is different. What they share is the starting point: a real body in motion, recorded, then handed to people whose job was to turn it into something else.

Of course I gagged when the blades cut the protagonist in Prince of Persia. But how someone managed to make me care about an animated horse made of pixels is something I’ll never know. It was the Turkoman, in case anyone’s asking, though the Perlino Andalucian comes a close second.