A gesture-controlled game where your body is the input.
And your body is also the surveillance.
For "Anything is an Interface" at ArtCenter, I built a gesture-controlled game where the player's body becomes the entire input system. No keyboard. No mouse. Just hands, your face, and a webcam.
The project translates Squid Game's red-light, green-light challenge into a real-time ml5.js prototype. When the light is green and the doll is turned away, a pinch of thumb and index finger advances the character. When the light turns red and the doll turns to face you, nose-tracking activates — any detectable movement of your nose ends the game.
The mechanic sounds simple. The body's response to it is not.
The project asks what happens when the interface is your body and the surveillance is real.
Squid Game's mechanic — move freely, freeze under threat of detection — translates into a small experiment in response inhibition under digital observation. The game isn't simulating that. It's literally reproducing it. Even players who know nothing is at stake experience the freeze response: the held breath, the involuntary stillness, the brief tension that comes from being watched.
The body responds even when nothing is at stake.
The game has two states.
Green Light. The doll faces away. A pinch gesture — thumb to index finger, tracked through ml5.js HandPose — advances the on-screen character toward the finish line. The longer the pinch holds, the further you go.
Red Light. The doll turns. The nose-tracking model (ml5.js BodyPose) activates. Any detectable movement of the nose ends the game.
The state alternates on a timer: three seconds of green, four seconds of red. Reach the finish line during a green phase to win.
State is communicated without text — only the doll's behavior, geometric indicator shapes, and a neon outline around the field that shifts with each phase: pink before start, green during movement, red during freeze.
I am not a coder. I built this prototype with Claude Code as the implementation partner — me providing direction, judgment, and design decisions; Claude writing and debugging the p5.js and ml5.js. The arc below is three moments where that division of labor was visible.
"Give me ideas on how to meet this brief using Squid Game. I want it to be embodied — the user's body should be the control."
A list of body-part inputs — eye blinking, hand raising, leg movement, posture detection, full-body silhouette.
Picked hand-raising with ml5.js image classification. Most recognizable to first-time players, simplest to train via Teachable Machine, fit the technical brief.
"The hand-raising approach isn't responsive enough. Detection lags. Players lose because the model takes too long to register the gesture. What are the alternatives?"
Several ml5.js-native pose detection options including HandPose (real-time hand landmark detection) and BodyPose (full-body pose with individual joint positions). Both run continuously rather than depending on trained image classification.
Redesigned the input entirely. HandPose for the movement gesture — a thumb-and-index pinch, deliberate, hard to do accidentally. BodyPose's nose-tracking for the freeze enforcement — nose movement is nearly impossible to suppress voluntarily, which made the freeze mechanic actually feel like a freeze. This shift made the game feel responsive and embodied for the first time.
"Add visual state communication for the green-light vs. red-light phases."
A colored bounding box around the entire field that switched color with each state — green during green light, red during red light.
Rejected. Asked Claude to replace the box with a glowing neon outline traced around the field's existing border. Pink before start, green during the green phase, red during the red phase. Same state communication, integrated into the geometry instead of layered over it.
A pattern emerged across the work. Claude was reliably useful for code: pose detection logic, state management, debugging detection thresholds. Less so for design. Most of the visual identity of the game came from rejecting Claude's first attempts and rewriting them. The collaboration worked because the division was honest — Claude's hands, my judgment.
Below: a complete win, then a complete loss. The live prototype is hosted on the p5.js editor and requires webcam access to play.
I came to this project because I loved Squid Game. Not because I had a critique. The mechanic was visually striking, the source material was charged, and the brief — "anything is an interface" — gave me permission to translate one into the other.
Building it taught me something the show was already saying. The body responds to surveillance even when the player knows nothing is at stake. The freeze isn't a choice. It's what the body does when it senses observation under threat. Squid Game's politics — forced compliance, the cost of being watched — turn out to be a real physical thing the moment you put the mechanic into a player's actual body.
The case for AI-collaborative practice this project makes is the inverse of the one people usually fear. I didn't lose creative agency to Claude. The division of labor made it more visible — Claude's job was implementation, mine was deciding what to make and how it should feel. That distinction kept design judgment human, exactly where it has to live.
The body doesn't ask whether it's a game.
Play the live prototype