Description
The Controller Project v3 is one in a series of interactive toys meant to capture the eye and encourage play. In this incarnation, a new form factor was created resembling an old arcade game or slot machine.
The Controller Project v3 was part of a larger series — each version built around a different controller, routing its signals through MIDI to drive real-time visual effects in Unity. V1 established that concept with a bank of traditional MIDI controllers feeding visuals out to Resolume. V2, known as The Orb, replaced physical buttons with hand gestures via a Leap Motion, displaying the results in a spherical projector. By v3, the goal had evolved further: this machine wasn’t just a standalone toy, it was a live interface for festival audiences to shape real-time visuals for an entire stage.
Inputs
For controllers, this time it uses a midi fighter for the fun arcade buttons, a Leap Motion hand tracker to capture motion of the user, and a 3D camera to capture the user’s shape and depth, converting their body into a point cloud — a 3D object where every individual point became a particle emitter, creating visuals that literally reflected the person standing in front of the machine. Audio input ensures all elements are fully audio reactive.
Outputs
It housed a 55-inch monitor for visual feedback, but its output extended further. The Controller was mirrored to a dedicated VJ rig connected to a projector, which served as the main visual display for one of the festival stages. Some participants knew they were controlling the stage graphics; others were simply playing with the machine, unaware their interactions were being projected for the crowd.
Software
All visuals were built and run in Unity, processing the combined inputs — MIDI, hand tracking, 3D capture, and audio — in real time to generate reactive, generative graphics.
Initial Designs
Designs where done in both Illustrator, Cinema4d and Fusion3d. I should have used Fusion 3d for the whole thing, but C4d was much quicker for the mock up and illustrator much quicker to plan the final cuts for the CNC machine.
Initial Mock Ups
A cardboard mock up was made by hand to make sure the form factor felt nice for the controls.
Fabrication
Close friends Justin Young and Dave Plotkin helped with the fabrication of the console. 3/4 inch Ultra Light MDF was selected for the base material, the initial cuts were made on the CNC machine at Humanmade SF and then assembled at Justin’s warehouse.
Final Product
The Controller has been shown at Sunset Campout and a number of regional art exhibits across the San Francisco Bay Area, drawing crowds absorbed in its interactive audio visualizations.
In retrospect, next time Ill make a smaller form factor as one of its biggest downsides is it’s lack of transportability.
