IDEA LAB · 2026

Color & Noise

A private instrument I designed and built for making audio-reactive video: sound, motion and code on one live 3D canvas.

Role
Solo · design and build
Year
2026
Stack
Sound · Motion · Code

THE PROBLEM

Four strong tools, four clocks.

I started by borrowing the best tools I could find. Sound was openDAW, a full browser DAW, running in a frame. Beats ran on a third engine, Strudel. Video was going to be FreeCut, another app in another frame. Each was strong on its own. Together they were four apps with four clocks, and they only talked through exported files.

DECISION 1

One stage, not four apps.

  • Problem: every switch replaced the picture, and every tool had its own look and its own idea of time.
  • What I did: I reversed my own embed-first decision and wrote five rules. The canvas never leaves. Left is what's in it, right is the thing you picked, bottom is time. One clock. One file. One look. The dock now changes the panels around the picture (Look, Sound, Motion, Agent), never the picture. Before building, I made the plan wait on one proof: that openDAW's engine could boot inside my app without its interface. It did.
  • Why: a musician thinks in one piece, not four applications.
  • Left is what's in it, right is the thing you picked

DECISION 2

Sound runs on one clock.

  • Problem: an audio track can't stay in time with animation keyed to a second clock.
  • What I did: openDAW's engine now plays under my own Drums, Synth, Audio and Master panels, and its position is the timeline's time. The song is saved as plain data (tracks, pads, steps and tempo), so it shares one undo with the picture and never depends on openDAW's file format. Small calls matter too: a new scene's beat stays off until you ask for it, and a pad you tap is always heard, because a silent pad on first touch reads as broken.
  • Why: if sound and picture can drift, nothing else can be trusted.
  • Drums: pads lit as they hit, on the same clock as the picture
  • Synth: a phrase drawn on the roll, playing

DECISION 3

The music writes the motion.

  • Problem: I had planned a separate video editor, which would have been a second app again.
  • What I did: Motion became the timeline plus export. Of three concepts, two lead. Conduct ties a move (a pulse, swell or shake) to a pad's hit, every beat or every bar, so changing the beat changes the motion. Shots keep the camera's place on the timeline and can carry a saved look, so a cut changes the color as well as the angle. An export renders the song itself for exactly the video's range, and loops snap to whole bars, because 10 seconds at 124 bpm is 5.2 bars and every loop was cutting mid-bar.
  • Why: motion that follows the music feels edited. Motion on a stopwatch feels pasted on.
  • Four shots, one to a bar, cutting with the music

DECISION 4

Code stays code.

  • Problem: code is fast to write and easy to lose behind a visual editor.
  • What I did: backdrops can be live-coded in Hydra, effects can run Shadertoy-style shaders, and the shape's surface is my own GLSL. A property can follow a value, a curve, the music or code, and the rule is that those influences stay visible and never silently override a manual edit. The Agent desk is honest about its limits: today it chats, and the app says so.
  • Why: people who code want their source back, untouched.
  • Live code: the backdrop as Hydra, applied as you type

HOW I WORKED

Log it, then measure it.

I log every decision, including the ones I overturned. I measure before I trust a fix: the shape looked shattered because its noise function had three transcription errors, and the trail effect was erasing the vignette and grain until I moved it first in the chain. I built it with an AI coding partner. The design calls and the verification were mine, checked on a real GPU. A trimmed version also runs as the live hero of this portfolio.

  • Noise, after the fix
  • Shards, the old output kept as a pattern
See Shards in the tool

WHAT I'D DO NEXT

Give the Agent its full job.

Next, the Agent desk proposes a change as a candidate I can preview, edit, accept or throw away, without touching the work I already have.

Try the portfolio version