earshot

earshot

Ears for coding agents.

Watch the 22 s launch film - every sound in it was made in code by earshot forge; no recordings, no music track.

AI coding agents can now write the code that renders a film, a product demo or a game. They still cannot hear what they made. They cannot tell a premium key press from a cheap one, notice that a hit lands 80 ms after the cut, or hear that the music swells in the wrong place.

earshot turns sound into evidence an agent can reason about: what a file sounds like in words, how well it fits a description, the best moment inside a long recording, whether it is a real recording or code-synthesized, a second-by-second listening transcript of a whole film, an annotated picture of it, and a scorecard against professional launch films. It runs locally on CPU, as a CLI, a Python API and an MCP server.

With ears, an agent can also make its own sounds in code and improve them by listening (earshot forge), and translate a recording into code it can vary (earshot fit). Both below.

New in 0.4: the sound of a music-led launch film. 51 new sounds made in code for the soft, short, mid-register style those films use: rounded UI pops, wood and plastic knocks, short cut hits and sub hits with a kick, gated glockenspiel / celesta / marimba notes and chime runs, water drops, camera shutters, reverse cymbals, a thump-whoosh, short dark risers, bright swishes, burst glitches and mid dings (the bank now holds 252). A new launch kit preset mixes every effect against the music bed instead of at a fixed level, says where each one sits against the cut (taps lead it, whoosh peaks land after it) and how close a run may go. And earshot fit was re-made against professional sound effects and measured on held-out clips (below).

New in 0.3: sound for code-made films. A bank of sound effects stored as code that render by id in seconds, with no models and no network: earshot sfx aims a whoosh, hit, click or note at a moment in the picture, and earshot kit renders a whole film's effects with measured sync points (below).

$ earshot describe enter.ogg
  0.56 s short transient (attack 3 ms, decays in 190 ms). Sounds most like: a single mechanical keyboard key press
  (z +5.4), a soft key release click (z +3.6) ... [clear reading]. peak -0.1 dBFS, brightness 1.5 kHz ...

$ earshot match key.ogg "a deep thocky mechanical keyboard press" --alts "a clicky blue switch keyboard" "a mouse click"
  -> strong (z +3.7; would rank #2 among 322 vocabulary descriptions) ...

$ earshot find keys.ogg "a single mechanical key press" --top 4
  0:03.94-0:04.19 (0.24 s stroke, z +4.1 strong); 0:10.60-0:10.86 ...

$ earshot critique film.mp4
  VERDICT: close - 21 of 23 scored metrics in range ... FIX FIRST: 1. [major] nothing reaches body level until 0:01.01 ...

What it does

Every command prints a plain-language summary first; --json gives the full result, --out f.json saves it. Errors are one line on stderr (earshot <command>: error: ..., exit 1). Each CLI call is a new process and loads the models (~20-40 s) before it answers (measure, sfx and kit load none): for more than a few calls use the MCP server (models stay loaded) or one Python script with from earshot import api.

What an agent sees when it listens to that film with earshot sheet (frames, labelled sound events, spectrogram, loudness, sync to the beat):

Make sounds: earshot forge

Listening makes iteration possible. The agent writes a sound as code: a recipe of physical synthesis (modal resonators, contact excitations, filtered noise, a computed room and a microphone; no recorded audio inside). The forge renders candidates from the recipe's parameters, and a critic built from earshot scores each one: description match against named confusions, realness, closeness to real reference recordings and measured shape, then model-free penalties against the references: 1/3-octave tonal balance, the spectro-temporal envelope (a tail that dies too fast shows here), artifacts (clicks, aliasing, DC / infrasound, harsh resonances, comb ring, cut tails, clipping) and polish (attack, onset snap, tail smoothness). A cheap model-free pre-screen ranks many renders with the same measurements, so the models only hear the best; an evolutionary search keeps the best and mutates them. Then the agent reads the critic's words ("sounds like: a cork pop, a glass ting", "decay +2.4 outside the refs", "bumpy tail"), looks at spectrograms next to the references, rewrites the recipe, and runs it again.

earshot forge --list                             # the recipes (click, impact, keypress, pop, whoosh, ...) + refsets
earshot forge keypress --refs path/to/real/keypresses/ --budget 144 --variants 4 --dir made/keypress
earshot forge whoosh --refset whoosh_fast --budget 96 --dir made/whoosh      # a reference set's 'forge' split
earshot forge whoosh --target "a fast crisp air whoosh" --alts "a wind gust" "a sword swing" \
  --refs path/to/real/whooshes/ --mode real --dir made/whoosh
earshot forge --render keypress --params made/keypress/report.json --seeds 1 2 3 --dir made/again

Output (--dir): the best diverse sounds as WAVs; report.md (what was asked, the search per generation, each pick in the critic's words, the measured gap to the references); compare.png (each pick's spectrogram, envelope and 1/3-octave balance next to its nearest reference); audition.wav (made, ref, made, ref ... to listen in pairs); report.json with every candidate's parameters and scores, the same critic's scores of the real references (leave-one-out) and held-out AudioSet tags. Picks are re-rendered on new seeds (--robust, default 3) and must differ in parameters and in sound. Reference sets (earshot/forge/refsets.py) are split by source into a 'forge' side the search may use and a 'bench' side only the fixed benchmark (eval/forge/bench.py) hears. The DSP toolkit for recipes: earshot/forge/dsp.py (sources: modal, noise, air, sweeps, Shepard risers, sub drops, bells, FM; layering) and earshot/forge/fx.py (reverb, saturation, transient shaping, limiter, EQ match, Doppler, filters, glitch, granular). In Python: from earshot.forge import forge.

forge v2 (0.2.0). Recipes come as families of named characters: 34 motion-design categories (whooshes, swishes, swipes, risers, downlifters, zips, impacts, sub drops, logo stings, clicks, taps, toggles, hover ticks, pops, notification dings, shimmers, glitches, key presses), 16 of them modelled on the designed effects of professional launch films (docs/motion-sfx-study.md), 4-6 variants each. A fixed benchmark scores them against held-out recordings the forge never heard. Against v1 on it: variants are no longer near copies (CLAP cosine between variants 0.89-0.97 -> 0.71-0.85 in the five categories v1 had; real sets 0.52-0.77), tails that died too fast now reach the real length (click t40 0.56 -> 1.44 x the bench median, fast whoosh 0.51 -> 0.88), and the tonal balance is closer in 3 of those 5 (key press 8.2 -> 7.4 dB, fast whoosh 8.6 -> 6.2, impact 6.7 -> 4.4). Across all 34, the made balance is at least as close to the held-out recordings as a second set of real recordings in 18 of the 21 categories that have a number, and every final re-renders from its report.json (earshot forge --check). The model checks that v1 had saturated fell. A clean blind-test kit for the v2 sounds is eval/forge/blind_v2.py; no v2 score is published yet (docs/SFX.md says the same), so the blind test below is the v1 one. What changed, every number, and what is still weak: docs/FORGE_V2.md.

The reference recordings are licensed and not in this repo, so --refset needs a local registry: bring your own refs with --refs DIR, or lay them out as described in earshot/forge/refsets.py. Without refs the critic runs in designed mode (realness reported, not required). The CLI defaults changed in 0.2.0 (pre-screen, seed bank, robust re-renders, the recipe's reference set); the Python forge() defaults still reproduce v1. See CHANGELOG.md.

Blind test (v1 recipes). A key press, a UI click, a pop, a fast whoosh and an impact were made from scratch in code and played blind next to real recordings of the same kind of sound. The author picked the made one correctly 8 times out of 15: a coin flip. Only the made impact was spotted (its tail dies too fast). One listener and 15 trials is a small test, not a proof; it says the made sounds are no longer easy to tell from real ones.

Honest limits (Goodhart). The search optimises earshot's own models, and those can be pleased without the sound getting better. After a search the made sounds pass every model check, held-out ones included, so the model checks stop telling made from real; the 1/3-octave measurement still can. The realness model is easy to fool (a modal resonator plus a room and a noise floor reads "real"). Judge a made sound by the model-free measurements, spectrograms next to the references, robust re-renders, and a human blind test.

earshot fit: audio to code