verse-cue — Speech-to-Text Slide Automation for ProPresenter

The problem

I wanted the vocal mic to do that click. The ProPresenter API only exposes the slide already on screen, so there is no next line to listen for. Whisper was not built for sung lyrics: it mishears sung words, and repeated lines and choruses look the same to a naive matcher. Transcription also lags the singer, so a click that waits for the last word to be heard is already late.

Why it mattered

Two errors pull in opposite directions. A false Next fires before the slide's midpoint, earlier than any operator would click. A miss never fires, and the operator has to click anyway. A tool that fixes one error by inflating the other is no help, so I needed both numbers measured on every change.

What I built

I built a two-file Python tool: verse_cue.py runs the live loop and harvest.py downloads bench songs with synced lyrics, builds the alias table, and scores configurations. Every second it transcribes the last 4 seconds of the vocal feed with faster-whisper word timestamps. It maps known mishearings through a 138-entry alias table, aligns the words to the current slide with difflib, predicts when the singer reaches the last word, and calls /v1/trigger/next 0.3 s before it. Blank instrumental slides advance on the first voice (Silero VAD). Production, the bench, and the tests all call the same run() loop with no internal mocking. One TOML file holds every threshold, and install is three commands.

Key decisions

Bias Whisper with the slide text (rejected) vs. no prompt (chosen). On tiny.en, passing the slide to the decoder took misses down to 1.9%, but Whisper copied the lyric it was given, so opening words "matched" and false Nexts jumped to 71.3%. I kept prompt_mode = none.

Bigger model alone (rejected) vs. small model plus arming gates (chosen). distil-large-v3 cut misses from 28.6% to 13.4% but raised false Nexts from 26.5% to 30.6%: more heard words armed the timer earlier. The bench showed that false Nexts came from when the timer may arm, not from transcription accuracy. So I chose small.en with three gates: the match must land in the last 3 words, past the slide midpoint, with at least 4 words matched. It then waits from a real tail hearing at the measured singing rate. That pass cut false Nexts from 24.3% to 14.6% at the 0.5 s hop while misses held (24.9% → 25.9%).

Deadline click (rejected). Firing on a clock after any match cut misses to 15.7% at 1.0 s/word, but false Nexts rose to 30.3%. At 0.45 s/word, false Nexts reached 88%.

Hop 0.5 s (rejected) vs. hop 1.0 s (chosen). The best row was measured at a 0.5 s hop, but two of six songs ran at 3.64× and 3.91× real time, under the 4× speed gate. I shipped the 1.0 s hop.

Three overlapping audio streams (rejected). The original plan ran three offset 3-second streams. On one GPU they serialize and equal a single sliding window, so I kept one code path and let the bench sweep window and hop sizes.

Outcome

What a production version needs

An open-source tool with one decision loop, not a managed product. The published numbers come from a bench, not from a service:

• Studio mix, not a live vocal. Bench audio is the full song mix. The README says live numbers will move, and asks users to send back their metrics.jsonl. The live count of 2 manual clicks in about 100 slides is an operator report, not a committed metric.
• Misses are still 25.9% on the bench. On the studio-mix bench, about one slide in four still needs a manual click.
• The shipped hop is not the headline row. The 14.6% / 25.9% row was measured at a 0.5 s hop. The shipped config uses 1.0 s for speed headroom.
• Timing precision. Word timestamps are about ±100 ms, and the remaining-words wait is a rate estimate, not syllable-exact.
• Hardware. On CPU only, small.en may run under 4× real time.