Sharing an idea for caption transcription

[JasChiang]

Hi maintainers, I would like to share an idea for caption transcription that grew out of this project, along with my thanks. Nothing here needs any action, and please feel free to close this whenever.

Thank you for building Palmier Pro. I learned a great deal from it. I forked it to experiment with caption features I wanted, then extracted the caption pipeline into a standalone repo to keep the experiments clean: JasChiang/caption-lab (GPL-3, provenance in NOTICE).

The idea in one line: keep Apple SpeechTranscriber as the only timing source, add an independent LLM transcription that never sees the ASR text, and make every semantic decision (corrections, caption line breaks, which disfluencies to cut) in one single pass, so nothing can contradict anything else. The details and the approaches that did not work are below, so there is no need to visit the repo unless you want the full lab log.

The architecture, in short

flowchart TB
    A[Apple ASR] -->|word timings| J
    M[Multimodal LLM] -->|independent transcript| J
    J[One semantic pass] -->|annotations| K[Deterministic mechanics]
    K --> O[Annotated transcript JSON]

Loading

• Apple ASR: the only timing source; its per-word clock is never discarded.
• Multimodal LLM: watches the whole video; produces an independent transcript plus speaker labels and domain terms; never sees the ASR text, so it cannot be anchored to ASR mistakes.
• One semantic pass: a single text call corrects the ASR lines against that reference and emits caption line breaks, never-split terms and disfluency cut marks together.
• Deterministic mechanics: count-locked 1:1 timing writeback (LCS aligned), energy-peak re-timing for recovered words, valley-snapped ripple cuts. No AI in this layer.
• Annotated transcript JSON: the pipeline's product; SRT, cut lists and markers are pure functions of it.

Three results that surprised me:

1. Time-stretching fast speech before ASR swaps errors instead of reducing them.
2. A forced aligner timed words worse than Apple's own ASR boundaries.
3. Letting an LLM pick cut words from a bare word list mis-cuts CJK reduplicated words like 常常. Moving cut marking into the sentence-level pass fixed it.

For transparency: the architectural direction is mine, and the implementation is almost entirely by an AI coding agent (credited in the commit history). Please treat the repo as reference data from an experiment rather than borrowable code.

Thanks again for your time and for making Palmier Pro. Wishing the project all the best!