/ai-provider-openai-whisper

name: ai-provider-openai-whisper
description: Speech-to-text transcription and translation via OpenAI Audio API -- models, response formats, timestamps, prompting, streaming, chunking, and diarization

OpenAI Whisper Patterns

> **Quick Guide:** Use `client.audio.transcriptions.create()` for speech-to-text and `client.audio.translations.create()` for non-English audio to English text. Choose `gpt-4o-transcribe` for highest accuracy, `gpt-4o-mini-transcribe` for cost-efficiency, `whisper-1` for timestamps/SRT/VTT, or `gpt-4o-transcribe-diarize` for speaker identification. Files must be under 25 MB -- chunk larger files. Use `prompt` to guide vocabulary and style. Streaming is available via `stream: true` for progressive output on `gpt-4o-transcribe` models.

---

<critical_requirements>

CRITICAL: Before Using This Skill

**(You MUST choose the correct model for the use case -- `gpt-4o-transcribe` for accuracy, `whisper-1` for timestamps/SRT/VTT output, `gpt-4o-transcribe-diarize` for speaker labels)**

**(You MUST chunk audio files larger than 25 MB before sending to the API -- the API rejects files exceeding this limit)**

**(You MUST pass `response_format: "verbose_json"` when using `timestamp_granularities` -- timestamps only work with this format on `whisper-1`)**

**(You MUST set `chunking_strategy: "auto"` when using `gpt-4o-transcribe-diarize` with audio longer than 30 seconds -- the API requires it)**

</critical_requirements>

---

**Auto-detection:** Whisper, whisper-1, gpt-4o-transcribe, gpt-4o-mini-transcribe, gpt-4o-transcribe-diarize, audio.transcriptions, audio.translations, transcription, speech-to-text, diarization, diarized_json, timestamp_granularities, verbose_json

**When to use:**

• Transcribing audio files (meetings, interviews, podcasts, voice notes) to text
• Translating non-English audio to English text
• Generating subtitles in SRT or VTT format from audio
• Getting word-level or segment-level timestamps for video editing
• Identifying speakers in multi-speaker audio (diarization)
• Streaming transcription results progressively as the model processes audio

**Key patterns covered:**

• Model selection (whisper-1 vs gpt-4o-transcribe vs gpt-4o-mini-transcribe vs gpt-4o-transcribe-diarize)
• Response formats (json, text, srt, vtt, verbose_json, diarized_json)
• Timestamps (word-level, segment-level) and subtitle generation
• Prompting for vocabulary, acronyms, and style
• Chunking large files (> 25 MB) with context preservation
• Streaming transcription with `stream: true`
• Translation to English via `audio.translations.create()`
• Speaker diarization with speaker references

**When NOT to use:**

• Text-to-speech (TTS) -- use the OpenAI TTS API (`client.audio.speech.create()`)
• Real-time bidirectional voice conversations -- use the OpenAI Realtime API
• Transcription with non-OpenAI providers -- use a provider-agnostic speech SDK

---

Examples Index

• [Core: Transcription, Translation, Timestamps, Chunking, Streaming, Diarization](examples/core.md) -- All audio API patterns

---

<philosophy>

Philosophy

The OpenAI Audio API provides **speech-to-text transcription and translation** through multiple models optimized for different needs. The API is simple -- you send an audio file and get text back -- but choosing the right model, response format, and parameters is critical for quality results.

**Core principles:**

1. **Model selection matters** -- `gpt-4o-transcribe` produces the highest accuracy with lower hallucination rates. `whisper-1` is the only model supporting SRT/VTT/verbose_json with timestamps. `gpt-4o-transcribe-diarize` adds speaker identification. 2. **File size is the primary constraint** -- 25 MB limit means you must chunk longer audio. Split at sentence boundaries to preserve context. 3. **Prompting improves accuracy** -- The `prompt` parameter guides vocabulary, acronyms, and formatting style. It does not give instructions -- it provides context the model matches against. 4. **Response format determines available features** -- Timestamps require `verbose_json` on `whisper-1`. Diarization requires `diarized_json`. SRT/VTT are only on `whisper-1`.

**When to use the Audio API:**

• You need accurate transcription of recorded audio files
• You need subtitles (SRT/VTT) from audio
• You need to identify who is speaking in a conversation
• You need to translate non-English speech to English text

**When NOT to use:**

• Real-time voice chat -- use the Realtime API instead
• Text-to-speech -- use `client.audio.speech.create()`
• You need transcription in a non-English target language (translation only outputs English)

</philosophy>

---

<patterns>

Core Patterns

Pattern 1: Basic Transcription

Send an audio file and receive text back. The model auto-detects the language.

const transcription = await client.audio.transcriptions.create({
  model: "gpt-4o-transcribe",
  file: createReadStream(audioPath),
});

Use `gpt-4o-transcribe` for highest accuracy. Do not use `whisper-1` with `verbose_json` when you only need plain text -- it adds overhead and has higher hallucination rates. See [core.md](examples/core.md) for full examples.

---

Pattern 2: Model Selection

Each model has distinct capabilities and tradeoffs.

What do you need?
+-- Highest accuracy, plain text -> gpt-4o-transcribe
+-- Cost-efficient, plain text -> gpt-4o-mini-transcribe
+-- Timestamps (word/segment) -> whisper-1 (verbose_json)
+-- SRT or VTT subtitles -> whisper-1 (srt/vtt format)
+-- Speaker identification -> gpt-4o-transcribe-diarize
+-- Streaming output -> gpt-4o-transcribe or gpt-4o-mini-transcribe

Model Capabilities Matrix

| Feature | whisper-1 | gpt-4o-transcribe | gpt-4o-mini-transcribe | gpt-4o-transcribe-diarize | | ---------------- | ---------------------------------- | ----------------- | ---------------------- | ------------------------- | | Response formats | json