gettr-transcribe-summarize

# Gettr Transcribe + Summarize (MLX Whisper) ## Quick start ```bash # 1. Parse the slug from the URL (just read it — no script needed) # https://gettr.com/post/p1abc2def → slug = p1abc2def # https://gettr.com/streaming/p3xyz → slug = p3xyz # 2. Get the video URL # For /post/ URLs: use the extraction script python3 scripts/extract_gettr_og_video.py "<GETTR_POST_URL>" # For /streaming/ URLs: use browser automation directly (extraction script is unreliable) # See Step 1 below for browser automation instructions # 3. Run download + transcription pipeline bash scripts/run_pipeline.sh "<VIDEO_URL>" "<SLUG>" ``` To explicitly set the transcription language (recommended for non-English content): ```bash bash scripts/run_pipeline.sh --language zh "<VIDEO_URL>" "<SLUG>" ``` Common language codes: `zh` (Chinese), `en` (English), `ja` (Japanese), `ko` (Korean), `es` (Spanish), `fr` (French), `de` (German), `ru` (Russian). This outputs: - `./out/gettr-transcribe-summarize/<slug>/audio.wav` - `./out/gettr-transcribe-summarize/<slug>/audio.vtt` Then proceed to Step 3 (Summarize) to generate the final deliverable. --- ## Workflow (GETTR URL → transcript → summary) ### Inputs to confirm Ask for: - GETTR post URL - Output format: **bullets only** or **bullets + timestamped outline** - Summary size: **short**, **medium** (default), or **detailed** - Language (optional): if the video is non-English and auto-detection fails, ask for the language code (e.g., `zh` for Chinese) Notes: - This skill does **not** handle authentication-gated GETTR posts. - This skill does **not** translate; outputs stay in the video's original language. - If transcription quality is poor or mixed with English, re-run with explicit `--language` flag. ### Prereqs (local) - `mlx_whisper` installed and on PATH - `ffmpeg` installed (recommended: `brew install ffmpeg`) ### Step 0 — Parse the slug and pick an output directory Parse the slug directly from the GETTR URL — just read the last path segment, no script needed: - `https://gettr.com/post/p1abc2def` → slug = `p1abc2def` - `https://gettr.com/streaming/p3xyz789` → slug = `p3xyz789` Output directory: `./out/gettr-transcribe-summarize/<slug>/` Directory structure: - `./out/gettr-transcribe-summarize/<slug>/audio.wav` - `./out/gettr-transcribe-summarize/<slug>/audio.vtt` - `./out/gettr-transcribe-summarize/<slug>/summary.md` ### Step 1 — Get the video URL The approach depends on the URL type: #### For `/post/` URLs — Use the extraction script Run the extraction script to get the video URL from the post HTML: ```bash python3 scripts/extract_gettr_og_video.py "<GETTR_POST_URL>" ``` This prints the best candidate video URL (often an HLS `.m3u8`) to stdout. If extraction fails, ask the user to provide the `.m3u8`/MP4 URL directly (common if the post is private/gated or the HTML is dynamic). #### For `/streaming/` URLs — Use browser automation directly **Do not use the extraction script for streaming URLs.** The `og:video` URL from static HTML extraction is unreliable for streaming content — it either fails outright or the download stalls and fails near the end. Instead, use browser automation to get a fresh, dynamically-signed URL: 1. Open the GETTR streaming URL and wait for the page to fully load (JavaScript must execute) 2. Extract the `og:video` meta tag content from the rendered DOM: ```javascript document.querySelector('meta[property="og:video"]').getAttribute('content') ``` 3. Use that fresh URL for the pipeline in Step 2 If browser automation is not available or fails, see `references/troubleshooting.md` for how to guide the user to manually extract the fresh URL from their browser. ### Step 2 — Run the pipeline (download + transcribe) Feed the extracted video URL and slug into the pipeline: ```bash bash scripts/run_pipeline.sh "<VIDEO_URL>" "<SLUG>" ``` To explicitly set the language (recommended when auto-detection fails): ```bash bash scripts/run_pipeline.sh --language zh "<VIDEO_URL>" "<SLUG>" ``` The pipeline does two things: 1. Downloads audio as 16kHz mono WAV via ffmpeg 2. Transcribes with MLX Whisper, outputting VTT with timestamps #### If the pipeline fails with HTTP 412 (stale signed URL) This error occurs with `/streaming/` URLs when the signed URL has expired. If browser automation returned a stale URL, retry by re-running browser automation to get a fresh URL, then retry the pipeline. If browser automation is not available or fails, see `references/troubleshooting.md` for how to guide the user to manually extract the fresh URL from their browser. Notes: - By default, language is auto-detected. For non-English content where detection fails, use `--language`. - If too slow or memory-heavy, try smaller models: `mlx-community/whisper-medium` or `mlx-community/whisper-small`. - If quality is poor, try the full model: `mlx-community/whisper-large-v3` (slower but more accurate). - If `--word-timestamps` causes issues, the pipeline retries automatically without it. ### Step 3 — Summarize Write the final deliverable to `./out/gettr-transcribe-summarize/<slug>/summary.md`. Pick a **summary size** (user-selectable): - **Short:** 5–8 bullets; (if outline) 4–6 sections - **Medium (default):** 8–20 bullets; (if outline) 6–15 sections - **Detailed:** 20–40 bullets; (if outline) 15–30 sections Include: - **Bullets** (per size above) - Optional **timestamped outline** (per size above) Timestamped outline format (default heading style): ``` [00:00 - 02:15] Section heading - 1–3 sub-bullets ``` When building the outline from VTT cues: - Group adjacent cues into coherent sections. - Use the start time of the first cue and end time of the last cue in the section. ## Bundled scripts - `scripts/run_pipeline.sh`: download + transcription pipeline (takes a video URL and slug) - `scripts/extract_gettr_og_video.py`: fetch GETTR HTML and extract the `og:video` URL (with retry/backoff) - `scripts/download_audio.sh`: download/extract audio from HLS or MP4 URL to 16kHz mono WAV ### Error handling - **Non-video posts**: The extraction script detects image/text posts and provides a helpful error message. - **Network errors**: Automatic retry with exponential backoff (up to 3 attempts). - **No audio track**: The download script validates output and reports if the source has no audio. - **HTTP 412 errors**: Occurs with `/streaming/` URLs when the signed URL has expired. Re-run browser automation to get a fresh URL (see Step 1); if that fails, see `references/troubleshooting.md`. ## Troubleshooting See `references/troubleshooting.md` for detailed solutions to common issues including: - HTTP 412 errors (stale signed URLs) - Extraction failures - Download errors - Transcription quality issues