Speech translation & video
Beyond speech in and out, AiHummer offers two additional media capabilities: speech translation and video understanding. Both run on free/local engines: translation is assembled from the STT/TTS sidecars and the LLM you already have, while video runs as its own sidecar reached by URL.
[!NOTE] Speaker diarization and voice cloning are not part of the product — those capabilities were removed. If you saw them in older material, do not plan around them: the voice stack is STT, TTS, the voice turn (
/v1/voice/turn), speech translation and video understanding.
Speech translation
Speech translation turns audio in one language into spoken audio in another, so an agent can serve callers across languages. It is a pure translation pipeline with no agent turn: the inbound audio is transcribed (STT), the text is translated with a one-shot LLM completion, and the translation is spoken back (TTS).
audio in ─▶ STT ─▶ translate (LLM) ─▶ TTS ─▶ audio outNo extra sidecar is needed: the pipeline uses the same free/local sidecars as
the voice turn — faster-whisper (:8001) and edge-tts (:8002) — plus your
configured LLM provider.
| Item | Value |
|---|---|
| Endpoint | POST /v1/voice/translate?target=<lang> |
target parameter |
Target language (BCP-47 or a plain language name); required |
| Input | Audio in the request body (Content-Type, defaults to audio/ogg) |
| Output | Translated audio (audio/mpeg) |
| Response headers | X-Voice-Transcript — source text, X-Voice-Reply — the translation |
| Requires | STT, TTS and an LLM provider configured |
POST /v1/voice/translate?target=en HTTP/1.1
Host: localhost:8780
Content-Type: audio/ogg
<binary audio>Video understanding
Video understanding extracts what a clip contains by demuxing audio and sampling keyframes with ffmpeg — there is no ML model in this sidecar; it prepares audio and frames for the agent turn.
| Item | Value |
|---|---|
| Backed by | ffmpeg (demux + keyframes, no ML) |
| Sidecar port | :8005 |
| Endpoint | POST /v1/video/understand |
| Wired with | AIHUMMER_VIDEO_URL |
Wiring
Speech translation needs no dedicated variable: it is active once STT, TTS and an LLM provider are configured. Video understanding is enabled by its sidecar URL — the host-native installer can provision it, or point at an instance you already run.
# gateway.env — optional video sidecar
AIHUMMER_VIDEO_URL=http://127.0.0.1:8005[!TIP] The capabilities are independent — enable only the ones you need. A deployment that just needs transcription and synthesis can run the STT/TTS pair alone and leave the video sidecar unconfigured.
Where to next
- The default speech path: Speech in & out.
- How sidecars are wired and shared: Sidecars.