hayamimi
早耳 - Real-time multilingual speech-to-text on CPU only. Live subtitles, browser dashboard, speaker labels, translation. No GPU, no cloud.
What is it?
What it is
Real-time, multilingual speech-to-text system running entirely on CPU via quantized ONNX models. Provides live subtitles, speaker labels, on-the-fly translation, and a browser dashboard with sub-100ms final latency. Japanese for 'quick ear', designed for low-resource, high-accuracy transcription.
Why it exists
Most CPU-only setups default to general-purpose models like Whisper, accepting their accuracy ceiling. hayamimi instead routes each utterance to a language-specific specialist model—ReazonSpeech for Japanese, Paraformer-zh for Chinese, SenseVoice for Korean/Cantonese, Parakeet for English and 24 EU languages, and Omnilingual as fallback—all quantized to INT8 via sherpa-onnx. This achieves lower error rates (e.g., 5.8% Japanese CER vs 13.8% for Whisper on broadcast audio) while running 10-50x realtime on CPU, with memory bounded under 2GB.
Who should use it
Content creators and streamers needing live OBS subtitlesDevelopers building multilingual speech-to-text applicationsResearchers working with ASR systems and speech processingPrivacy-conscious users wanting fully offline transcriptionPeople needing high-accuracy Japanese speech recognition (5.8% CER vs Whisper's 13.8%)Multilingual event organizers requiring real-time captioningAccessibility tool developers for hearing-impaired users
Who should avoid it
Complete beginners without any CLI or Python experienceUsers requiring GPU acceleration (this project is CPU-only by design)Those needing code-switching support within sentences (not supported)Users on extremely constrained hardware (needs ~2GB RAM minimum)Anyone needing medical/financial transcription accuracy (translation has known quality limitations)Users expecting true speaker diarization (only does turn-taking labeling)
How it works
A quick walkthrough in plain English
How hayamimi works
Step 1 of 3
You interact with it
Open hayamimi, send a request, or connect it to your stack.
Features
Advantages
- Significantly better accuracy than standard Whisper-based solutions (5.8% CER for Japanese vs 13.8% for whisper-large-v3-turbo on real broadcast audio)
- Runs at 10-50x realtime on CPU with under 2GB RAM footprint
- In-progress partial subtitles provide immediate feedback during live speaking
- High-quality two-pass refinement improves final transcript accuracy
- Native speaker labeling for turn-taking in multilingual conversations
- Supports live translation of Japanese to multiple languages in real-time
- Works entirely on CPU without requiring GPU or cloud APIs
Disadvantages
- Code-switching mid-sentence is not supported; mixed-language sentences may have portions mangled or dropped
- Very short utterances following background music or BGM bursts can be misrouted due to language-switch guards
- First utterance always initializes with SenseVoice model, potentially causing initial language misconfirmation
- --hotwords currently has no effect on the Japanese (ReazonSpeech) tier due to incompatible token encoding
- Simultaneous speech from multiple speakers cannot be separated (only turn-taking labeling is provided)
- Translation quality has inherent limitations (numeric values not reliably preserved in ja→zh/ko translation)
- End-to-end microphone pipeline has not been independently verified beyond the project's own testing
Installation
native
Requirements: Python 3.10+ and ffmpeg on PATH Create virtual environment: python -m venv .venv Install dependencies: Windows: .venv\Scripts\pip install -r requirements.txt macOS/Linux: .venv/bin/pip install -r requirements.txt Download models (~3.1GB): Windows: .venv\Scripts\python scripts/download_models.py macOS/Linux: .venv/bin/python scripts/download_models.py Optional minimal install (~1.1GB ja/en-only): python scripts/download_models.py --minimal Run transcription: Windows: .venv/Scripts/python scripts/realtime_transcribe.py macOS/Linux: .venv/bin/python scripts/realtime_transcribe.py
FAQ
What is hayamimi and what makes it different from other CPU-only speech-to-text tools?
hayamimi (早耳, "quick ear") is a real-time, multilingual speech-to-text system that runs on CPU only with under 2GB RAM. Unlike most setups that fall back to a single general-purpose Whisper model, hayamimi routes each utterance to a dedicated specialist model best suited for its language (ja/zh/ko/yue/en+24 EU languages each get their own best-in-class model, with Meta's Omnilingual ASR handling the remaining ~1600 languages). On real broadcast Japanese audio this achieves 5.8% CER vs. whisper-large-v3-turbo's 13.8%, while running 10-50x realtime on a 6-core CPU.
What are the system requirements and how do I install hayamimi?
hayamimi requires Python 3.10+ and ffmpeg on PATH. It's developed and tested on Windows 11; macOS/Linux are expected to work but aren't CI-tested end-to-end yet. To install: create a venv (`python -m venv .venv`), run `pip install -r requirements.txt`, then run `scripts/download_models.py` to pull ~3.1GB of pretrained models (or pass `--minimal` for a ~1.1GB ja/en-only install). Finally run `scripts/realtime_transcribe.py` for mic transcription, or add `--serve` to also launch the browser dashboard and OBS overlay at http://localhost:8833.
How do I start the dashboard and OBS overlay for live subtitles?
Run `scripts/realtime_transcribe.py --serve` (optionally with a custom port). This exposes three views at that port: `/dashboard` for the live dashboard with partial-text strips, language badges, speaker chips, per-line latency, inline translations, and a refined-transcript column; `/` for a minimal OBS browser-source overlay (append `?show=final` or `?show=partial` to render just one row as its own OBS source); and `/transcript` for a plain scrolling transcript history.
Can hayamimi accept audio from a phone or ESP32 over the network instead of a local microphone?
Yes. Run `scripts/realtime_transcribe.py --input ws --serve` to start a WebSocket ingest endpoint at `ws://<host>:8766/ingest` (host configurable via `--ws-host`, port via `--ws-port`). The protocol is: send one JSON text frame (`{"sr": 16000, "format": "pcm_s16le", "channels": 1}`), then stream raw `pcm_s16le` audio as binary frames. The server resamples non-16kHz audio automatically, replies with the same partial/final/translation/refine JSON events the dashboard SSE stream uses, and accepts only one audio-producing client at a time. `scripts/ws_mic_client.py` is a dependency-free reference implementation that doubles as a template.
How accurate and fast is hayamimi on different languages?
On real speech without preroll/two-pass (single clips), measured performance is: ja 7.5% CER (ReazonSpeech), en 2.3% WER (Parakeet v3), zh 5.3% CER (Paraformer-zh), ko 8.1% CER (SenseVoice), yue 6.1% CER (SenseVoice, t2s-normalized). Real-time factors are 0.061-0.109, meaning each route runs roughly 9-16x faster than realtime on CPU alone. With two-pass refinement, Japanese real-broadcast CER improves from 15.5% to 12.0%. Mean final latency is ~100ms for Japanese (~236ms mean / 552ms max across a 5-language soak test with every feature enabled).
What are the known limitations I should be aware of before using hayamimi?
Key honest limitations from the README: (1) code-switching mid-sentence isn't supported — the router picks one language per utterance, so word-level mixing gets mangled; (2) `--hotwords` currently has no effect on the Japanese (ReazonSpeech) tier because of byte-level BPE token incompatibility — use `--replace` for ja proper nouns instead; (3) two overlapping speakers aren't separated — `--speakers` does turn-taking labeling, not true diarization; (4) translation quality has a real ceiling (numeric values aren't reliably preserved in ja->zh/ko); (5) very short utterances after jingles/BGM can misroute, mitigated by the `--lang-switch-guard` and `--lid-switch-confirm` flags; (6) the ja->en FuguMT translation model is CC BY-SA 4.0 (share-alike), unlike everything else which is MIT/permissive.
Featured in Videos
YouTube tutorials and walkthroughs for hayamimi
Alternatives
Similar projects ranked by category, topics, and text overlap.
