ds4
DeepSeek 4 Flash and PRO local inference engine for Metal, CUDA and ROCm
What is it?
What it is
A small native inference engine optimized for DeepSeek V4 Flash and PRO, providing self-contained loading, prompt rendering, tool calling, KV state handling (RAM and on-disk), server API, and integrated coding agent.
Why it exists
To enable credible local inference of large open weight models (like DeepSeek V4) on high-end personal machines starting from 96/128GB RAM, with end-to-end usability through official validation, long-context testing, and agent integration, taking a narrow bet on one model at a time.
Who should use it
Users with high-end machines (96GB+ RAM, Mac Studios, DGX Spark, etc.) who want to run DeepSeek V4 Flash/PRO locally for coding agent integration or experimentation, and are comfortable with AI-developed code.
Who should avoid it
Users unhappy with AI-developed code, those with low-end hardware (<96GB RAM without fast SSD for streaming), those seeking a generic GGUF runner, or those needing stable production tools.
How it works
A quick walkthrough in plain English
How ds4 works
Step 1 of 3
You interact with it
Open ds4, send a request, or connect it to your stack.
Features
Advantages
- Optimized for capable open weight models (DeepSeek V4 Flash/PRO) that resist 2-bit quantization
- Designed for high-end personal machines (MacBooks, DGX Spark, etc.) with 96GB+ RAM
- KV cache as a disk citizen enables very big contexts using fast SSD storage
- SSD streaming allows running models larger than RAM by caching routed MoE experts
- Distributed inference enables running models too large for one machine and speeds up prefill
- Power usage control reduces heat, fan noise, and thermal stress without changing model output
- Native agent provides low latency, live progress bar, native tool handling, and impossible KV cache mismatches
- Session persistence via disk KV cache allows saving and resuming sessions without prefill
- Compatible with popular coding agents (via OpenAI/Anthropic APIs)
- Built upon the reliable foundation of llama.cpp and GGML
- Specific optimizations for Metal and CUDA backends for performance
- CPU path available for correctness checks and diagnostics
- Comprehensive tooling for GGUF generation, imatrix, quality testing, and benchmarking
- Beta quality but usable and making progress
- Acknowledgements show gratitude and reliability
Disadvantages
- Beta quality code (may have issues, not yet stable)
- Only works with specific GGUF files provided (not a general GGUF loader)
- CPU inference on macOS is not usable due to a virtual memory bug that crashes the kernel
- Distributed generation is slower than single-process due to cross-machine communication
- SSD streaming is slower than having the model fully in RAM
- Requires high-memory machines (starting from 96/128GB RAM) for optimal use
- Opportunistic nature: may drop support for old models when new ones are released
- Agent is alpha quality (not ready for prime time)
- Server does not batch multiple requests (concurrent requests wait their turn)
- Disk KV cache only holds one live session in memory (switching sessions requires disk)
- Distributed inference protocol is not release-stable and requires same commit and trusted network
- No encryption or authentication in the distributed protocol
- The `--power` feature is an experimental workaround (inserting sleeps)
- Narrow focus: one model at a time (may require switching for different models)
Installation
native
make; make cuda-spark; make cuda-generic; make cpu
FAQ
How do I download the model weights for DwarfStar?
Use the provided download script. For example, for 96/128 GB RAM machines with imatrix-tuned q2, run: `./download_model.sh q2-imatrix`. The script downloads from Hugging Face and stores files under `./gguf/`, updating `./ds4flash.gguf` to point at the selected model.
How do I build DwarfStar for different backends?
For macOS Metal (default): `make`. For Linux CUDA on DGX Spark/GB10: `make cuda-spark`. For Linux CUDA on other GPUs: `make cuda-generic`. For CPU-only diagnostics: `make cpu`.
How can I run a model that is larger than my RAM using SSD streaming?
Use the `--ssd-streaming` flag. For example, on a 64GB MacBook with the 2-bit Flash GGUF: `./ds4 -m ./ds4flash.gguf --ssd-streaming --ssd-streaming-cache-experts 32GB --ctx 32768 --nothink`. The automatic cache budget is recommended; if setting manually, start around 32GB and adjust based on startup logs.
How do I set up distributed inference across two machines?
Assign one machine as coordinator and the other as worker. Coordinator: `./ds4 -m <gguf> --role coordinator --layers 0:30 --listen <IP> <PORT>`. Worker: `./ds4 -m <gguf> --role worker --layers 31:output --coordinator <IP> <PORT>`. Replace `<gguf>` with the appropriate split model (e.g., PRO Q4 layer-split GGUFs) and adjust layer ranges as needed.
How do I start the OpenAI-compatible server?
Run `./ds4-server --ctx 100000 --kv-disk-dir /tmp/ds4-kv --kv-disk-space-mb 8192` to start a local server. The server provides endpoints like `/v1/chat/completions` and is compatible with OpenAI/Anthropic clients.
How do I reduce GPU power usage to lower heat and fan noise?
Use the `--power N` flag, where N is a percentage (e.g., `--power 50` targets about 50% GPU usage). This is available on the CLI, server, agent, and benchmark tools.
Featured in Videos
YouTube tutorials and walkthroughs for ds4
Alternatives
Similar projects ranked by category, topics, and text overlap.
