deltafin
Run full Kimi K3 on a single device. And an OpenAI-compatible API server for local chat and coding agents.
What is it?
What it is
Deltafin is a research experiment that runs the 2.8 TB Kimi K3 Mixture‑of‑Experts model on a single local workstation by loading a resident spine from disk and fetching only the 16 experts per layer that the router selects, using native kernels on MPS, CUDA or CPU.
Why it exists
It proves that a very large MoE can be run locally without a GPU cluster, by exploiting expert routing, disk‑cached experts, and platform‑specific optimizations, and provides a testbed for streaming‑inference techniques.
Who should use it
Researchers, developers, and AI enthusiasts interested in running large MoE models locally, especially on Apple Silicon or Linux systems.
Who should avoid it
Users without sufficient hardware resources (e.g., <64GB RAM), non-technical users, or those unable to meet Python 3.12+ and compiler requirements.
How it works
A quick walkthrough in plain English
How deltafin works
Step 1 of 3
You interact with it
Open deltafin, send a request, or connect it to your stack.
Features
Advantages
- Enables inference of a 2.8 TB MoE model on a single workstation without cloud costs
- Provides a reference implementation that is fully open source and MIT‑licensed
- Offers a streaming mode that reduces disk footprint to ~215 GB while still allowing on‑demand expert loading
- Supports both Apple Silicon and Linux with automatic fallback to CPU when GPU is unavailable
- Int8 spine and output head cut I/O by ~50 % without noticeable quality loss
- OpenAI‑compatible API makes it easy to integrate into existing tooling and agents
- Built‑in caching and prefetching reduce network latency for streaming installs
- Extensible configuration allows fine‑grained performance tuning
- Command‑line utilities simplify setup, upgrade, and debugging
- Provides a research platform for exploring MoE inference optimizations
Disadvantages
- Requires operational ownership for self-hosted setups
- Community support varies by project maturity
Installation
FAQ
What are the hardware requirements for running Deltafin?
Deltafin supports Apple Silicon macOS and x86-64/aarch64 Linux. It requires Python 3.12+ and a C/C++ compiler (Xcode Command Line Tools on macOS or build-essential/GCC/Clang on Linux). For NVIDIA acceleration, a CUDA-enabled PyTorch build is required.
How much disk space do I need for the model?
The amount of space depends on the installation mode: the '--full' mode requires approximately 1.7 TB of disk space, while the '--stream' mode requires approximately 215 GB. The full mode is recommended for significantly faster inference speeds.
What is the difference between '--full' and '--stream' installation modes?
The '--full' mode downloads the entire model locally, allowing for much faster inference (e.g., ~14.6 s/token on an M1 Max). The '--stream' mode downloads experts from Hugging Face on demand, which uses much less disk space but results in significantly slower inference (e.g., ~3 min/token) due to network latency.
Can I upgrade Deltafin without re-downloading the model?
Yes. Once installed, you can use `./venv/bin/python tools/upgrade.py`. This command is conservative: it updates dependencies and rebuilds native libraries but does not run model setup or conversion, leaving your downloaded models and caches intact.
Does Deltafin support an OpenAI-compatible API?
Yes, Deltafin can serve an OpenAI-compatible server using `./venv/bin/python tools/serve_openai.py --port 8000`. This allows you to use the standard OpenAI SDK or other chat interfaces by pointing the base URL to your local server.
How does Deltafin handle different hardware accelerators?
Deltafin automatically selects the best available backend: MPS for Apple Silicon, CUDA for NVIDIA GPUs, or CPU. It uses native MXFP4 expert kernels for Metal, CUDA, or CPU (via SIMD like AVX2 or NEON) to optimize performance.
Featured in Videos
YouTube tutorials and walkthroughs for deltafin
Alternatives
Similar projects ranked by category, topics, and text overlap.
