kimi-k3-mlx
MLX port of moonshotai/Kimi-K3 (2.78T multimodal MoE): streaming converter, REAP expert pruning, and per-language expert-overlap analysis
What is it?
What it is
An MLX (Apple Silicon) port of the Kimi-K3 multimodal MoE model, featuring text and vision towers with Mixture of Experts (MoE), KDA attention, AttnRes, and SiTU-GLU activation. Optimized for Apple Silicon with MXFP4 quantization and pruning strategies.
Why it exists
To enable the large Kimi-K3 model (2.78T total parameters) to run efficiently on Apple Silicon devices by optimizing memory usage through quantization, pruning, and specialized architecture adaptations.
Who should use it
Developers, researchers, and ML engineers who need to run or fine‑tune the large Kimi‑K3 multimodal model on Apple Silicon (M1/M2/M3) hardware and are comfortable with MLX and custom model registration.
Who should avoid it
Users without Apple Silicon machines, those with less than 512 GB unified memory, or beginners who lack experience with large‑scale model conversion, quantization, and custom MLX integration.
How it works
A quick walkthrough in plain English
How kimi-k3-mlx works
Step 1 of 3
You interact with it
Open kimi-k3-mlx, send a request, or connect it to your stack.
Features
Advantages
- Bit-exact MXFP4 tier provides zero-error reproduction of source weights
- Pruning allows running massive models on consumer Apple Silicon (e.g., M3 Ultra)
- Validated end-to-end vision parity against Torch reference (1.5e-6 error)
- Granular control over model size vs. quality via domain-targeted pruning
- Efficient memory management via streaming conversion and layer-wise processing
Disadvantages
- Unpruned model is impossible to run on any existing Apple Silicon machine
- Pruning reduces decode speed (e.g., 2.06x penalty for graded two-bank builds)
- 2-bit quantization is a double-quantization and incurs quality loss
- AWQ is fragile and can be domain-skewed depending on calibration
- Pruning a specific domain (e.g., Chinese) can cause total capability collapse in that domain
Installation
FAQ
Why can't I run the unpruned Kimi-K3 model on a single Apple Silicon machine?
The unpruned model requires 883 GB of memory, which exceeds the 512 GB unified memory limit of the largest Apple Silicon machine (M3 Ultra Mac Studio). Only pruned tiers (like those in the Published section) fit within 512 GB.
What is the smallest pruned tier that runs on a 512 GB machine and what is its token generation speed?
The smallest pruned tier is the 350 GB build with 179 experts (REAP80), which achieves 5.54 tokens per second on a 512 GiB M3 Ultra.
How does the multimodal wrapper handle image placeholders in the prompt?
The processor rewrites each `<|kimi_image_placeholder|>` into a sequence that includes exactly one `<|media_pad|>` token per image. This token must expand into the entire feature block of the image (e.g., 256 tokens for a 448x448 image), which is critical for the model to access all visual information. A same-length scatter would discard most of the features and lead to a blind model.
What is the purpose of the REAP expert pruning and how does it affect the model?
REAP pruning selects the most salient experts per layer based on a calibration set, allowing the model to fit within memory constraints. It does not improve speed (since per-token traffic depends on top_k and non-expert precision) but enables deployment on memory-constrained hardware. The graded two-bank variant (which allocates bit-width instead of discarding experts) provides ~35% more experts at half the decode speed due to synchronization overhead.
How do I register the Kimi-K3 model with mlx-lm for usage?
After installing mlx-lm, copy the `kimi_k3.py` file from the converted model repository (e.g., from Hugging Face) into the mlx-lm models directory. For example: pip install mlx-lm python - <<'PY' import os, shutil, mlx_lm from huggingface_hub import hf_hub_download dst = os.path.join(os.path.dirname(mlx_lm.__file__), "models", "kimi_k3.py shutil.copy(hf_hub_download("pipenetwork/Kimi-K3-MLX-mxfp4", "kimi_k3.py\)), dst) print("registered kimi_k3 ->", dst) PY
Featured in Videos
YouTube tutorials and walkthroughs for kimi-k3-mlx
Alternatives
Similar projects ranked by category, topics, and text overlap.
