Soup
Fine-tune LLMs from one YAML. Layer streaming trains an 8B model on a 4 GB laptop GPU.
What is it?
What it is
Soup is a tool for fine-tuning and post-training large language models (LLMs) with a simplified workflow. It enables users to fine-tune models in one command, automating tasks like quantization, batch sizing, and GPU detection while supporting techniques like LoRA, layer streaming, and DPO.
Why it exists
Soup exists to address the pain points of LLM fine-tuning, where teams often spend significant time on infrastructure rather than model improvement. It automates complex workflows, reduces manual configuration, and enables efficient local training on consumer-grade hardware.
Who should use it
Developers, researchers, and ML engineers who want to fine-tune LLMs locally with minimal setup, especially those with limited GPU VRAM (e.g., 4-8 GB) seeking to avoid infrastructure complexity.
Who should avoid it
Users requiring multi-node distributed training, those needing GRPO/PPO support (excluded by design), or teams dependent on legacy workflows involving SSH-based GPU management.
How it works
A quick walkthrough in plain English
How Soup works
Step 1 of 3
You interact with it
Open Soup, send a request, or connect it to your stack.
Features
Advantages
- Fine-tune 8B LLMs on consumer GPUs as low as 4GB VRAM
- Eliminates SSH and infrastructure complexity
- Single YAML config handles all training parameters
- Automatic optimization of batch size, quantization, and layer streaming
- Works fully locally — no cloud dependency
- Bit-exact correctness verified via layer streaming protocol
- Memory-efficient reference model handling in DPO/KTO
- Extensive format and model support via Hugging Face integration
- Rich export options for deployment (Ollama, llama.cpp, etc.)
- Active development with frequent updates and benchmarks
- Strong community support via Discord, docs, and tutorials
- Free and open-source (Apache-2.0) with transparent development
- GPU-accelerated Docker image for reproducible environments
- Includes tools for data inspection, evaluation, and model card generation
Disadvantages
- Layer streaming is still marked as BETA in recent versions
- GRPO and PPO are intentionally excluded due to incompatibility with streaming
- Training speed may be impacted by 1.52x layer reads in DPO vs SFT
- Requires Python 3.10+ and CUDA-capable GPU for practical training
- CPU fallback is available but very slow for training
- Some advanced features require installing specific extras (e.g., [fast], [serve])
- Documentation can be overwhelming due to extensive feature set
- Streaming functionality may have edge cases with adapter loading (fixed in v0.72.1+)
- Multi-GPU and DeepSpeed/FSDP support is documented but may require additional setup
- Windows users must use double quotes for pip extras due to shell parsing limitations
Installation
docker
docker pull ghcr.io/makazhanalpamys/soup:latest # Run Soup without installing CUDA or PyTorch locally docker run --gpus all -v $(pwd):/workspace ghcr.io/makazhanalpamys/soup train --config soup.yaml
native
pip install soup-cli # Add the training stack (torch, transformers, peft, trl, datasets, …) pip install "soup-cli[train]" # Everything (train + serve + ui + data) in one shot pip install "soup-cli[all]" # Or from GitHub (latest dev) pip install git+https://github.com/MakazhanAlpamys/Soup.git
compose
docker compose up # or build locally
FAQ
How can I fine-tune large models on hardware with limited VRAM?
You can use Soup's 'layer streaming' feature (enabled via `stream_layers: true` in your config). This keeps the frozen base model in system RAM and feeds it to the GPU one decoder layer at a time, allowing you to train models like Llama-3.1-8B on as little as 4 GB of VRAM.
What is the difference between the different installation methods?
Use `pip install soup-cli` for a light installation (CLI, config, and data tools). Use `pip install "soup-cli[train]"` to add the training stack (PyTorch, transformers, etc.), or `pip install "soup-cli[all]"` for everything including serving and UI.
Does layer streaming affect the quality of my trained model?
No. Soup ensures that streamed runs are bit-exact against normal, non-streamed runs, maintaining a 0.0 difference in loss.
Which training tasks and preference losses are supported?
Soup supports Supervised Fine-Tuning (SFT) and various preference optimization methods including DPO, ORPO, SimPO, and KTO.
How do I verify if my environment and GPU are correctly configured?
You can run the command `soup doctor` to check your GPU, system resources, dependencies, and version information in one place.
Can I export my trained models for use in other applications?
Yes, you can use the `soup export` command to convert your model to formats like GGUF (for Ollama/llama.cpp), ONNX, TensorRT, AWQ, or GPTQ.
Featured in Videos
YouTube tutorials and walkthroughs for Soup
Alternatives
Similar projects ranked by category, topics, and text overlap.
