DeepSpec
DeepSpec: a full-stack codebase for training and evaluating speculative decoding algorithms
What is it?
What it is
DeepSpec is a full-stack Python codebase for preparing data, training, and evaluating draft models used in speculative decoding.
Why it exists
It provides a reproducible pipeline to train draft models (DSpark, DFlash, Eagle3) against target LLMs and measure speculative-decoding acceptance, enabling researchers to reproduce paper results and experiment with new algorithms.
Who should use it
Researchers and machine learning engineers interested in speculative decoding, draft model training, and LLM inference acceleration; users with access to multi-GPU nodes and experience training large language models.
Who should avoid it
Beginners without deep learning background, users lacking sufficient GPU resources (e.g., <8 GPUs), or those only seeking high-level APIs for model usage.
How it works
A quick walkthrough in plain English
How DeepSpec works
Step 1 of 3
You interact with it
Open DeepSpec, send a request, or connect it to your stack.
Features
Advantages
- Enables faster inference via speculative decoding
- Provides pretrained checkpoints that match paper results
- Modular design allows easy addition of new algorithms
- Integrates with established codebases (SpecForge, DFlash)
- Clear, step-by-step workflow from data prep to evaluation
- Open-source MIT license facilitates reuse and extension
- Supports multiple popular target model families (Qwen3, Gemma)
Disadvantages
- Target cache generation can require very large storage (~38 TB for default Qwen3-4B)
- Data preparation depends on an external inference engine to serve the target model
- Default configuration assumes a single node with 8 GPUs, limiting accessibility for smaller setups
- Training and evaluation demand significant computational resources
- Currently limited to a predefined set of target models and algorithms
- Evaluation benchmarks, while varied, may not cover all domain-specific use cases
- Complexity of the pipeline may pose a learning curve for newcomers
Installation
native
python -m pip install -r requirements.txt
FAQ
How do I install the required Python dependencies for DeepSpec?
Run `python -m pip install -r requirements.txt` to install the core dependencies. Additional dependencies for data preparation (e.g., an inference engine to serve the target model) are described in `scripts/data/README.md`.
What are the main steps in the DeepSpec workflow and what does each step produce?
The workflow consists of three stages: 1) Data Preparation – download prompts, regenerate target answers, and build the target cache (which can be ~38 TB for the default Qwen/Qwen3-4B setting). 2) Training – train a draft model against the cached target outputs using `scripts/train/train.sh`. 3) Evaluation – measure speculative-decoding acceptance on benchmark datasets via `scripts/eval/eval.sh`. Each stage’s output feeds the next.
How do I train a draft model, and what configuration options are available?
Launch training with `bash scripts/train/train.sh`. The script calls `train.py` and spawns one worker per visible GPU. Choose a draft algorithm and target model by setting `config_path` to a file under `config/` (e.g., `config/dspark/dspark_qwen3_4b.py`). You can override `config_path`, `target_cache_dir`, or individual fields via `--opts`. Checkpoints are saved to `~/checkpoints/<project_name>/<exp_name>/step_*`. The default configs assume 8 GPUs; adjust `CUDA_VISIBLE_DEVICES` for fewer GPUs.
How do I evaluate a trained draft model using the provided evaluation script?
Run `bash scripts/eval/eval.sh`. You must set `target_name_or_path` to the target model the draft was trained against (e.g., `Qwen/Qwen3-4B`) and `draft_name_or_path` to the draft checkpoint (e.g., `~/checkpoints/deepspec/dspark_block7_qwen3_4b/step_latest`) or a Hugging Face repo ID from the Released Checkpoints table. The script evaluates on the datasets in `eval_datasets/` (gsm8k, math500, aime25, humaneval, mbpp, livecodebench, mt-bench, alpaca, arena-hard-v2).
Where can I find the released checkpoints mentioned in the paper, and how should I use them?
Released checkpoints are listed in the ‘Released Checkpoints’ section of the README, with Hugging Face links for each algorithm (Eagle3, DFlash, DSpark) and target model size (Qwen/Qwen3-4B, -8B, -14B, google/gemma-4-12B-it). To use them, set `draft_name_or_path` to the corresponding repo ID when running evaluation, and ensure `target_name_or_path` matches the model the checkpoint was trained against. For meaningful comparisons, align your setup with the training settings in this repo; otherwise consider fine‑tuning the draft on your own data.
Featured in Videos
YouTube tutorials and walkthroughs for DeepSpec
Alternatives
Similar projects ranked by category, topics, and text overlap.
