lora-speedrun
Speedrunning LoRA fine-tuning: frozen task, frozen hardware, public wall-clock leaderboard. modded-nanogpt for fine-tuning.
What is it?
What it is
LoRA Speedrun is a public benchmark and leaderboard for measuring wall-clock time to fine-tune LoRA adapters on fixed tasks (GSM8K with Qwen2.5-1.5B and SQuAD v1.1 with SmolLM2-1.7B) using identical L40S hardware on Modal, with independent 3‑seed verification.
Why it exists
It exists to make LoRA speedup claims falsifiable by providing a frozen task, referee‑verified runs, and a public leaderboard so techniques can be compared on equal footing and generalizable across model families.
Who should use it
machine learning researchersAI engineersLLM fine-tuning enthusiastsModal usersspeedrun competitors
Who should avoid it
non-technical usersthose without GPU accesspeople avoiding competitive benchmarksnon-LoRA fine-tuning workflows
How it works
A quick walkthrough in plain English
How lora-speedrun works
Step 1 of 3
You interact with it
Open lora-speedrun, send a request, or connect it to your stack.
Features
Advantages
- Eliminates unfalsifiable performance claims
- Provides empirical evidence for training optimization techniques
- Low barrier to entry using free compute credits
- Ensures generalizability through diverse task tracks
- Transparent and auditable verification process
Disadvantages
- Strict constraints may limit experimental freedom
- Requires specific hardware (L40S) for official verification
- Risk of overfitting to specific model/task combinations
Installation
cloud
Make a Modal account, then: git clone https://github.com/Saivineeth147/lora-speedrun && cd lora-speedrun pip install modal pyyaml && modal setup python harness/modal_verify.py --prefetch python harness/modal_verify.py --submission submissions/000-baseline --runs 1
native
git clone https://github.com/Saivineeth147/lora-speedrun && cd lora-speedrun pip install modal pyyaml && modal setup python harness/modal_verify.py --prefetch # Official-hardware run: python harness/modal_verify.py --submission submissions/000-baseline --runs 1 # Local iteration: bash scripts/setup_gpu.sh python harness/run_submission.py submissions/000-baseline --runs 1
FAQ
How do I run an official verification on Modal?
Clone the repo, install dependencies, and run the harness: 1. `git clone https://github.com/Saivineeth147/lora-speedrun && cd lora-speedrun` 2. `pip install modal pyyaml && modal setup` (one‑time browser auth). 3. Cache the model and data: `python harness/modal_verify.py --prefetch`. 4. Verify a submission: `python harness/modal_verify.py --submission submissions/000-baseline --runs 3`. The script will re‑run the training three times with fresh seeds on a single L40S in a network‑blocked sandbox and produce a verification report.
What GPU is required for local iteration?
Any GPU with at least 24 GB of memory will run the baseline for quick experimentation. Use the provided script: `bash scripts/setup_gpu.sh` to install dependencies, then `python harness/run_submission.py submissions/000-baseline --runs 1`. Local times are not official; the leaderboard clock is the Modal L40S.
What are the specifications of each track?
Track 1: Base model `Qwen/Qwen2.5-1.5B`, task GSM8K math, target ≥ 57.0 % EM, training data GSM8K train split only, 1 × L40S, ≤ 30 M trainable params. Track 2: Base model `HuggingFaceTB/SmolLM2-1.7B`, task SQuAD v1.1 QA, target ≥ 75.5 % EM, training data SQuAD train split only, same hardware and param constraints. Full specs are in `spec.yaml` and `spec-t2.yaml`.
How are records verified and what does the verification process involve?
When a PR is submitted, CI first checks the code statically and a Claude security review screens the diff. A maintainer then comments `/verify`, triggering the harness to: 1. Run the training three times with fresh seeds on a modal L40S sandbox. 2. Audit the adapter parameter count and verify model/data hashes. 3. Evaluate on the test set. All three runs must meet the target; the mean time becomes the official record. The verification report is committed to `records/verifications/`.
Why is LoRA (Low‑Rank Adaptation) chosen for this competition?
LoRA is the standard, inexpensive way to fine‑tune a frozen language model by training a small adapter. It keeps the base model frozen, limits trainable parameters to ≤ 30 M, and allows fair comparison of training speed across participants. The competition focuses on wall‑clock time, so using a lightweight, well‑understood method like LoRA ensures that any speed improvements come from algorithmic or engineering tricks rather than model changes.
Featured in Videos
YouTube tutorials and walkthroughs for lora-speedrun
Alternatives
Similar projects ranked by category, topics, and text overlap.
