What is it?
What it is
A guide and setup instructions for running state-of-the-art large language models (LLMs) locally using high-end GPUs and custom hardware configurations.
Why it exists
To enable users to run powerful LLMs locally without relying on cloud services, reducing costs and latency while leveraging advanced hardware like PCIe Gen4 switches for optimized GPU communication.
Who should use it
Tech enthusiasts, researchers, developers with budget for high-end hardware, and users prioritizing local LLM deployment for privacy or cost reasons
Who should avoid it
Users without significant financial resources, non-technical individuals, or those preferring cloud-based solutions
How it works
A quick walkthrough in plain English
How local-llm works
Step 1 of 3
You interact with it
Open local-llm, send a request, or connect it to your stack.
Features
Advantages
- Cost-effective base system using last-gen DDR4 and eBay parts
- Reduced latency via PCIe4 switch for tensor parallelism
- Full control over model weights and data privacy
- Scalable architecture for future model upgrades
- Ready-to-run Docker setups simplify deployment
- High bandwidth (27.5/50.4 GB/s P2P) for efficient inference
- Open-source tooling for web browsing, search, and collaboration
Disadvantages
- High upfront cost for 4x RTX Pro 6000 GPUs (~$46k)
- Complex BIOS and hardware configuration required
- Dependency on specific PCIe4 switch hardware
- Power regulation challenges on a single 110V circuit
- Limited to users with technical expertise for setup
- Potential latency issues if ACS is not properly disabled
- Requires significant physical space for custom GPU mount
Installation
FAQ
What does a $2,000 local LLM setup look like and what models can I run?
A $2k rig typically uses two RTX 3090 GPUs (48 GB total VRAM). With that you can run Qwen 3.6‑27B and the latest Whisper‑large‑v3 for speech‑to‑text. The README includes a ready‑to‑run Docker compose in `./runners/stt` that assumes ~11 GB VRAM per GPU.
What hardware and cost are required for a $40,000 high‑performance LLM rig?
The recommended build uses four NVIDIA RTX PRO 6000 Blackwell GPUs (384 GB VRAM total). The base system is a last‑gen EPYC motherboard, DDR4 RAM, and a c‑payne Gen4 PCIe switch. Total cost is about $46,000 for GPUs plus ~$5,600 for the host, roughly $40k+ depending on pricing.
How do I configure the BIOS and system to enable GPU‑to‑GPU P2P over the c‑payne Gen4 switch?
Set the switch slot to x16 width and force Gen4 link speed. Disable ASPM, enable Re‑Size BAR, disable SR‑IOV, and set Preferred IO to Auto (or manual bus 81). In the BIOS, also reduce the redriver gain to level 3. These settings prevent link downgrades and allow full 96 GB VRAM BAR exposure.
What is the purpose of disabling PCIe ACS and how can I do it?
With ACS enabled, P2P traffic is routed through the CPU root port instead of staying inside the switch, negating the switch’s benefit. The README provides a `disable-acs.sh` script that writes zero to the ECAP_ACS register for each device, and a systemd oneshot service to run it on boot. Verify with `lspci -vvv | grep ACSCtl` and `nvidia-smi topo -m` showing PIX links.
How can I limit GPU power draw to fit a single 110 V circuit?
Enable persistence mode and set a power cap per GPU. Run `sudo nvidia-smi -pm 1` and `sudo nvidia-smi -pl 350` (or 260 W during a 1700 W PSU phase). Check limits with `nvidia-smi --query-gpu=index,power.limit,power.draw --format=csv`.
Where can I find ready‑to‑run Docker configurations for models and STT?
All Docker compose files live in the `./runners` directory. For example, `./runners/GLM-5.2-594B` runs the GLM‑5.2 model, and `./runners/stt` contains the Whisper‑large‑v3 STT setup. Each container mounts `~/storage/models` read‑only to access locally cached weights.
Featured in Videos
YouTube tutorials and walkthroughs for local-llm
t5
70,223,459 views
Starts at 00:00
Alternatives
Similar projects ranked by category, topics, and text overlap.
