inference

inference-school

Swift

82

A hands-on Swift and Metal course for building LLM inference from first principles on Apple silicon, with 48 guided lessons, runnable exercises, a native macOS Studio, and a complete companion book.

llama.cpp

C++

90

LLM inference in C/C++

shard

Python

77

Pipeline-parallel LLM inference across GPUs on separate machines.

ds4

C

90

DeepSeek 4 Flash and PRO local inference engine for Metal, CUDA and ROCm

tokenspeed

Python

85

TokenSpeed is a speed-of-light LLM inference engine.

bindwidth

JavaScript

77

Evidence-aware on-prem LLM inference sizing and TCO calculator

cliare

Rust

82

CLI agent-readiness measurement, command-shape inference, and CI scorecards

transformers

Python

90

🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.

Audar-ASR-V1

Python

73

Arabic-first generative speech recognition — Audar-ASR-V1 (Flash + Turbo). #1 on the Open Universal Arabic ASR Leaderboard. Model cards, benchmarks & inference.

ollama

Go

90

Get up and running with Kimi-K2.6, GLM-5.2, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models.

local-llm

Shell

80

Everything I know about running LLMs locally

open-webui

Python

95

User-friendly AI Interface (Supports Ollama, OpenAI API, ...)

MTPLX

Python

82

3x decode TPS increase On Qwen 3.6 27B @ temp 0.6 | Native MTP Speculative Decoding On Apple Silicon With No External Drafter.