inference-engine

ds4

C

95

DeepSeek 4 Flash and PRO local inference engine for Metal, CUDA and ROCm

tokenspeed

Python

90

TokenSpeed is a speed-of-light LLM inference engine.

open-webui

Python

100

User-friendly AI Interface (Supports Ollama, OpenAI API, ...)

MTPLX

Python

90

3x faster speeds on MLX | Qwen 3.8 27B | Native MTP Speculative Decoding On Apple Silicon With No External Drafter.

shard

Python

72

Pipeline-parallel LLM inference across GPUs on separate machines.

inference-school

Swift

77

A hands-on Swift and Metal course for building LLM inference from first principles on Apple silicon, with 48 guided lessons, runnable exercises, a native macOS Studio, and a complete companion book.

apex-inference-chip

Python

68

An inference chip design that runs a real LLM (Qwen2.5-0.5B) on FPGA — one transformer decoder layer in RTL, every silicon value bit-exact against a golden model. 0.56 tok/s measured, a 140× climb, full evidence trail.

llama.cpp

C++

95

LLM inference in C/C++

bindwidth

JavaScript

72

Evidence-aware on-prem LLM inference sizing and TCO calculator

cliare

Rust

72

CLI agent-readiness measurement, command-shape inference, and CI scorecards

transformers

Python

95

🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.

Audar-ASR-V1

Python

68

Arabic-first generative speech recognition — Audar-ASR-V1 (Flash + Turbo). #1 on the Open Universal Arabic ASR Leaderboard. Model cards, benchmarks & inference.

ollama

Go

95

Get up and running with Kimi-K2.6, GLM-5.2, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models.

local-llm

Shell

70

Everything I know about running LLMs locally

kimodo.cpp

C++

72

Animate skeletons with natural language; NVIDIA's Kimodo ported to C++/GGML

hayamimi

Python

72

早耳 - Real-time multilingual speech-to-text on CPU only. Live subtitles, browser dashboard, speaker labels, translation. No GPU, no cloud.