apex-inference-chip
An inference chip design that runs a real LLM (Qwen2.5-0.5B) on FPGA — one transformer decoder layer in RTL, every silicon value bit-exact against a golden model. 0.56 tok/s measured, a 140× climb, full evidence trail.
What is it?
What it is
A fully open, verification-first LLM-inference tile (APEX) — a tinyNPU that performs attention computation and KV-cache compression in the same datapath. Implemented in RTL on FPGA hardware (Lattice ECP5-85F, AWS F2), it places the KV codec directly in the datapath with adaptive precision tiers, enabling constant-reading-speed inference even as context grows.
Why it exists
Edge large-model inference suffers from two dominant bottlenecks: moving weights (bandwidth) and remembering context (KV cache growth). Traditional accelerators treat KV cache as a software problem (quantize on CPU/GPU), but APEX addresses both by integrating KV compression into the hardware datapath and enforcing bit-exact verification against an executable golden model for every RTL block.
Who should use it
Researchers and engineers working on AI hardware acceleration, LLM inference optimization, or FPGA-based systems. Ideal for those needing verified, hardware-efficient attention mechanisms with adaptive KV-cache compression.
Who should avoid it
Users seeking plug-and-play solutions or lacking expertise in hardware design, FPGA implementation, or low-level verification. Not suitable for high-level ML practitioners without technical depth.
How it works
A quick walkthrough in plain English
How apex-inference-chip works
Step 1 of 3
You interact with it
Open apex-inference-chip, send a request, or connect it to your stack.
Features
Advantages
- Architectural focus on memory efficiency: KV-cache compression reduces context memory traffic
- Bit-exact verification ensures correctness from simulation to silicon deployment
- Open-source design enables transparency, reproducibility, and community collaboration
- Lower energy per token compared to desktop GPUs for edge deployment scenarios
- Flat reading speed as context grows due to in-datapath KV compression
- Modular tile design allows for scalable integration into larger chip architectures
- Comprehensive documentation with measured vs. projected performance tracking
- End-to-end pipeline verified with real Qwen models through golden reference
- FPGA proof-of-concept demonstrates practical hardware implementation
- Anti-fabrication rule ensures all published numbers are traceable to test logs
Disadvantages
- Limited to single-tile scope: no integrated DRAM controller, PCIe, or NoC components
- Current FPGA implementation optimized for Qwen2.5-0.5B; 7B models only verified in software
- Projected performance metrics depend on unbuilt components (native-W4 path, wide LPDDR)
- Throughput may be lower than GPUs for single-stream inference due to hardware constraints
- Development requires specialized FPGA toolchains and verification expertise
- Weight streaming infrastructure adds complexity to memory subsystem design
- Adaptive precision requires additional logic for importance tracking and tier decisions
- Open-source nature may limit commercial adoption without additional IP integration
- Performance projections assume ideal calibration of analytic models
- Synthesis-level defects can occur despite verification, requiring differential debugging
Installation
native
┌──────────────────────────── apex_top ─────────────────────────────┐
│ │
x ──► seam ─┼─► RMSNorm ──► MXE: W_Q·x W_K·x W_V·x ──► RoPE ──► KVQ compress │
│ (ASU) (systolic GEMM) (Q,K) │ │
│ ▼ │
│ KV cache — INT4 + outlier │
│ lane, on-tile SRAM │
│ │ │
│ ┌──────────────────────────────────────────┤ │
│ ▼ FAQ
What is APEX and what problem does it solve?
APEX is a fully open, verification-first LLM-inference tile that implements one transformer decoder layer in real RTL. It solves the two dominant costs of edge LLM inference: moving weights (bandwidth) and remembering context (KV cache growth). Unlike most accelerators that treat KV cache as a software problem, APEX puts the KV codec inside the datapath so keys and values are compressed in-flight and decompressed on read, with an importance unit that allocates bits where they matter most.
How does the KV-compression engine work and what makes it different?
The KVQ engine uses per-channel INT4 keys, per-token INT4 values, and an fp16 outlier lane for channels that refuse to quantize. It implements tiered precision (KVQ8/KVQ4/KVQ4+) driven by the TIP unit's token-importance tracking. Compression happens between RoPE and cache write; decompression happens inside the attention read path — no fp16 copy exists anywhere. Every overhead (tags, outlier lanes, padding, scale bank) is counted in the compression ratio.
What verification methodology ensures correctness?
Every RTL block is bit-exact verified against an executable NumPy golden model — not 'close', but bit-identical. The flow includes mutation-tested testbenches (mutants must be caught), SVA assertions, coverage, and an anti-fabrication rule where published numbers are machine-extracted from suite logs. Sim-to-silicon differential discipline compares FPGA captures bit-for-bit against Verilator simulation, which caught a synthesis-level hardware defect during bring-up.
Which real models have been run through the pipeline and on what hardware?
Qwen2.5-0.5B runs on FPGA hardware (Lattice ECP5-85F and AWS F2 VU47P). Qwen2.5-7B tokens have run through the software-verified golden pipeline (not silicon). Accuracy is measured on the full 10,042-document HellaSwag validation set for 0.5B/1.5B/7B models with paired per-document statistics.
How can I reproduce the verification and FPGA results?
Requires Verilator 5.x and Python 3.11+NumPy. Run: `make -C golden test` for golden models, `make -C verif/top/smoke smoke` for end-to-end attention tile, `make -C verif/top l3` for full layer vs golden, `make -C verif/seq_walker` for walker suite with mutation gates. For FPGA: `bash scripts/fpga/f2/run_walked_demo.sh agfi-030a812cd224b409d` runs the 193-check battery on AWS F2; `bash scripts/fpga/f2/run_chat_demo.sh` starts an interactive prompt. Model weights derive from `mlx-community/Qwen2.5-0.5B-Instruct-4bit`.
What performance numbers are measured vs projected?
Measured: cycle counts, compression ratios, check totals in STATUS.md and per-campaign results; 0.56 tok/s on A2 (15.625 MHz) and 0.25 tok/s on reference image, with 140× climb from host-driven baseline. Projected (not measured): 7B at reading speed on ~3W with 32k–64k context held flat by KV compression; 5–10× less energy/token than desktop GPU at same single-stream job. Projections come from analytic model with asserted calibration anchors in docs/results/perf_model/PERF_MODEL.md.
Featured in Videos
YouTube tutorials and walkthroughs for apex-inference-chip
Alternatives
Similar projects ranked by category, topics, and text overlap.