shard
Pipeline-parallel LLM inference across GPUs on separate machines.
What is it?
What it is
A permissionless compute network engine that pools GPUs across the internet to run large models by splitting them into shards, enabling distributed inference over WAN with verified receipts.
Why it exists
To overcome latency and transport limitations of decentralized compute by enabling speculative decoding, pipelining, and CUDA-graphed drafts for high-throughput model serving without centralized infrastructure.
Who should use it
Researchers and engineers interested in decentralized AI inference, distributed GPU computing, and experimental sharded model serving; contributors to permissionless compute networks.
Who should avoid it
Beginners without CUDA/distributed systems experience, users seeking simple out-of-the-box inference APIs, or those needing production-ready, centrally managed solutions.
How it works
A quick walkthrough in plain English
How shard works
Step 1 of 3
You interact with it
Open shard, send a request, or connect it to your stack.
Features
Advantages
- Scales to frontier‑size models without a datacenter
- No single node holds the entire model, enhancing privacy
- High throughput over WAN (~30‑40 tokens/s) via speculative decoding
- Open source and modular architecture for any model
- Built‑in fault tolerance and verifiable receipts
- Lossless temperature/top‑p/top‑k sampling
- Decentralized, no central inference server
- Supports consumer GPUs, lowering entry barrier
- Authenticated transport protects against passive eavesdropping
Disadvantages
- High complexity of deployment and maintenance
- Latency over WAN still a limiting factor for some use cases
- Requires reliable network connectivity and NAT hole‑punching
- Intermediate activations can leak partial user data
- Security risk if malicious nodes intercept activations
- Performance overhead from transport and verification
- Limited to models that can be split into contiguous blocks
- Potential regulatory and legal concerns for unfiltered inference
Installation
FAQ
What is Shard and how does it differ from traditional compute networks?
Shard is a permissionless compute network that pools GPUs globally to run large models across distributed hardware, unlike centralized systems. It shares VRAM and compute instead of disk, enabling models larger than any single GPU to run over the open internet.
How does Shard split and distribute large models across GPUs?
Shard divides a model into contiguous layer blocks, assigning one shard per GPU. Activations are streamed through these shards in order, with no single node holding the entire model, enabling inference of models exceeding individual GPU capacities.
What technical innovations enable Shard's high throughput (e.g., 30 tok/s)?
Key innovations include speculative decoding with a CUDA-graphed draft model, async pipelining to overlap verification traversals, and a direct-return ring architecture that reduces WAN latency. These steps collectively shift the bottleneck from network latency to compute efficiency.
How does Shard ensure security and integrity of data transmission?
Shard uses authenticated encryption (ChaCha20-Poly1305) with a shared key for sealed framing, preventing passive observation or tampering. Verifiable receipts track GPU UUIDs, IPs, regions, and output hashes to ensure reproducibility and detect malicious nodes.
Can anyone join the Shard network, and how is resource allocation managed?
Yes, anyone can join by running a single command. The network dynamically allocates layer blocks to available GPUs, enabling decentralized participation. Fault tolerance is supported via mid-request healing, allowing requests to resume on spare nodes if a node fails.
What are the current limitations or challenges Shard aims to address next?
Phase 1 focuses on improving WAN reliability (e.g., NAT hole-punching, relay fallbacks) and activation quantization. Phase 2 expands speculative decoding to support lossless sampling. Long-term goals include enabling general compute and training workloads on the same infrastructure.
Featured in Videos
YouTube tutorials and walkthroughs for shard
Alternatives
Similar projects ranked by category, topics, and text overlap.