spec-ptc
Speculative programmatic tool calling (sPTC) for harnesses like RLM, CodeAct, etc.
What is it?
What it is
Speculative programmatic tool-calling (sPTC) is a technique and library that speculates and queues up tool calls in partially-generated code while an LLM is streaming tokens to generate a REPL call. The repository provides a Speculator object with a `spec.tool` decorator for marking speculatable tools, a shadow REPL for speculation, and a daemon (spec-ptc-daemon) for out-of-process execution via JSON-lines messages over a socket.
Why it exists
Harnesses like Recursive Language Models (RLMs) and CodeAct rely on programmatic tool-calling where tools are embedded as functions in a code REPL. Sub-LLM and sub-RLM calls in these systems are expensive and often block code execution, consuming the majority of runtime. sPTC allows the RLM to batch and asynchronously compute expensive calls while the full codeblock is still being generated, overlapping tool execution with code generation logic to reduce latency.
Who should use it
Developers and researchers building LLM harnesses that use programmatic tool calling, especially those implementing recursive language models (RLMs) or similar designs where sub‑LLM calls are expensive and can benefit from speculative overlapping. Users familiar with REPL‑based tool integration and wanting to reduce latency by speculating tool calls while the root LLM streams code.
Who should avoid it
Beginners new to LLM tool calling or users who only need simple synchronous tool invocation without speculative execution. Those not using a REPL harness or who don't have expensive sub‑LLM calls to overlap.
Stars over time
No change stars in the last 1 day
How it works
A quick walkthrough in plain English
How spec-ptc works
Step 1 of 3
You interact with it
Open spec-ptc, send a request, or connect it to your stack.
Features
Advantages
- Overlaps expensive tool calls (e.g., sub-LLM calls) with code generation, reducing idle time
- Batches and asynchronously computes calls while the codeblock is still being generated
- General technique applicable to various harness designs (RLMs, CodeAct, etc.)
- Provides both in-process and out-of-process (daemon) options for flexibility
- Simple integration with existing harnesses (e.g., one line for RLM)
Disadvantages
- Requires modifications to existing harness code (adding speculator, feeding tokens, etc.)
- Adds complexity with shadow REPL and speculation logic
- Not all tools can be speculated (side effects, impure functions must be excluded)
- Daemon mode requires running a separate process and socket communication
- Experimental nature (relatively low star count, may have edge cases)
Installation
native
clone this repository (uses uv), or install with pip install spec-ptc
FAQ
What is Speculative Programmatic Tool Calling (sPTC) and how does it improve performance?
sPTC is a technique for harnesses using tools like sub-agents or sub-calls in code. While the LLM streams tokens to generate a codeblock, sPTC speculates and queues up tool calls in the partially-generated code as Futures. This allows expensive tool calls (like sub-LLM queries) to run asynchronously and overlap with the code generation process, rather than blocking serially after generation completes.
How do I install the spec-ptc library?
You can install the library directly via pip using the command `pip install spec-ptc`. Alternatively, you can clone the repository, which uses `uv` for dependency management.
How do I register tools as speculatable versus non-speculatable?
You use the `@spec.tool` decorator. To mark a tool for speculation (e.g., pure functions like sub-LLM queries), use `@spec.tool(speculatable=True, pure=True)`. For tools with side effects that should never be speculated (like sending an email), simply use `@spec.tool()` without the speculatable flag, ensuring they only run when actually executed.
How does the token streaming and execution flow work in the core loop?
Inside a `spec.turn(repl_locals=ns)` context manager, you feed streamed tokens using `t.feed(delta)` as they arrive. The speculator parses closed statements and launches the tool calls immediately in the background (as Futures). Once generation is complete, you execute the full code block with `exec(code, ns)`, which claims the already-computed results and runs any remaining non-speculated code instantly.
How can I integrate sPTC with an RLM (Recursive Language Model) harness?
If you are using the RLM library, integration is extremely simple: you just need to import and run a single patch function: `from demo.rlm import patch_rlm; patch_rlm()`. This patches the RLM to automatically use sPTC for its sub-LLM calls.
What is the `spec-ptc-daemon` and how does the client communicate with it?
The `spec-ptc-daemon` is a simple out-of-process daemon that runs the shadow REPL and store, allowing arbitrary harnesses to use sPTC without deep integration. It communicates via a Unix socket (default `/tmp/spec-ptc.sock`) using four JSON-lines messages: `turn_begin` (to snapshot variables), `feed` (to stream tokens), `resolve` (to check for results), and `turn_end` (to clean up). You can use the lightweight stdlib client `plugins.client.SpecClient` to interface with it.
Featured in Videos
YouTube tutorials and walkthroughs for spec-ptc
Alternatives
Similar projects ranked by category, topics, and text overlap.
