tau-0-vla
This repo is the official implementation of "τ0-VLA: a Hierarchical Robot Foundation Model with World-Model-Guided Test-Time Computation".
What is it?
What it is
τ₀-VLA is a hierarchical robot foundation model that combines a memory‑augmented high‑level policy with a world‑model‑guided test‑time computation strategy, and a generalist low‑level policy based on a Qwen3.5 vision‑language backbone and a flow‑matching action expert.
Why it exists
It was created to enable long‑horizon manipulation tasks by generating subtasks with a high‑level policy and executing them with a versatile low‑level policy, while allowing additional reasoning at test time through a world model.
Who should use it
Robotics researchersAI engineers working on foundation modelsDevelopers building long-horizon manipulation systemsResearchers studying hierarchical robot policiesMachine learning engineers specializing in vision-language-action (VLA) models
Who should avoid it
Beginners without experience in PyTorch or CUDAUsers looking for plug-and-play end-effector (EEF) serving (not supported in v1)Developers without access to high-compute GPU resources
How it works
A quick walkthrough in plain English
How tau-0-vla works
Step 1 of 3
You interact with it
Open tau-0-vla, send a request, or connect it to your stack.
Features
Advantages
- Enables complex, multi-step robotic tasks through hierarchical planning
- Adapts to diverse robot hardware via modular adapter architecture
- Leverages large-scale real-world data for robust generalization
- Efficient test-time reasoning reduces computational overhead
- Strong language model integration for multimodal understanding
- Apache-2.0 license facilitates open research and deployment
Disadvantages
- Requires specific hardware (CUDA 12.8, PyTorch 2.7.1) for deployment
- Training demands extensive data (40k+ hours) for optimal performance
- Limited EEF serving support in current release
- Example data restricted to AgiBot World subset
- Complex setup and configuration process
- Potential computational costs for test-time search
Installation
native
git clone git@github.com:sii-research/tau-0-vla.git cd tau-0-vla bash scripts/setup.sh
FAQ
What is τ₀-VLA and what problem does it solve?
τ₀-VLA is a hierarchical robot foundation model that combines a memory‑augmented high‑level policy for subtask planning with a generalist low‑level policy (Qwen3.5 vision‑language backbone + Mixture‑of‑Transformers action expert) to execute long‑horizon manipulation across robot embodiments, using world‑model‑guided test‑time computation when extra reasoning is needed.
What are the installation steps and required environment?
Clone the repo, navigate into it, and run the setup script: git clone git@github.com:sii-research/tau-0-vla.git; cd tau-0-vla; bash scripts/setup.sh. The reference environment uses Python 3.11, CUDA 12.8, and PyTorch 2.7.1.
How can I train τ₀‑VLA on my own dataset?
Start from the template configs in configs/_template/ and the corresponding adapter in src/tau0_vla/adapters/_template/. Prepare your data in LeRobot v3.0 format, then launch training with bash scripts/train.sh <your_config>/train.yaml --model_name_or_path /path/to/base_checkpoint.
How do I serve a trained joint‑control checkpoint for inference?
After training, run the policy server: python -m deploy.server --model outputs/<run_name>. This serves joint‑control checkpoints; native EEF serving is not supported in this release.
What is included in the example_data/ directory and how is it licensed?
example_data/ contains a small AgiBot World subset in LeRobot v3.0 format, used for quick experimentation. Its license is described in example_data/README.md, while the code and model weights are under Apache‑2.0.
Where can I find details about the dataset format, data pipeline, and robot adapters?
See src/tau0_vla/data/DATASET_FORMAT.md for the dataset specification, src/tau0_vla/data/README.md for the data pipeline, and src/tau0_vla/adapters/README.md for information on embodiment‑specific adapters and deployment I/O.
Featured in Videos
YouTube tutorials and walkthroughs for tau-0-vla
Alternatives
Similar projects ranked by category, topics, and text overlap.
