OpenPI Open Source Vision-Language-Action Model Library for Robotics

OpenPi offers three core VLA model families, optimized for robotic manipulation tasks:

• π₀: A flow-based vision-language-action model, designed for flexible robotic control.
• π₀-FAST: An autoregressive VLA built on the FAST action tokenizer, balancing speed and command-following capability.
• π₀.₅: An upgraded π₀ with superior open-world generalization (only flow matching head supported for training/inference in this repo).

Clone the Repository

Ensure submodules are included:

git clone --recurse-submodules git@github.com:Physical-Intelligence/openpi.git

# For existing clones:
git submodule update --init --recursive

Basic Inference (Pre-Trained Models)

from openpi.training import config as _config
from openpi.policies import policy_config
from openpi.shared import download

# Load config and checkpoint
config = _config.get_config("pi05_droid")
checkpoint_dir = download.maybe_download("gs://openpi-assets/checkpoints/pi05_droid")

# Initialize policy
policy = policy_config.create_trained_policy(config, checkpoint_dir)

# Run inference on a sample observation
example = {
    "observation/exterior_image_1_left": ...,  # Replace with real sensor data
    "observation/wrist_image_left": ...,
    "prompt": "pick up the fork"
}
action_chunk = policy.infer(example)["actions"]

• Test with the inference notebook.
• Platform-specific guides: DROID, ALOHA.
• Remote Inference: Run models on a remote GPU server (websocket streaming): Remote Inference Docs.
• No-Robot Testing: Test inference with random observations: Simple Client Script.

OpenPi now supports PyTorch for π₀ and π₀.₅ (π₀-FAST not supported yet; mixed precision/FSDP/LoRA/EMA training unavailable).

Setup

1. Update dependencies: uv sync
2. Verify transformers 4.53.2: uv pip show transformers
3. Apply patches (note: modifies uv cache):

cp -r ./src/openpi/models_pytorch/transformers_replace/* .venv/lib/python3.11/site-packages/transformers/

Undo with: uv cache clean transformers

Convert JAX → PyTorch

uv run examples/convert_jax_model_to_pytorch.py \
    --checkpoint_dir /path/to/jax/checkpoint \
    --config_name  \
    --output_path /path/to/pytorch/checkpoint

PyTorch Inference/Server/Fine-Tuning

• Inference: Same API as JAX (point to converted PyTorch checkpoint).
• Policy Server: Use converted PyTorch checkpoint path.
• Training:

# Single GPU
uv run scripts/train_pytorch.py  --exp_name 

# Multi-GPU (single node)
uv run torchrun --standalone --nnodes=1 --nproc_per_node= scripts/train_pytorch.py  --exp_name 

Precision Notes

• JAX: Inference (bfloat16/float32); Training (mixed precision, float32 weights/gradients).
• PyTorch: Inference (matches JAX); Training (bfloat16 default, float32 optional via pytorch_training_precision).