Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 56 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -476,6 +476,62 @@ thresholds:
max_missed_control_budget: 0
```

## Local Studio dataset admission (SmolVLA)

A local dataset must be a completed Studio corrected export created with the
`verified-moments-v1` statistics policy and `smolvla` normalization profile.
Keep the entire export directory intact, including its sibling receipt and
provenance. `--dataset-root` points to `EXPORT/dataset` itself.

```bash
tether finetune \
--base /path/to/your/smolvla-checkpoint \
--dataset studio-local/my-corrected-dataset \
--dataset-root /path/to/EXPORT/dataset \
--dataset-manifest-sha256 <export-receipt-sha256> \
--output ./local-admission-check \
--dry-run
```

The digest is `export-receipt.json`'s canonical `sha256` field (SHA-256 of the
sorted, compact JSON receipt excluding that field), not the hash of the pretty
printed file or `artifact_manifest_sha256`. Tether verifies that digest, every
inventoried file's size/hash, the complete local inventory, retained source and
split bindings, LeRobot 0.5.1 writer/source pins, and feature/statistics metadata.
Only development rows exported to the `train` split are admitted. Missing files,
symlinks, incomplete markers, unsupported profiles, and integrity failures block
before training. Tether admission, preflight and command verification read only
local dataset metadata and never substitute Hub metadata. The downstream
LeRobot trainer has its own download fallback if files later disappear; this
lane requires the verified export to remain complete and immutable.

Preflight reads the actual base config and retains SmolVLA's declared state and
action padding limits. A local base config makes this check offline; remote
bases may require fetching their config. ACT and other policies, local resume,
`--skip-preflight`, dataset revisions and conflicting Python extra arguments are
unsupported for this lane. Output must be outside the export directory.

`--dataset` remains the LeRobot repository label for local input. No dataset
publication is performed. The existing Hub dataset workflow remains available
when the two local options are omitted. Python callers can use
`FinetuneConfig(dataset_root=..., dataset_manifest_sha256=..., ...)` and the
stdlib-only shared `tether.finetune.local_dataset.verify_local_export` function;
refusals raise `LocalDatasetError` with a stable `code`.

The export processor check uses dataset statistics and records no ImageNet
substitution. Training keeps LeRobot's existing `use_imagenet_stats=True`
default (or an explicit `dataset.use_imagenet_stats` Python extra argument).
Preflight records both facts. SmolVLA uses `VISUAL=IDENTITY`, so these image
statistics are not consumed by that normalization mode. Tether never silently
changes the training setting.

Admission checks retained evidence and local bytes; it does not rerun the
Parquet loader, instantiate a policy, or demonstrate training quality. Keep the
export immutable during preflight and training. The command builder verifies
again before emitting `dataset.root`, but path rechecks do not eliminate
filesystem changes between verification and use. Studio's job-owned copy is a
separate snapshot boundary.

## Evidence knobs, not extra products

Advanced teams can enable more runtime evidence with `tether serve` flags:
Expand Down
1 change: 1 addition & 0 deletions src/tether/eval/local_runner.py
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,7 @@ def run_local_libero(
save_video_dir=config.output_dir + "/videos" if config.video else "",
label=checkpoint.identity,
use_native=True,
verification_device="cuda",
evidence_dir=config.output_dir + "/evidence" if config.capture_evidence else "",
evidence_limits=CaptureLimits(
max_bytes=config.evidence_max_bytes,
Expand Down
20 changes: 15 additions & 5 deletions src/tether/finetune/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,14 @@ def finetune_command(
"", "--dataset-revision",
help="Exact Hugging Face revision for the training dataset.",
),
dataset_root: str = typer.Option(
"", "--dataset-root",
help="Actual EXPORT/dataset directory from a qualified Studio SmolVLA export.",
),
dataset_manifest_sha256: str = typer.Option(
"", "--dataset-manifest-sha256",
help="Canonical sibling export-receipt.json SHA-256 (excludes its sha256 field).",
),
output: str = typer.Option(
...,
"--output",
Expand Down Expand Up @@ -103,8 +111,8 @@ def finetune_command(
skip_preflight: bool = typer.Option(
False,
"--skip-preflight",
help="Skip preflight validation. Escape hatch for local-dataset "
"or gated-repo flows where preflight can't resolve schema. "
help="Skip preflight validation for remote/gated dataset flows. "
"Never permitted with --dataset-root. "
"Only set if you know what you're doing.",
),
resume: bool = typer.Option(
Expand Down Expand Up @@ -140,6 +148,8 @@ def finetune_command(
base_revision=base_revision or None,
dataset=dataset,
dataset_revision=dataset_revision or None,
dataset_root=Path(dataset_root) if dataset_root else None,
dataset_manifest_sha256=dataset_manifest_sha256 or None,
output=Path(output),
num_steps=num_steps,
batch_size=batch_size,
Expand All @@ -157,18 +167,18 @@ def finetune_command(
resume=resume,
)

console.print(f"[bold]tether finetune[/bold] — v0.3 MVP (SmolVLA LoRA)")
console.print("[bold]tether finetune[/bold] — v0.3 MVP (SmolVLA LoRA)")
console.print(f" base: {cfg.base}")
console.print(f" dataset: {cfg.dataset}")
console.print(f" output: {cfg.output}")
console.print(f" steps: {cfg.num_steps} batch={cfg.batch_size} "
f"lr={cfg.learning_rate} lora_r={cfg.lora_rank}")
console.print(f" backend: lerobot (openpi-JAX + hf_transformers in v0.5+)")
console.print(" backend: lerobot (openpi-JAX + hf_transformers in v0.5+)")
console.print()

result = run_finetune(cfg)

console.print(f"\n[bold]Result[/bold]")
console.print("\n[bold]Result[/bold]")
console.print(f" status: {result.status}")
if result.error:
console.print(f" [red]error:[/red] {result.error}")
Expand Down
14 changes: 11 additions & 3 deletions src/tether/finetune/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ class FinetuneConfig:
"""HF model id of the base checkpoint (e.g. lerobot/smolvla_base)."""

dataset: str
"""HF dataset id to fine-tune on (e.g. lerobot/libero)."""
"""HF dataset id, or an explicit LeRobot repository label for dataset_root."""

output: Path
"""Output directory. Will contain model.onnx + VERIFICATION.md after
Expand Down Expand Up @@ -92,8 +92,8 @@ class FinetuneConfig:
Useful for validating config before committing to a multi-hour run."""

skip_preflight: bool = False
"""If True, skip preflight validation. Escape hatch for local-dataset
or gated-repo flows where preflight can't resolve the schema.
"""If True, skip preflight validation. Escape hatch for remote gated-repo flows where preflight cannot resolve
the schema. Never permitted with dataset_root.
Only set if you know what you're doing."""

phase: str = "train"
Expand Down Expand Up @@ -178,8 +178,16 @@ class FinetuneConfig:
contract yet and fail before starting work.
"""

dataset_root: Path | None = None
"""Actual EXPORT/dataset directory of a qualified local Studio export."""

dataset_manifest_sha256: str | None = None
"""Canonical sibling export-receipt.json digest, excluding its sha256 field."""

def __post_init__(self) -> None:
self.output = Path(self.output)
if self.dataset_root is not None:
self.dataset_root = Path(self.dataset_root)
if not (0.0 <= self.mix_ratio <= 1.0):
raise ValueError(
f"mix_ratio must be in [0, 1], got {self.mix_ratio}"
Expand Down
Loading
Loading