[jacoblambert]

Summary

This PR adds t4migrate. It turns a 0.x dataset into a new 1.0 dataset. The 0.x dataset is only read.

t4migrate <0.x DATASET_ID>/<VERSION> <OUTPUT_DIR> --dataset-version <N>

It writes <OUTPUT_DIR>/<DATASET_ID>/<N>, prints the migration notes, and runs t4sanity on the result. It exits with 1 if a sanity rule fails.

The version comes from the caller, because Web.Auto assigns it on upload. Migration will run in Web.Auto.

V0Migrator does the same in Python, channel by channel or all at once with migrate().

What it does

Tables are loaded with load_dataset(), as 1.0 records. Then each channel is rewritten, and its sensor and sample_data records are updated.

LiDAR: one Parquet file per channel, one row group per frame.

• Each field gets its 1.0 type (for example intensity uint8) when every value fits.
• If a value does not fit, the field keeps its 0.x type. The migration warns and writes a note. Other vehicles may store intensity differently, so we do not guess.
• A ring (channel) that is -1 on every point was not recorded. It is dropped, with a note.
• .info files become source_index and source_timestamp. source_index is the source LiDAR's sensor.index.
• lidarseg becomes label and instance. 0.x labels are semantic only, so instance is 0. Frames without labels are null. Category 0 must be unpainted, or the migration stops.
• Source LiDARs (merged into another channel) keep their own sensor row and get the merged file's format and fields.

Cameras: one AV1 video per channel, frames in timestamp order. --keep-images copies the images instead.

• Frames with is_valid: false are 0.x placeholders for dropped frames. They are encoded as they are, so the records stay 1:1 with 0.x. We can drop them in the converter for new data.

Every channel: next and prev are set from timestamp order. Some 0.x placeholder frames are only half linked: they point to their neighbors, but the neighbors skip them. Each fix is noted.

map/ and input_bag/ are copied as they are.

manifest.json has migrated_from_schema_version: "0.x" and the notes. 0.x has no provenance, so it stays empty for backfilling. The recording start and end are the first and last sample times, with a note.

Tests

• LiDAR: the 0.x sample migrates with the same values. Its fractional intensity stays float32, with warnings.
• LiDAR: a modified sample with integer intensity, ring -1 and lidarseg. Intensity becomes uint8, the ring is dropped, and labels appear only on the labeled frame. Without an unpainted category, it is refused.
• Camera: each video frame matches its own source image, in timestamp order.
• Copied images are byte-identical.
• A half-linked frame is linked back, with a note.
• CLI: the 0.x sample migrates and passes sanity.

Checked on real data

LiDAR on 4 datasets (4, 5, 7 and 9 columns): every frame matches the 0.x values.

Full migration of 7aa8888b (lidarseg, 8 merged LiDARs, 6 cameras, input bag):

	0.x	1.0
Sensor data	8.7 GB	2.6 GB
input_bag/, map/	19 GB, 0.77 GB	copied
Time, memory		6 min, 4 GB

Sanity: 101 passed, 0 failed. The 2 warnings are the placeholder vehicle and map IDs (to backfill) and hashes not re-checked. 6 rules are skipped because this dataset has no boxes. The first run found the half-linked camera frame (REF107), which is why relinking is in this PR.

One camera frame (CAM_FRONT, 2880x1860): video is 0.18 of the JPG size, 37–38 dB against the JPGs, 69 ms per frame to encode.

Not in this PR

• Backfilling metadata 0.x does not have (vehicle, map IDs, provenance). That needs the rosbag and Web.Auto, in tier4_perception_dataset.

[ktro2828]

LGTM