[jacoblambert]

Summary

This PR adds a writer for 1.0 datasets. Migration (next PR) and later converters use it.

The writer writes files. The caller decides what goes in them.

Data files

t4_devkit/common/io.py, next to the readers:

• save_pointcloud writes the frames of a LiDAR channel to one Parquet file. Each frame is one row group, so a frame can be read alone. Empty frames get a row group too, so frame indices stay aligned. A field that is missing or None in a frame is written as null on every point, for point labels of frames without labels. The reader leaves such a column out, so what it returns can be written back as it is.
• save_video writes the frames of a camera channel to one AV1 video. It uses the SVT-AV1 encoder that comes with PyAV, so no FFmpeg install is needed. Every frame is a keyframe, so any frame can be read alone. The timescale is the frame rate, so a frame's pts is its index. uint16 frames give a 10-bit video.

DatasetWriter

t4_devkit/schema/v1/writer.py:

• write_lidar and write_camera write a channel to data/<CHANNEL>/<CHANNEL>.parquet or .mp4.
• copy copies files as they are, for example the map or input_bag/.
• save writes the tables, then manifest.json. The manifest lists every table and every file under data/, map/ and input_bag/ with its hash and size.

save also fills in two manifest fields:

• generators: t4-devkit is added.
• capabilities: computed from the tables, unless the caller gives them.

Capabilities

Capabilities.from_tables works out what a dataset contains from its tables. The writer uses it to fill capabilities. Sanity rules REC023 and REC024 use it to compare the manifest with the tables. Before, sanity had its own copy of this logic.

prediction and planning are now checked too:

• prediction: every sample has a key frame, so all frames are annotated.
• planning: scenario.json has records.

The 1.0 sample has scenarios, so its manifest now lists planning.

Dataset ID and version

The writer does not choose or increase the version.

The caller gives the folder, usually <DATASET_ID>/<VERSION>. The caller also gives dataset_id and version for manifest.json. The writer refuses a folder that already exists, so it never overwrites a version.

This is on purpose. The version number comes from Web.Auto when a new version is uploaded. Migration will run in Web.Auto, so Web.Auto sets the version. A local tool cannot know the next number.

The folder and the manifest are not tied together by the writer. If they differ, sanity rule STR015 warns.

Parquet settings

save_pointcloud picks the encoding of each column by its type:

• floats: byte-stream-split;
• 8-bit integers (intensity, return type, source index, label): dictionary;
• wider integers (ring, timestamps, instance): delta;
• zstd level 1.

We benchmarked this on 13 real 0.x datasets (6,296 frames, 40.8 GB of .bin). These were the smallest settings that every reader we tried can read: pyarrow 14+, Polars and DuckDB.

On 4 of them (4, 5, 7 and 9 columns), the result is 43.4 % of the .bin size, with every value unchanged. Before, it was about 46 %. The 7-column dataset with per-point timestamps gains the most: 44.5 % to 38.6 %.

Two settings were left out on purpose:

• Byte-stream-split on integers: about 0.1 % smaller, but DuckDB and pyarrow before 16 cannot read it.
• zstd level 3: about 1.5 % smaller. We kept level 1: the gain is small.

Sample dataset

make_sample_v1.py now writes the 1.0 sample with DatasetWriter, like a real dataset. It no longer needs an FFmpeg build with SVT-AV1, or npx.

• Table contents are unchanged. Only the JSON layout changed: arrays are no longer collapsed by prettier.
• LiDAR points are bit-exact (the file changes: new encodings). The video is encoded again: pixels differ from the old video by at most 2.
• generators now has the real t4-devkit version instead of "sample".
• Running it twice gives the same files.

prettier now skips tests/sample/t4dataset_v1/. It reformatted the JSON after the hashes in manifest.json were taken.

Other changes

• Table JSON is written with 2-space indent.
• In sanity record/base.py, constants used only once moved into their function.
• save_video sets SVT_LOG=1 unless it is already set. Without it, SVT-AV1 prints about 20 lines for every video.

Tests

• Parquet round trip, bit-exact, with uint8 / uint16 / uint32 columns and an empty frame; each column has the encoding for its type.
• A label column that is None in one frame: written as null, left out when read, and written back the same.
• Video round trip, 8-bit and 10-bit, frames read back out of order.
• Write the 1.0 sample again with the writer, then load it: every table has the same records. All 1.0 sanity rules pass with --checksum.
• LiDAR and camera files written again through the writer: LiDAR is bit-exact, and the video block matches the sample.
• An existing folder is refused.
• Capabilities: prediction and planning appear and disappear with their tables.