Skip to content

feat(schema): caller-given metadata in V0Migrator.migrate (backfill) - #319

Merged
jacoblambert merged 1 commit into
v1-devfrom
v1/migrate-backfill
Oct 22, 2026
Merged

jacoblambert merged 1 commit into
v1-devfrom
v1/migrate-backfill

Conversation

@jacoblambert

@jacoblambert jacoblambert commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Summary

This PR lets a caller add the metadata 0.x never stored when migrating with V0Migrator. It is what the backfill in tier4_perception_dataset (convert_t4_to_t4_v1) builds on.

migrator = V0Migrator("<0.x dataset>", "out/<DATASET_ID>/<DATASET_VERSION>")
migrator.tables(SchemaName.VEHICLE)[0].model = "j6_gen2"  # records: edit or append
migrator.migrate("<DATASET_ID>", <DATASET_VERSION>, manifest_fields={"project_id": "x2_dev", "provenance": {...}})
  • migrate(..., manifest_fields=...): the given manifest fields replace the defaults (provenance, project_id, generators, ...). Their migration_notes come after the migration's own.
  • When the caller gives provenance, the note "record start / end are the first / last sample times" is not added.
  • Records were already editable through migrator.tables(...); the docs now say so.

One spec line changes: ego_pose.acceleration is what an IMU measures, gravity included (az ≈ +9.8 m/s² at rest). The backfill reads /sensing/imu/imu_data (base_link). /localization/acceleration is longitudinal only, and the raw Tamagawa IMU is FRD, so neither fits.

Tests

  • test_migrate_with_backfilled_metadata: an edited vehicle and caller-given project, provenance, generators and notes end up in the written dataset, which passes sanity.

Checked on three real db_j6gen2_v12 scenes (1d095bce, 610e6b9f, 3a0c82f1): t4sanity --checksum 101 passed, 0 failed. Vehicle, map ids, provenance, Autoware version, ego twist / acceleration / GNSS and vehicle state are filled; REC025 (placeholders) no longer warns.

Related PRs/branches:

@jacoblambert
jacoblambert added this pull request to stack #315 October 21, 2026 01:32
@jacoblambert
jacoblambert removed this pull request from stack #315 October 22, 2026 08:55
Base automatically changed from v1/cleanup to v1-dev October 22, 2026 09:58
@jacoblambert
jacoblambert merged commit ae425e6 into v1-dev Oct 22, 2026
0 of 5 checks passed
@jacoblambert
jacoblambert deleted the v1/migrate-backfill branch October 22, 2026 09:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants