feat(schema): load 0.x vehicle states and traffic light links, backfill source LiDARs and capture time

docs/schema/v1/index.md

@@ -138,9 +138,11 @@ Provenance {
 
 Every 0.x frame becomes a `sample`: the files of all channels with the same frame number (e.g. `00042.jpg` and `00042.pcd.bin`, as the 0.x converter names them) form one frame. 0.x samples, which are annotated frames only, keep their tokens; 0.x samples of the same frame (one per camera image in camera-only datasets) are merged into the earliest one. A frame with a 2D label is a key frame.
 
-Each 0.x `calibrated_sensor` row becomes a `sensor` row, keeping its token. The camera block takes its intrinsics and distortion from it, its size from `sample_data`, and the distortion model from the number of coefficients (0: `none`, 5: `plumb_bob`, 8: `rational_polynomial`). The LiDAR block is described from the first frame. Map `sha256` and `projector` are read from `map/lanelet2_map.osm` and `map/map_projector_info.yaml`.
+Each 0.x `calibrated_sensor` row becomes a `sensor` row, keeping its token. The camera block takes its intrinsics and distortion from it, its size from `sample_data`, and the distortion model from the number of coefficients (0: `none`, 5: `plumb_bob`, 8: `rational_polynomial`). The LiDAR block is described from the first frame. The map is the dataset's own `map/` folder: `sha256` and `projector` are read from `map/lanelet2_map.osm` and `map/map_projector_info.yaml`, and map element ids (e.g. lamp ways) refer to it. An empty `log.data_captured` becomes the log's first sample time (ISO 8601, UTC).
 
-Labels are `manual` unless 0.x marks them as autolabel (`provenance` = model names). `surface_ann` rows become `object_ann` rows without an instance, with `bbox` computed from the mask. 0.x visibility `unavailable` (or any unknown level) becomes `visibility_token: null`, and the aliases `v0-40`, `v40-60`, `v60-80`, `v80-100` map to `none`, `partial`, `most`, `full`.
+Labels are `manual` unless 0.x marks them as autolabel (`provenance` = model names). Categories without `index` get their position in `category.json`, as the 0.x devkit reads them (all set or none). `surface_ann` rows become `object_ann` rows without an instance, with `bbox` computed from the mask. 0.x visibility `unavailable` (or any unknown level) becomes `visibility_token: null`, and the aliases `v0-40`, `v40-60`, `v60-80`, `v80-100` map to `none`, `partial`, `most`, `full`.
+
+`vehicle_state` rows get `source: vehicle_interface` (where the 0.x converter reads them) and shift `FORWARD`/`HIGH` becomes `DRIVE`; `additional_info.speed` is dropped and the velocities are `null`. The 0.x `traffic_light` table (`primitive_id`) and `traffic_light_instance_map` table (`traffic_light_linestring_id`, which wins if both list a lamp) link a lamp instance to its lanelet2 way (`type=traffic_light`) and have no states: the way becomes that instance's `map_element` (`type: way`), and the 1.0 `traffic_light` table stays empty. Source LiDARs of a concatenation have no frames of their own in 0.x: they take the concatenated channel's format and `fields`, and `frame_count` is the number of concatenated frames whose `.info` file lists points from them.
 
 1.0 requires some values that 0.x never stored. They are filled with these placeholders (`V0Converter.FALLBACKS`), which keep the dataset loadable but say nothing about it. In particular, `vehicle.model == "unknown"` means the vehicle dimensions are placeholders and must not be used, and `"unknown"` map IDs mean the map is not identified:
 

t4_devkit/schema/v1/convert_v0.py

@@ -4,7 +4,9 @@
 import hashlib
 import os.path as osp
 import re
+import warnings
 from collections import defaultdict
+from datetime import datetime, timezone
 from glob import glob
 from typing import TYPE_CHECKING, Any, ClassVar
 
@@ -89,6 +91,8 @@ def __init__(self, data_root: PathLike) -> None:
         }
         # 0.x samples are annotated frames only; 1.0 has one sample per frame.
         self.samples, self.sample_of_frame, self.sample_remap = self._frame_samples()
+        self.infos: dict[str, list[Record]] = {}  # filled by `_frame_infos`
+        self.traffic_light_ways = self._traffic_light_ways()
 
     def convert(self) -> dict[SchemaName, list[Record]]:
         """Convert every table.
@@ -118,7 +122,7 @@ def convert(self) -> dict[SchemaName, list[Record]]:
             self.convert_sample_data(record) for record in v0["sample_data"]
         ]
 
-        tables[SchemaName.CATEGORY] = [self.convert_category(record) for record in v0["category"]]
+        tables[SchemaName.CATEGORY] = self.convert_categories()
         tables[SchemaName.ATTRIBUTE] = [
             self.convert_attribute(record) for record in v0["attribute"]
         ]
@@ -131,6 +135,9 @@ def convert(self) -> dict[SchemaName, list[Record]]:
             self.convert_object_ann(record) for record in v0["object_ann"]
         ] + [self.convert_surface_ann(record) for record in v0["surface_ann"]]
         tables[SchemaName.KEYPOINT] = [self.convert_keypoint(record) for record in v0["keypoint"]]
+        tables[SchemaName.VEHICLE_STATE] = [
+            self.convert_vehicle_state(record) for record in v0["vehicle_state"]
+        ]
         # after the labels: an instance's count and first / last label are over all its labels
         annotations = self._instance_annotations(tables)
         tables[SchemaName.INSTANCE] = [
@@ -182,7 +189,7 @@ def convert_log(self, record: Record, log2map: dict[str, str]) -> Record:
             "token": record["token"],
             "logfile": self._none_if_empty(record["logfile"]),
             "vehicle_token": self._vehicle_token(record),
-            "data_captured": record["data_captured"],
+            "data_captured": record["data_captured"] or self._capture_time(record["token"]),
             "location": self._none_if_empty(record["location"]),
             "map_token": log2map.get(record["token"]),
         }
@@ -259,14 +266,11 @@ def convert_camera(self, calibration: Record) -> Record:
     def convert_lidar(self, calibration: Record) -> Record:
         """From the first frame: its format, its `.info` file (if any) and its header."""
         frames = self.frames[calibration["token"]]
-        first = frames[0] if frames else {}
-        info_filename = first.get("info_filename")
-        info = (
-            load_json(osp.join(self.data_root, info_filename))
-            if info_filename and osp.exists(osp.join(self.data_root, info_filename))
-            else {}
-        )
-        if first.get("fileformat") == "pcd":
+        if not frames:
+            return self._convert_source_lidar(calibration)
+        first = frames[0]
+        info = self._frame_infos(calibration["token"])[0]
+        if first["fileformat"] == "pcd":
             filepath = osp.join(self.data_root, first["filename"])
             data = pypcd4.PointCloud.from_path(filepath).pc_data
             container_format = "pcd"
@@ -312,13 +316,27 @@ def convert_sample_data(self, record: Record) -> Record:
 
     # --- label tables ---
 
-    def convert_category(self, record: Record) -> Record:
+    def convert_categories(self) -> list[Record]:
+        """Categories without `index` get their list position, as the 0.x devkit reads them.
+
+        Raises:
+            ValueError: If some categories have an `index` and others do not.
+        """
+        records = self.v0["category"]
+        indices = [record.get("index") for record in records]
+        if None in indices:
+            if any(index is not None for index in indices):
+                raise ValueError("Category index is not set for some categories.")
+            indices = list(range(len(records)))
+        return [self.convert_category(record, index) for record, index in zip(records, indices)]
+
+    def convert_category(self, record: Record, index: int) -> Record:
         """`is_thing` is false only for categories used by `surface_ann` (road, sky) and nothing else."""
         token = record["token"]
         return {
             "token": token,
             "name": record["name"],
-            "index": record.get("index"),
+            "index": index,
             "description": self._none_if_empty(record.get("description")),
             "is_thing": token in self.thing_categories or token not in self.surface_categories,
             "has_orientation": record.get("has_orientation", False),
@@ -350,8 +368,12 @@ def convert_visibilities(self) -> list[Record]:
     def convert_instance(
         self, record: Record, annotations: dict[str, tuple[int, str, str]]
     ) -> Record:
-        """`nbr_annotations` and the first / last annotation come from `annotations`: 1.0 counts
-        3D boxes and 2D labels, and 0.x tools count them differently (some write 0 for 2D)."""
+        """A lamp listed in 0.x traffic_light points to its lanelet2 way in `map_element`.
+
+        `nbr_annotations` and the first / last annotation come from `annotations`: 1.0 counts
+        3D boxes and 2D labels, and 0.x tools count them differently (some write 0 for 2D).
+        """
+        way_id = self.traffic_light_ways.get(record["token"])
         count, first, last = annotations.get(record["token"], (0, None, None))
         return {
             "token": record["token"],
@@ -361,7 +383,7 @@ def convert_instance(
             "nbr_annotations": count,
             "first_annotation_token": first,
             "last_annotation_token": last,
-            "map_element": {"id": None, "type": None},
+            "map_element": {"id": way_id, "type": "way" if way_id is not None else None},
         }
 
     def convert_sample_annotation(self, record: Record, known_visibility: set[str]) -> Record:
@@ -396,8 +418,9 @@ def convert_object_ann(self, record: Record) -> Record:
             "instance_token": record["instance_token"],
             "category_token": record["category_token"],
             "attribute_tokens": record["attribute_tokens"],
-            "bbox": record["bbox"],
-            "mask": record.get("mask"),
+            # 0.x leaves bbox out for mask-only labels; 1.0 requires it.
+            "bbox": record.get("bbox") or self._mask_bbox(record["mask"]),
+            "mask": self._non_empty_mask(record.get("mask")),
             "orientation": record.get("orientation"),
             "number": record.get("number"),
             "annotation_type": annotation_type,
@@ -408,16 +431,14 @@ def convert_object_ann(self, record: Record) -> Record:
     def convert_surface_ann(self, record: Record) -> Record:
         """A 0.x semantic mask becomes an `object_ann` without instance; its box is the mask's."""
         mask = record["mask"]
-        rle = {"counts": base64.b64decode(mask["counts"]), "size": mask["size"]}
-        x, y, w, h = (float(v) for v in cocomask.toBbox(rle))
         annotation_type, provenance, confidence = self._label_origin(record)
         return {
             "token": record["token"],
             "sample_data_token": record["sample_data_token"],
             "instance_token": None,
             "category_token": record["category_token"],
             "attribute_tokens": [],
-            "bbox": [x, y, x + w, y + h],
+            "bbox": self._mask_bbox(mask),
             "mask": mask,
             "orientation": None,
             "number": None,
@@ -440,6 +461,34 @@ def convert_keypoint(self, record: Record) -> Record:
             "confidence": confidence,
         }
 
+    # --- planning tables ---
+
+    def convert_vehicle_state(self, record: Record) -> Record:
+        """0.x vehicle states come from the Autoware vehicle interface (`/vehicle/status/*`).
+
+        `additional_info.speed` (norm of the longitudinal and lateral velocity) is dropped: it can't
+        be split back into the 1.0 velocities.
+        """
+        shift_state = record.get("shift_state")
+        # The 0.x devkit names forward gears FORWARD or HIGH; the 0.x converter writes DRIVE.
+        if shift_state in ("FORWARD", "HIGH"):
+            shift_state = "DRIVE"
+        return {
+            "token": record["token"],
+            "timestamp": record["timestamp"],
+            "accel_pedal": record.get("accel_pedal"),
+            "brake_pedal": record.get("brake_pedal"),
+            "steer_pedal": record.get("steer_pedal"),
+            "steering_tire_angle": record.get("steering_tire_angle"),
+            "steering_wheel_angle": record.get("steering_wheel_angle"),
+            "shift_state": shift_state,
+            "indicators": record.get("indicators"),
+            "longitudinal_velocity": None,
+            "lateral_velocity": None,
+            "heading_rate": None,
+            "source": "vehicle_interface",
+        }
+
     # --- helpers ---
 
     def _frame_samples(self) -> tuple[list[Record], dict[str, str], dict[str, str]]:
@@ -572,6 +621,110 @@ def _read_projector(self) -> Record:
             },
         }
 
+    def _convert_source_lidar(self, calibration: Record) -> Record:
+        """A LiDAR without frames of its own: a source of a concatenated channel.
+
+        Its points are stored in the concatenated files, so it has their format and fields; it
+        counts a frame wherever the `.info` file of a concatenated frame lists points from it.
+        """
+        sensor_token = calibration["sensor_token"]
+        fallbacks = self.FALLBACKS["lidar"]
+        container_format, fields, frame_count = "bin", [], 0
+        for concat in self.v0["calibrated_sensor"]:
+            infos = self._frame_infos(concat["token"])
+            lengths = [
+                sum(
+                    s["length"]
+                    for s in info.get("sources", [])
+                    if s["sensor_token"] == sensor_token
+                )
+                for info in infos
+            ]
+            if any(lengths):
+                concat_lidar = self.convert_lidar(concat)
+                container_format = concat_lidar["container"]["format"]
+                fields = concat_lidar["fields"]
+                frame_count = sum(length > 0 for length in lengths)
+                break
+        return {
+            "container": {
+                "format": container_format,
+                "path": None,
+                "frame_count": frame_count,
+                "compression": fallbacks["compression"],
+            },
+            "fields": fields,
+            "source_channels": [],
+            "frame_id": fallbacks["frame_id"],
+            "is_undistorted": fallbacks["is_undistorted"],
+        }
+
+    def _capture_time(self, log_token: str) -> str:
+        """The log's first sample time, ISO 8601 UTC, for logs without `data_captured`."""
+        scenes = {
+            record["token"] for record in self.v0["scene"] if record["log_token"] == log_token
+        }
+        timestamp = min(
+            record["timestamp"] for record in self.v0["sample"] if record["scene_token"] in scenes
+        )
+        return datetime.fromtimestamp(timestamp / 1e6, tz=timezone.utc).isoformat()
+
+    def _traffic_light_ways(self) -> dict[str, int]:
+        """The lanelet2 way (`type=traffic_light`) each lamp instance shows, by instance token.
+
+        Two 0.x tables hold this link: `traffic_light` (`primitive_id`, #288) and
+        `traffic_light_instance_map` (`traffic_light_linestring_id`, from the unmerged
+        `feat/traffic-light-instance-map` branch), which wins when both list a lamp.
+        `traffic_light` rows without `primitive_id` are skipped, as no 0.x devkit reads them.
+        """
+        ways = {}
+        skipped = 0
+        for record in self.v0["traffic_light"]:
+            if "primitive_id" in record:
+                ways[record["instance_token"]] = int(record["primitive_id"])
+            else:
+                skipped += 1
+        if skipped:
+            warnings.warn(f"Skipped {skipped} traffic_light rows without primitive_id")
+        for record in self.v0["traffic_light_instance_map"]:
+            ways[record["instance_token"]] = int(record["traffic_light_linestring_id"])
+        return ways
+
+    def _frame_infos(self, calibration_token: str) -> list[Record]:
+        """The 0.x `.info` file of each frame of a channel ({} where there is none), read once."""
+        if calibration_token not in self.infos:
+            self.infos[calibration_token] = [
+                load_json(osp.join(self.data_root, frame["info_filename"]))
+                if frame.get("info_filename")
+                and osp.exists(osp.join(self.data_root, frame["info_filename"]))
+                else {}
+                for frame in self.frames[calibration_token]
+            ]
+        return self.infos[calibration_token]
+
+    @staticmethod
+    def _non_empty_mask(mask: Record | None) -> Record | None:
+        """None for a mask without pixels.
+
+        Some 0.x tools give box-only labels a placeholder: an empty RLE whose `size` is
+        (width, height) instead of the (height, width) of real masks.
+        """
+        if mask is None or "counts" not in mask:
+            return None
+        rle = {"counts": base64.b64decode(mask["counts"]), "size": mask["size"]}
+        return mask if cocomask.area(rle) > 0 else None
+
+    @staticmethod
+    def _mask_bbox(mask: Record) -> list[float]:
+        """Box around a 0.x RLE mask, as (xmin, ymin, xmax, ymax) on pixel edges.
+
+        Same convention as 0.x boxes, which the 0.x converter writes as (x, y, x + w, y + h)
+        clipped to the image size. Mask `size` is (height, width), as in COCO.
+        """
+        rle = {"counts": base64.b64decode(mask["counts"]), "size": mask["size"]}
+        x, y, w, h = (float(v) for v in cocomask.toBbox(rle))
+        return [x, y, x + w, y + h]
+
     @staticmethod
     def _vehicle_token(log: Record) -> str:
         """Deterministic token: 0.x has no vehicle table."""