--- name: matlab/matlab-driving-data-importer source: https://app.decimal.ai/s/matlab-matlab-driving-data-importer@1/SKILL.md source_sha256: bfb815c37552 --- # Driving Data Importer This skill loads raw driving sensor data into `scenariobuilder.*` objects (`GPSData`, `CameraData`, `LidarData`, `ActorTrackData`, `Trajectory`, `laneData`) and provides the CLI for every preprocessing step DLA exposes (sync, crop, offset, normalize, convert timestamps). It also covers the `drivingLogAnalyzer` (DLA) app **as an opt-in inspection tool** — see *Rule 0* below; DLA is never a default step. After wrapping is done, the canonical next move is `matlab-scenario-builder` (trajectory smoothing, scene/scenario generation, lane localization, RoadRunner / drivingScenario / OpenSCENARIO / OpenDRIVE / OpenCRG / Unreal export). ## Rule 0 — DLA is opt-in only (HARD RULE, READ FIRST) **Never call `drivingLogAnalyzer` unless the user explicitly asks for it or has reported a sensor-data problem DLA is built to debug.** Auto-launching DLA after wrapping data is a first-attempt-success failure: it stalls the user (a UI app forces context-switch, scrub, click, confirm) and signals that the agent is not confident the import worked. **When DLA IS allowed — only these two cases:** 1. **Explicit user request** — user types `DLA`, `drivingLogAnalyzer`, "open in DLA", "inspect / visualize / explore / replay / analyze the recording", "open the driving log analyzer". 2. **Reported sensor-data problem** that DLA is the right tool for: - "the sensors look out of sync" - "camera and lidar timestamps don't match" - "actor cuboids float above the cars" (overlay alignment) - "I see missing frames / a gap in the timeline" - "the offset looks wrong / the timeline is shifted" **When DLA is NOT allowed (defaults — go straight to `matlab-scenario-builder`):** - "Virtualize this data", "build a scenario from this data", "generate a scene", "export to RoadRunner / drivingScenario / OpenSCENARIO / OpenDRIVE / OpenCRG / Unreal", "simulate this drive", "I have data, do something with it". - *Anything that has a downstream simulation or scenario target.* If you ever feel an urge to add a "let me open DLA so you can verify" step after wrapping — **stop.** Save the wrapped objects to `sandbox/_wrapped.mat`, print a short summary, and hand off. ## When to Use - User has raw driving dataset files (KITTI, nuScenes, Waymo, custom logs, ROS/ROS2 bags, .mat, .csv, .xls, video) and needs them loaded into `scenariobuilder.*` objects - User says **"open in DLA"**, **"drivingLogAnalyzer"**, **"inspect / visualize / explore / replay / analyze this dataset"** *(triggers DLA — Rule 0 case 1)* - User wants to map raw structs/tables/rosbag topics to `GPSData`, `CameraData`, `LidarData`, `ActorTrackData`, `Trajectory`, or `laneData` - User wants to attach camera/lidar **Mounting Location / Mounting Angles / Intrinsics / Ego Origin Height** - User asks for **multi-sensor synchronization** of any kind: "sync", "align sensors", "match sample rates", "resample to a common timeline", "sensor-to-sensor alignment" - User asks for **offset correction** ("drag-to-align", "time offset", "shift this sensor by X seconds") - User asks to **crop / trim / extract a segment** of a recording across sensors - User asks to **normalize timestamps** ("common t=0", "time origin", "POSIX to seconds", "datetime to numeric") - User wants to **export sensor data** to MAT, workspace, timetable, or a reproducible script - User needs to inspect dataset structure, identify available sensor modalities, validate calibration transforms, or check for pre-computed annotations - User needs to compute actor tracks from lidar when no annotations exist (clustering / detector / camera-based pipeline) - User reports a **sensor-data problem** (Rule 0 case 2) — wrap, then offer DLA as the debugging path ## When NOT to Use - **Do NOT auto-launch DLA after wrapping.** If the user asked to virtualize / build a scenario / export to a sim format, finish wrapping and hand off to `matlab-scenario-builder` directly. (Rule 0.) - User wants to **build / generate / export a scenario** (RoadRunner, drivingScenario, OpenSCENARIO, OpenDRIVE, OpenCRG, Unreal) — wrap here, then hand off to **`matlab-scenario-builder`**. - User wants to **smooth a trajectory**, **localize ego on a lane**, **correct height on a terrain scene**, **place static objects** (signs/trees/poles), **extract a road surface (OpenCRG)** from lidar, **generate 3D assets** from images, **add elevation** to a map, or **georeference point clouds** — all `matlab-scenario-builder` - User wants to **run sensor-fusion tracking** (`multiSensorTargetTracker`, JPDA + smoother) to get cleaner tracks — `matlab-scenario-builder` Workflow 14 - User is debugging general MATLAB code unrelated to dataset import — use `matlab-debugging` - User wants to install a toolbox or check MATLAB products — use `matlab-list-products` / `matlab-install-products` - Task is about non-driving sensor data (medical imaging, audio, etc.) — out of scope **Boundary heuristic:** wrapping into `scenariobuilder.*` and any sync/crop/offset/normalize CLI work belongs here. The moment the user says *scenario / scene / RoadRunner / simulate / drive in a virtual world / export to OpenSCENARIO / virtualize* — wrap, save, hand off. DLA stays parked unless invoked by name or summoned by a reported problem. ## IMPORTANT — Execution Rules ### Rule 1: Inspect Before Importing **Always inspect the dataset structure first.** Before writing any import code: 1. List the top-level directory structure 2. Identify what sensor modalities are available (GPS, lidar, camera, annotations) 3. Determine if pre-computed annotations/labels exist (3D bounding boxes, tracks) 4. Check calibration files for coordinate frame definitions ### Rule 2: Check for Existing Annotations Before Computing Tracks **Never run a lidar tracker if the dataset already provides actor tracks or 3D bounding box annotations.** Always check first: - Look for annotation files (`object_detection.json`, `labels/`, `annotations/`, `tracking/`) - Check if annotations are per-frame (temporal tracks) or single-keyframe only - If annotations exist, map them directly to `ActorTrackData` - If only keyframe annotations exist (no temporal tracking), inform the user and discuss options ### Rule 3: Understand Dataset Structure Types Driving datasets commonly have two types of recordings: | Type | Duration | Annotations | Use Case | |------|----------|-------------|----------| | **Drives/Logs** | Long (1-10 min) | Often none | Ego trajectory + road network | | **Sequences/Clips** | Short (10-30s) | Usually yes (keyframe or full) | Actor tracks + ego | **Always clarify which type the user's data is** before proceeding. If data lacks annotations, inform the user that actor tracks must be computed (via lidar detection/tracking or camera detection) and set expectations about quality. ### Rule 4: Validate Coordinate Frames Before using any transform, verify: 1. What coordinate frame convention the dataset uses (e.g., X-forward vs Y-forward) 2. Whether extrinsic transforms are sensor-to-ego, sensor-to-vehicle, or sensor-to-sensor 3. Validate by checking that transformed ground points have Z near 0 in ego frame ### Rule 5: Report Data Summary to User After initial inspection, always present a summary: ``` Dataset: Recording: Duration: Available sensors: - GPS: - Lidar: - Camera: - Annotations: Calibration: ``` --- ## Common Dataset Formats ### GPS / GNSS / IMU | Format | How to Read | |--------|-------------| | JSON (lat/lon/alt arrays) | `jsondecode(fileread(file))` | | HDF5 (fields in groups) | `h5read(file, '/group/field')` | | CSV | `readtable(file)` | | ROS bag | `scenariobuilder.GPSData("file.bag", "/topic")` | | NMEA | Custom parser needed | **Key fields needed:** timestamps, latitude, longitude, altitude ### Lidar Point Clouds | Format | How to Read | |--------|-------------| | PCD | `pcread(file)` | | PLY | `pcread(file)` | | BIN (KITTI format) | `reshape(fread(fid,'single'),[4,Inf])'` — columns: x,y,z,intensity | | NPY (custom struct) | Custom reader needed — parse header, read structured bytes | | LAS/LAZ | `lasFileReader(file)` then `readPointCloud` | **Important:** Always check the point cloud coordinate frame. Common conventions: - **X-forward, Y-left, Z-up** (ROS/vehicle standard) - **X-right, Y-forward, Z-up** (some lidars) - **X-forward, Y-right, Z-up** (KITTI) ### Camera Images | Format | How to Read | |--------|-------------| | Directory of images | `imageDatastore(dir)` | | Video file | `VideoReader(file)` | | ROS bag | `rosbag` then read image messages | **Key info needed:** timestamps, file paths, intrinsics, distortion model, extrinsics (camera-to-ego) ### Calibration Calibration files typically provide: - **Intrinsics:** focal length, principal point, distortion coefficients - **Extrinsics:** 4x4 homogeneous transforms between sensor frames - **Distortion model:** pinhole, fisheye (Kannala), equidistant, etc. **Common pitfall:** The naming of extrinsic transforms is inconsistent across datasets. A field named `lidar_extrinsics` could mean: - lidar-to-ego (most common) - lidar-to-camera - ego-to-lidar (inverse) **Always verify** by checking translation values against physical sensor mounting positions (e.g., lidar mounted ~1.7m high should have Z translation ~1.7 in lidar-to-ego). ### 3D Bounding Box Annotations Common formats: ``` Per object: - class: "Car", "Truck", "Pedestrian", etc. - location_3d: [x, y, z] — center position in some reference frame - size: [length, width, height] in meters - orientation: quaternion or yaw angle - track_id: persistent ID across frames (if temporal tracking exists) ``` **Frame of reference:** Annotations may be in: - Ego/vehicle frame (most common for driving datasets) - World/global frame - Sensor frame (lidar or camera) Always check which frame and transform to ego if needed. --- ## Import Pipeline ### Step 1: GPS → GPSData **Canonical GPSData construction is THREE lines, always together** — bare `scenariobuilder.GPSData(...)` is incomplete. The post-construction `convertTimestamps` + `normalizeTimestamps` calls are part of the canonical construction, not optional cleanup. Downstream APIs (`synchronize`, `trajectory`, `actorprops`, `localizeEgoUsingLanes`, RoadRunner export) expect numeric timestamps starting at t=0. ```matlab % Load timestamps, lat, lon, alt from dataset gpsData = scenariobuilder.GPSData(timestamps, latitude, longitude, altitude); % Canonical post-construction pair (always run both) convertTimestamps(gpsData, "numeric"); timeRef = normalizeTimestamps(gpsData); ``` If you skip these two lines, the object will silently fail later (sample-rate mismatches in `synchronize`, scenario time bounds wrong, RoadRunner export errors). Run them every time, even when you "just" construct a `GPSData` for inspection — and even when the prompt only says "construct the GPSData object." **Altitude handling:** - If using OpenStreetMap roads: **zero the altitude** (`altitude = zeros(...)`) — OSM has no elevation - If using a scene with terrain elevation: keep real altitude - If altitude is missing: use zeros ### Step 2: Annotations → ActorTrackData When per-frame 3D annotations with track IDs exist: ```matlab % For each timestamp, collect track IDs and positions timestamps = ; trackIDs = cell(N, 1); % each cell: Mx1 string array positions = cell(N, 1); % each cell: Mx3 [x y z] in ego frame for i = 1:N % Get annotations for frame i frameAnnots = ; trackIDs{i} = string({frameAnnots.track_id}'); positions{i} = [frameAnnots.x, frameAnnots.y, frameAnnots.z]; end trackData = scenariobuilder.ActorTrackData(timestamps, trackIDs, positions); ``` **If annotations are in world frame** (not ego frame): ```matlab % Transform world positions to ego-relative positions % ActorTrackData expects positions relative to ego at each timestamp for i = 1:N worldPos = positions_world{i}; egoPos = egoPositionAtTime(i); % from GPS/odometry egoYaw = egoYawAtTime(i); R = [cos(egoYaw) sin(egoYaw) 0; -sin(egoYaw) cos(egoYaw) 0; 0 0 1]; positions{i} = (worldPos - egoPos) * R'; end ``` ### Step 3: Camera → CameraData ```matlab % Match camera image files to timestamps imageFiles = dir(fullfile(camDir, '*.jpg')); camTimestamps = ; cameraData = scenariobuilder.CameraData(camTimestamps, ... fullfile(camDir, {imageFiles.name}'), Name="FrontCamera"); ``` ### Step 4: Lidar → LidarData **Always wrap lidar via `scenariobuilder.LidarData`** — this is the only wrapper DLA accepts and the only one downstream Scenario Builder APIs (Workflow 10 OpenCRG extraction, Workflow 11 georeferencing) consume. Do not hand DLA raw `pointCloud` arrays or paths. ```matlab % Match lidar files (.pcd, .ply, .las/.laz) to timestamps lidarFiles = dir(fullfile(lidarDir, '*.pcd')); lidarTimestamps = ; lidarData = scenariobuilder.LidarData(lidarTimestamps, ... fullfile(lidarDir, {lidarFiles.name}'), Name="OSLidar"); ``` For multi-lidar setups, build one `scenariobuilder.LidarData` per sensor with a distinct `Name` (e.g., `"OSLidar"`, `"VLP32"`, `"OuterLeft"`). DLA will render each in its own pane. ### Step 5: Synchronize All Sensors ```matlab convertTimestamps(gpsData, "numeric"); convertTimestamps(trackData, "numeric"); timeRef = normalizeTimestamps(gpsData); normalizeTimestamps(trackData, timeRef); synchronize(trackData, gpsData); ``` ### Step 6: Launch DLA — only when explicitly requested (Rule 0) **`drivingLogAnalyzer` does NOT accept programmatic sensor inputs.** It opens the app; the user then imports sensors via the GUI (`Import → From Workspace`). Make sure the wrapped objects are in the base workspace, then launch the app bare: ```matlab % Wrapped objects must already exist in the base workspace % (gpsData, cameraData, lidarData, trackData) drivingLogAnalyzer; % user clicks Import → From Workspace ``` Do NOT call forms like `drivingLogAnalyzer(sensors, Plot=true)` or `drivingLogAnalyzer(gpsData, Plot=true)` — those signatures are not supported and will error. Remember Rule 0: launch DLA only when the user explicitly asks for it or reports a sensor-data problem DLA is built to debug. (See [`workflow-driving-log-analyzer.md`](references/workflow-driving-log-analyzer.md) for the full opt-in recipe.) --- ## When No Annotations Exist — Computing Actor Tracks from Lidar If the dataset has NO pre-computed 3D bounding boxes or temporal tracks, actor tracks must be computed. **Always inform the user** about: 1. This requires significant processing and tuning 2. Results depend heavily on scene complexity (open highway = good, dense urban = poor) 3. Deep learning detectors (PointPillars, CenterPoint) give much better results than clustering ### Option A: Deep Learning Detector (Preferred) ```matlab % Requires a trained model (e.g., PointPillars) detector = pointPillarsObjectDetector(net, pcRange, classNames, anchorBoxes); bboxes = detect(detector, ptCloud); ``` ### Option B: Classical Pipeline (Clustering + Tracking) ```matlab % Per frame: % 1. Transform lidar to ego frame % 2. Remove ground plane % 3. Filter ROI % 4. Euclidean clustering % 5. Size filtering % 6. Feed detections to tracker (JPDA or GNN) ``` **Known limitations of clustering approach:** - Dense urban scenes: parked cars, buildings, and road infrastructure form continuous surfaces that cannot be segmented into individual objects - Mega-clusters: thousands of points merging into single clusters spanning 20+ meters - False positives: poles, signs, trees, walls pass size filters - Works best on: highways, open roads with isolated vehicles ### Option C: Camera-Based Detection ```matlab % Use pretrained camera detector detector = vehicleDetectorYOLOv2(); % or vehicleDetectorFasterRCNN() [bboxes, scores] = detect(detector, img); ``` **Limitation:** Gives 2D boxes only. Requires depth estimation or lidar fusion for 3D positions. --- ## Projection: Lidar Points → Camera Image To verify detection alignment or overlay lidar on camera: ```matlab % Transform lidar to camera frame T_lidar2cam = inv(T_cam2ego) * T_lidar2ego; % compose transforms pts_cam = (T_lidar2cam * [pts_lidar, ones(N,1)]')'; % Project to pixels (pinhole model) inFront = pts_cam(:,3) > 0; u = fx * pts_cam(inFront,1) ./ pts_cam(inFront,3) + cx; v = fy * pts_cam(inFront,2) ./ pts_cam(inFront,3) + cy; % Display imshow(img); hold on; scatter(u, v, 1, depth, 'filled'); ``` **For fisheye cameras:** Standard pinhole projection will have errors at image edges. Use the camera's distortion model for accurate projection. --- ## Checklist Before Starting Import - [ ] What sensors does the dataset provide? - [ ] Are there pre-computed annotations/3D bounding boxes? - [ ] Are annotations per-frame (temporal) or keyframe-only? - [ ] What coordinate frame are positions in? - [ ] What is the calibration transform naming convention? - [ ] Does altitude data exist? Will OSM or scene-based roads be used? - [ ] What is the recording duration? Need to crop? ---- Copyright 2026 The MathWorks, Inc. ----