Telemetry & Data Pipeline

The Nexus SDV telemetry pipeline is engineered for high-throughput and low-latency data ingestion. It ensures that vehicle signals are not just stored, but are immediately accessible for real-time analytics, AI workloads, or any other connected vehicle services requiring transactional data processing.

---

In-Vehicle Data Generation

Data generation starts at the vehicle edge, where signals are serialized using Protocol Buffers (Protobuf). This ensures a compact binary format that is ideal for constrained automotive networks and high-frequency transmission.

• Protobuf Definition: The core data structure is defined in telemetry.proto. This file acts as the single source of truth for both the vehicle SDKs and the cloud ingestion logic.
• Static vs. Dynamic Mapping: Nexus introduces a dual-mapping strategy to optimize how data is persisted in the cloud:
  • Static Mapping: Used for attributes that rarely change (e.g., vehicle metadata, hardware versions). These map to the static column family in Bigtable.
  • Dynamic Mapping: Used for high-frequency sensor data and time-series signals (e.g., speed, battery temp, GPS coordinates). These map to the dynamic column family, allowing for efficient time-range queries.

---

Connectivity & Messaging (NATS)

Nexus leverages NATS as its high-performance, distributed messaging backbone, effectively decoupling data producers from cloud consumers by providing a low-latency pub/sub infrastructure.

• Capability-Based Subject Architecture: Top-level NATS subjects in Nexus serve as a functional directory of the platform’s capabilities.
  • telemetry: The current primary subject for high-frequency data ingestion.
  • command (Planned): Future extension point for Command & Control patterns, allowing bidirectional interaction with the vehicle.
• Hierarchical Pattern: To enable efficient routing, sharding, and security at scale, Nexus follows a strictly structured subject pattern: telemetry.prod.bigtable.{vin}
• Core Differentiator: Unlike traditional brokers (e.g., classic MQTT), NATS subjects are ephemeral resources that require no pre-definition or central management. A subject “exists” only as long as there is an active interest (subscription) or a message flow, making the system significantly more lightweight and dynamic.
• Massive Scale Handling: NATS is designed to handle tens of millions of unique subjects effortlessly. In a production environment with hundreds of thousands of vehicles, NATS manages these subjects implicitly; they consume no server resources when inactive and are not “stored” as stateful entities, ensuring linear scalability as the fleet grows.

Implementation

The hierarchical subject logic is a cross-platform standard within the Nexus ecosystem:

• Python & Go Clients: Both the Python Sample Client and the Go-based reference clients implement this identical pattern. This ensures that the platform can automatically shard traffic or apply fine-grained security policies based on the unique Vehicle Identification Number (VIN), regardless of the client’s language stack.

---

Nexus nats-bigtable-connector

The bridge between the messaging layer and the storage engine is the nats-bigtable-connector. This component is responsible for consuming NATS messages and performing the write operations to Google Cloud Bigtable.

• Deployment & Orchestration: The connector is managed via Helm. Its deployment logic is defined in the nats-connector.yaml.gotmpl, ensuring it can be scaled independently within the GKE cluster.
• The Mapping Logic: The core intelligence of the connector resides in its ConfigMap Template. This configuration defines the Proto-to-Bigtable mapping, instructing the connector on which Protobuf fields to route into which Bigtable columns and families.

---

The “Cloud ECU” (Bigtable)

At the end of the pipeline, Google Cloud Bigtable acts as the primary time-series sink, effectively serving as a Virtual/Cloud ECU that mirrors the vehicle’s state in real-time.

• Table: telemetry
• Column Families:
  • static: Optimized for point-lookups of vehicle configuration data.
  • dynamic: Designed for high-scale ingestion of sensor signals, supporting millions of writes per second per cluster.

This schema-on-read approach allows Nexus to store raw signals 1:1, providing the flexibility needed for future AI training, real-time fleet analytics, or transactional connected vehicle services via BigQuery and Vertex AI.

Bigtable RowKey Design

In Cloud Bigtable, the RowKey is the only indexed field and the primary mechanism for data distribution and retrieval. Nexus SDV uses a classic time-series pattern for its RowKeys to ensure both high write-throughput and efficient query performance.

RowKey Structure

The RowKey follows a structured, dimension-based pattern: [VehicleID]#[Timestamp] Example: VEHICLE001#2026-02-12T16:54:52.746457Z

• VehicleID (Prefix): By using the unique vehicle identifier as the prefix, Nexus ensures Data Locality. All data points for a single vehicle are stored physically close to each other on the same tablet, minimizing disk I/O for vehicle-specific lookups.
• Separator (#): Acts as a clear delimiter between dimensions, preventing “bleeding” between variable-length IDs.
• ISO Timestamp: Timestamps are appended in ISO 8601 format. Since Bigtable stores RowKeys in lexicographical order, this ensures that data for each vehicle is automatically sorted chronologically.

---

Efficient Data Retrieval

The RowKey design is optimized for “search-friendly” access patterns required by automotive engineering teams:

• Vehicle-Wide Scans: To retrieve the entire history of a specific car, a simple Prefix Scan on VEHICLE001# is performed.
• Time-Range Queries: Because of the lexicographical sorting, querying a specific time window (e.g., “last 5 minutes”) is highly efficient. The application simply defines a start and end RowKey (e.g., VEHICLE001#2026-02-12T16:50... to VEHICLE001#2026-02-12T16:55...) to fetch a contiguous block of data.

Technical Reference: The exact mapping of Protobuf fields into the RowKey is configured via the NATS-Bigtable-Connector ConfigMap, where the _key field is dynamically constructed during the ingestion process.