Real-Time Telemetry Analytics Pipeline

A racing simulator emits a dense stream of binary UDP telemetry, around 30 packets a second across several packet types, and the goal was to turn that stream into live dashboards and a long-term performance record. The protocol is a set of fixed-width C structures with variable-length tails and no framing beyond a shared header, so a decoding error does not raise an exception; it silently sends data to the wrong place.

Storage raised a second problem. With 30-day retention, the time-series database was quietly discarding historical best results. And a lap's total time is only reported during the following lap, so a completed lap cannot be read from any single moment in the stream.

The design separates raw data from derived results. A containerized Python ingester validates and decodes each packet type into InfluxDB with session tags, Grafana serves live dashboards, and InfluxDB keeps 90 days of raw data as a safety net for re-deriving results.

An hourly Jenkins job reconstructs completed laps from the stream, writes one durable row per lap to PostgreSQL, runs its analysis and publishes the results to Confluence. It is built to run unattended: it checks for new data before doing anything expensive, skips publishing when nothing has changed and cleans up after itself even when a build is aborted.

The system runs as a Docker Compose stack plus one self-contained Jenkins pipeline that reaches every service over the network.

• The header-offset bug: packets were being silently misrouted because the decoder read the packet ID from byte offset 5; the header's first field is two bytes wide, so that byte was actually the version. Unpacking the full header structure fixed it, and synthetic packets confirmed the whole path end to end.
• Size guards: every packet type is checked against its documented length, so malformed or unexpected packets are logged and skipped rather than mis-decoded, and variable-length sections are never parsed blindly.
• Lap reconstruction: each lap pairs its own sector times and validity with the total reported during the next lap, derives the final sector, and works from one-second windows.
• Relational model: PostgreSQL tables for laps, runs, configurations, corners and braking points, with views for personal and theoretical bests. Inserts are idempotent, and a high-water mark lets the job skip idle runs.
• Query performance: a server-side Flux join that did not finish in 400 seconds was replaced by two focused queries of a few seconds each, merged on timestamp in Python, bringing a full rebuild down to about nine minutes.
• Abort safety: each build runs in a uniquely named container that a post { always } block removes, after orphaned containers from aborted builds were found overloading the database.
• Rule-based promotion: a challenger configuration replaces the current best only when it meets minimum sample-size, improvement, median and invalid-rate thresholds, and every promotion is recorded for audit.
• Idempotent publishing: generated Confluence sections are replaced between stable anchors through the REST API, and pages are left untouched when no data changed.