Media Ingestion & Curation Pipeline

A self-hosted Plex server covering movies, TV and concert video had grown through ad-hoc work: hand-edited queue files, a batch job whose logic lived only in the Jenkins UI, and helper scripts that existed only on the file server. One bad URL stopped the whole batch and stranded every entry after it, and a long-standing bug meant the "already processed" flag was never actually honored. The download image was never rebuilt, so its downloader aged until downloads began failing without anyone noticing.

Metadata had problems of its own. Automatic matching was sometimes wrong, curation done in the Plex UI could not be versioned or queried, and the linear-TV service had no air dates for local episodes, so chronological scheduling did not work.

The pipeline was rebuilt around a single ingestion path. A small FastAPI form writes exactly the queue file a person would write by hand, and a version-controlled Jenkins pipeline consumes it. Each entry runs as its own containerized download build, failures are collected rather than fatal, and failed entries are written back out as a retry queue. Waiting stages run without an agent so the batch cannot starve the executors it depends on, and a weekly image rebuild with a live smoke test keeps the downloader current.

PostgreSQL became the catalog of record for the Plex server, with a daily change-detected sync pushing locked titles, cast and collections through the Plex API. Movie and TV workflows standardize malware scanning, track cleanup, naming and tagging before anything reaches the NAS, and generated metadata sidecars give the linear-TV channels real air dates.

The pipeline is built from declarative Jenkins pipelines, containers and Python tooling, all kept in version control.

• Submission form: FastAPI validates every field (no tabs or newlines, HTTP(S) URLs only, relative destination paths, allow-listed target directories) and writes queue files atomically, so the batch job can never read a half-written file. Its health check verifies both that the queue is writable and that Jenkins is reachable.
• Batch orchestration: a four-stage pipeline in which only the parse and finalize stages take an agent, eliminating the executor-holding deadlock pattern.
• Continue on failure: each entry's build is isolated; failed entries are copied verbatim into a retry file with an error log, the build is marked unstable, and fully successful queue files are archived.
• Quoting-safe dispatch: job fields cross the SSH boundary as a single encoded payload, so apostrophes and empty values can never shift arguments.
• Downloader container: yt-dlp merges to MP4, filenames are normalized, FFmpeg writes tags with a stream copy, and a size-verified copy lands on the NFS share. A manifest records only successful files, and the batch cross-checks it against successful builds.
• Image freshness: a weekly no-cache rebuild runs a live extraction smoke test and refuses to push an image that cannot download.
• Catalog and sync: a unified PostgreSQL table (one row per movie or episode) feeds a daily sync that pushes only rows that changed since the last push, with a force option to re-assert everything.
• Library workflows: malware scanning, English-only audio and subtitle cleanup, standard Title (Year) and Season NN/SxxEyy naming, and AtomicParsley tagging on local disk before the copy. Generated NFO sidecars and a nightly playout rebuild keep the linear channels scheduled correctly.