Self-Hosted DevOps Platform

What began as a handful of home-lab services had grown into a real estate: media servers, monitoring, internal tools and public websites spread across physical hosts, virtual machines and several Kubernetes clusters. Changes were made by hand, credentials lived in too many places, and internal services had no consistent naming or TLS.

The CI system itself needed to be dependable. Long-running jobs could hold build executors while waiting on other jobs, and an agent that rebooted could quietly stall every build behind it. The goal was an environment that behaves like production: reproducible, auditable and recoverable, with every change going through a pipeline.

The environment was rebuilt as a small platform with pipeline-driven operations. Proxmox hosts virtual machines and LXC containers, three on-premises K3s clusters separate application, utility and media workloads, and a single-node K3s cluster on a DigitalOcean droplet serves the public websites.

Jenkins is the system of record for change. Jobs are generated from Job DSL seed scripts, pipelines load their Jenkinsfiles from Git, and those pipelines run Ansible playbooks, Terraform plans and Kubernetes actions. AWS provides the shared services: ECR for images, Secrets Manager for credentials and S3 with DynamoDB for Terraform state. CoreDNS, an nginx reverse proxy with a wildcard certificate and a Cloudflare Tunnel give every internal service a consistent name, HTTPS, and remote access without opening inbound ports.

The platform spans 18 managed hosts, a two-node Proxmox cluster and four K3s clusters, with everything defined in version control.

• Jobs as code: Job DSL seed scripts generate the Jenkins jobs from lists of applications, and a shared pipeline library provides build notifications to every job.
• Ansible through Jenkins: a single parameterized job exposes 60+ playbooks (from a repository of 70+ playbooks and 19 roles) and runs them on a dedicated agent, so every production change is logged against a build.
• Terraform: reusable Proxmox VM and LXC modules, remote state in S3 with DynamoDB locking, and a Jenkins job with plan, saved-plan and apply stages.
• Executor-deadlock fix: pipelines that only wait on downstream builds were moved to agent none so they run on lightweight executors instead of holding build agents, and scheduled wrappers were given hard timeouts. The pattern was applied across eight pipelines.
• Agent recovery: a documented Script Console procedure frees SSH agents that hang after a host reboot, without restarting the controller.
• Split-horizon DNS and TLS: 80+ CoreDNS records and around 60 proxied services, secured by a Let's Encrypt wildcard certificate issued through DNS-01 validation against the DigitalOcean API.
• Registry credentials: scheduled jobs refresh ECR pull secrets across cluster namespaces well ahead of the 12-hour token expiry.
• Bounded CI storage: build-retention limits on generated and repository pipelines keep the controller's disk use predictable.