Jenkins
Jenkins is a tool I’ve run in production for years — leading controller and pipeline migrations, standing up dynamic Kubernetes agents, and being the one paged when a queue stalls at 5pm. No specifics about any particular employer’s infrastructure here; what follows is the set of general principles that actually hold up at scale, learned the way most operational knowledge is — by being on the hook for it.
Where it fits
Section titled “Where it fits”Jenkins is where I reach for CI proper — build, test, and parameterized builds — jobs that take inputs (a branch, a version, a target environment) and need a UI or API for humans and other systems to trigger them with those inputs filled in. A build that takes a branch, a version, a target, or a feature flag as input — triggered by a human from the UI or by another system via the API — is Jenkins doing what it’s best at; design the parameters deliberately (typed choices over free-text where possible) so a bad input fails fast instead of mid-pipeline. Deployment is a separate concern handled by ArgoCD, and pure scheduled/cron-style work tends to fit Concourse better.
Principles
Section titled “Principles”- Pipelines as code — declarative pipelines and shared libraries keep pipeline logic in source control, not the UI.
- Agent/controller lifecycle — agents are cattle, the controller is not.
- Dynamic Kubernetes agents — provisioning ephemeral agents from composable pod templates.
- Bounding growth — plugin sprawl and unbounded build history are the most common sources of a slow, fragile controller.
- Credential scoping — narrow credentials limit blast radius.
- Setup & Sizing — general Helm chart guidance, sized to team and job volume.
What tends to go wrong
Section titled “What tends to go wrong”Most Jenkins pain I’ve seen traces back to one of: unbounded growth (history, workspaces, plugins), UI-configured jobs with no equivalent in source control, or agents that were hand-tuned once and never reproducible again. The fixes are boring and consistent — which is exactly why they’re easy to skip under deadline pressure and expensive to skip for long.
See Performance Tuning for how these principles translate into concrete resource and retention settings.