Backup, recovery & agents at scale

JENKINS_HOME, restore drills, ephemeral agents.

Everything Jenkins knows lives in one directory: JENKINS_HOME. Job configs, build history, the credentials store, plugin state, fingerprints — all of it. That makes backup conceptually simple (protect that directory) and makes its loss catastrophic (lose it and you lose every pipeline and secret). A real backup strategy plus a tested restore is not optional for anything teams depend on.

# a JENKINS_HOME backup (stop or quiesce first for consistency)
$ tar czf jenkins-backup-$(date +%F).tgz -C /var/jenkins_home \
    --exclude='workspace/*' --exclude='caches/*'   # skip re-creatable data
# store it off the Jenkins host; the credentials store is in here, so treat it as sensitive

Test the restore, not just the backup

A backup you have never restored is a hope, not a plan. Periodically stand up a throwaway Jenkins from the backup (and your JCasC file) and confirm jobs, history, and credentials come back — a restore drill. The combination of JCasC (the configuration, in Git) plus a JENKINS_HOME backup (the state and secrets) is what lets you rebuild the instance quickly and confidently after a failure.

Ephemeral agents at scale

As build volume grows, static agents become a liability — they accumulate state between builds (a security risk) and sit idle costing money. The scalable, secure pattern is ephemeral agents: Jenkins provisions a fresh container or cloud VM per build (via the Docker or Kubernetes plugin), runs the build in it, and destroys it. Every build gets a clean, uncontaminated workspace, and there is no long-lived agent for an attacker to persist on.

# with the Kubernetes plugin, each build runs in a throwaway pod:
#   agent { kubernetes { yaml podTemplateYaml } }
# the pod is created for the build and deleted after — clean slate, no persistence