Signals, traps, timeouts & graceful shutdown

SIGTERM/SIGINT handling, cleanup, timeouts and job control.

A script that starts background work, writes temp files, or holds a lock must handle the ways it can be told to stop. Signals are how the OS and orchestrators (systemd, Kubernetes, CI runners) ask a process to shut down; ignoring them means half-written files, orphaned children, and stuck locks. Handling them well means your script cleans up and exits promptly whether it finishes, errors, or is killed.

The signals that matter and the EXIT trap

SIGTERM is the polite "please stop" that orchestrators send first (Kubernetes sends it, then waits the grace period, then SIGKILL). SIGINT is Ctrl-C. SIGHUP is a lost terminal or a reload request. You cannot trap SIGKILL or SIGSTOP — that is the point of them. The trick that keeps cleanup simple is to put teardown in one function on the EXIT trap, which runs no matter how the script leaves, and let the signal traps just record intent and exit.

#!/usr/bin/env bash
set -Eeuo pipefail
 
tmp="$(mktemp -d)"; child_pid=
 
cleanup() {
  local rc=$?
  [[ -n "$child_pid" ]] && kill "$child_pid" 2>/dev/null || true
  rm -rf "$tmp"
  exit "$rc"
}
trap cleanup EXIT                    # runs on ANY exit
trap 'echo "caught signal, shutting down" >&2; exit 143' TERM INT
 
long_running_thing & child_pid=$!
wait "$child_pid"

Timeouts so nothing hangs forever

Any command that talks to the network can hang, and a hung automation job is worse than a failed one because nothing alerts. Wrap external calls in timeout, which sends SIGTERM after the deadline and SIGKILL a bit later with -k. For a whole script, an alarm using a background sleep that signals the main shell gives you a hard ceiling. Always bound the wait.

# kill a stuck command: TERM at 30s, KILL 5s later if it ignores TERM
timeout -k 5 30 curl -fsS https://slow.example.com/health
 
# whole-script watchdog: self-destruct after 10 minutes
( sleep 600 && kill -TERM "$$" ) & watchdog=$!
trap 'kill "$watchdog" 2>/dev/null || true' EXIT
# ... main work ...

Graceful shutdown under an orchestrator

Under Kubernetes or systemd your script is PID 1 or close to it, and the platform expects it to react to SIGTERM within a grace period. Trap SIGTERM, stop accepting new work, let in-flight work drain up to a deadline, then exit with a conventional code (128 + signal number, so 143 for SIGTERM). If you exec a child as the real workload, make sure signals reach it rather than dying at the shell.