Resilience: exceptions, retries, timeouts & cleanup

Exception hygiene, retry/backoff, timeouts, and context managers that always clean up.

Automation runs against systems that fail intermittently — networks blip, APIs rate-limit, disks fill. Resilient code expects this: it retries the things worth retrying, gives up on the things that are not, bounds every wait, and cleans up its resources no matter how it exits. The difference between a fragile script and a dependable one is almost entirely error handling.

Exception hygiene

Catch the narrowest exception that you can actually handle, and let everything else propagate. A bare except: hides bugs (including KeyboardInterrupt and MemoryError) and makes failures invisible. When you re-raise as a higher-level error, chain the cause with raise ... from err so the traceback keeps the original. Handle where you can do something useful; do not handle just to silence.

class FetchError(Exception): ...
 
def fetch(url: str) -> bytes:
    try:
        r = httpx.get(url, timeout=10)
        r.raise_for_status()
        return r.content
    except httpx.HTTPStatusError as e:
        # a 404 is not retryable; surface it as a domain error, keep the cause
        raise FetchError(f"{url} -> {e.response.status_code}") from e
    except httpx.TimeoutException as e:
        raise FetchError(f"{url} timed out") from e
    # connection errors, etc. propagate unchanged — the caller decides

Retries, backoff, and jitter

Retry only idempotent operations and only on transient failures (timeouts, 5xx, connection resets) — never on a 400 or an auth error, which will fail identically forever. Back off exponentially so you do not amplify an outage, add jitter so a fleet of clients does not retry in lockstep (the thundering herd), and cap both the number of attempts and the total time. tenacity encodes all of this declaratively.

from tenacity import (retry, stop_after_attempt, wait_exponential_jitter,
                       retry_if_exception_type)
 
@retry(
    retry=retry_if_exception_type((httpx.TimeoutException, httpx.ConnectError)),
    wait=wait_exponential_jitter(initial=0.5, max=20),
    stop=stop_after_attempt(5),
    reraise=True,
)
def get_with_retry(client, url):
    r = client.get(url, timeout=10)
    if r.status_code >= 500:
        raise httpx.ConnectError("server error, retryable")
    r.raise_for_status()          # 4xx: raised, NOT retried
    return r

Context managers guarantee cleanup

A resource acquired must be released on every path, including exceptions — that is what with is for. Files, locks, network sessions, temp directories, and subprocesses all support (or can be wrapped in) context managers so cleanup is automatic. Write your own with contextlib for anything that has a setup/teardown pair; ExitStack composes several so you never nest five withs or forget one.

from contextlib import contextmanager, ExitStack
import tempfile, shutil, os
 
@contextmanager
def workdir():
    d = tempfile.mkdtemp(prefix="tool.")
    try:
        yield d                    # hand the resource to the body
    finally:
        shutil.rmtree(d, ignore_errors=True)   # ALWAYS runs
 
with ExitStack() as stack:
    d = stack.enter_context(workdir())
    f = stack.enter_context(open(os.path.join(d, "out"), "w"))
    # both the file and the temp dir are cleaned up, in reverse order,
    # even if the next line raises
    do_work(f)