IT Checklists

Web Application Security Review

Mon, 01 Jan 0001 00:00:00 +0000

Most web applications are not broken into through exotic zero-days. They are broken into through a missing authorisation check on an object ID, a session that never expires, or a file upload that lands in a directory the web server will happily execute. This review walks the request path from the browser to the database and back, in the order an attacker would probe it.

info

Who runs this: an application security engineer together with a developer who knows the codebase. When: before the first public release, after any change to authentication or authorisation, and at least annually for anything internet-facing.

1. Authentication

Passwords are stored with a memory-hard hash — bcrypt, scrypt, or Argon2id with tuned parameters; SHA-256 with a salt is not adequate against modern GPU cracking.
Credential stuffing is throttled per account and per source — rate limit on the username as well as the IP, because attackers spread a single password across many accounts from many addresses.
Multi-factor authentication is available and enforced for privileged accounts — TOTP or WebAuthn; SMS is a fallback, not a control you rely on for admins.
Account enumeration is not possible through login, registration, or password reset — identical responses and identical timing whether or not the account exists.
Password reset tokens are single-use, random, and short-lived — bound to the account, invalidated on use, and expiring within an hour.
The reset flow does not trust the Host header — a reset link built from an attacker-controlled Host sends the token to the attacker.
Default and seeded credentials are removed from every environment — including the demo admin account someone added for a sales pitch.

2. Session management

Session cookies carry HttpOnly, Secure, and SameSite=Lax or Strict — HttpOnly blocks theft via XSS, Secure blocks it over plaintext, SameSite blunts CSRF.
The session identifier is rotated on login and on privilege change — otherwise an attacker who fixes a victim’s session ID before login inherits the authenticated session.
Sessions expire on both idle timeout and absolute lifetime — an idle timeout alone lets a stolen token live forever if it is used periodically.
Logout invalidates the session server-side — deleting the cookie in the browser does nothing to a token an attacker already copied.
JWTs, if used, pin the algorithm server-side and validate issuer, audience, and expiry — accepting the alg header from the token is how none and key-confusion attacks work.
There is a way to revoke a session or token before it expires — needed the moment a laptop is lost or an account is compromised.

3. Authorisation and access control

Every request is authorised server-side against the authenticated principal — hiding a button in the UI is not access control.
Object-level authorisation is checked on every object reference — the classic IDOR is GET /invoices/1042 returning someone else’s invoice because only authentication was checked.
Authorisation is enforced in a central, hard-to-bypass layer — a per-controller if statement will eventually be forgotten on a new endpoint.
The default for a new endpoint is deny — allow-listing what is public beats blocklisting what is protected.
Horizontal and vertical privilege escalation have been tested explicitly — log in as a low-privilege user and replay an admin request with their session.
Multi-tenant queries are scoped by tenant at the data layer — a missing WHERE tenant_id = ? is a cross-customer data breach, not a bug.
Mass assignment is prevented — bind request bodies to an explicit allow-list of fields, or a user updates their own role field.

report

Blocking: broken object-level authorisation is the single most commonly exploited web flaw and is invisible to most automated scanners. Test it by hand, with two accounts, before release.

4. Input handling and injection

All database access uses parameterised queries or a well-used ORM — string concatenation into SQL is still the fastest route to full data loss.
Dynamic query fragments such as sort columns are mapped through an allow-list — parameter binding cannot protect an identifier or an ORDER BY clause.
OS command execution avoids the shell and passes arguments as an array — and no user input reaches a command line at all where it can be avoided.
Templating engines are used in their auto-escaping mode with no raw interpolation of user data — server-side template injection escalates straight to code execution.
Deserialisation of untrusted data is avoided, or restricted to a safe format with a type allow-list — native object deserialisation is remote code execution waiting for input.
Input validation is positive — validate against expected type, length, format, and range rather than trying to filter known-bad strings.
XML parsers have external entity resolution disabled — XXE turns a document upload into a file-read and internal-network probe.

5. Output encoding and browser-side defences

Output is contextually encoded at the point of rendering — HTML body, attribute, JavaScript, URL, and CSS contexts each need different escaping.
Any use of a raw-HTML sink is reviewed and sanitised with a maintained library — innerHTML, dangerouslySetInnerHTML, and their equivalents are where DOM XSS lives.
A Content Security Policy is deployed without unsafe-inline or unsafe-eval — nonce or hash based, so an injected script has nowhere to execute.
State-changing requests require an anti-CSRF token or are restricted to same-site requests — SameSite cookies help but do not cover every browser or every flow.
CORS does not reflect arbitrary origins and does not combine a wildcard with credentials — reflecting Origin with Allow-Credentials: true is equivalent to no same-origin policy at all.
Security headers are set — HSTS with a long max-age, X-Content-Type-Options: nosniff, and a restrictive Referrer-Policy and Permissions-Policy.
Clickjacking is prevented — frame-ancestors in the CSP, with X-Frame-Options for legacy clients.

6. Business logic and abuse resistance

Server-side price, quantity, and discount values are authoritative — never trust an amount that came back from the client.
Workflow steps cannot be skipped or replayed — verify that jumping straight to the confirmation endpoint fails.
Race conditions on limited resources are handled with locking or atomic operations — parallel requests are how a single-use coupon gets redeemed fifty times.
Expensive endpoints are rate limited and quota bounded — search, export, and report generation are cheap to request and expensive to serve.
Server-side requests to user-supplied URLs are blocked or proxied through an allow-list — SSRF into cloud metadata endpoints is a standard path to credential theft.

7. Data protection and privacy

All traffic is HTTPS with modern TLS and HTTP redirected permanently — including internal service-to-service calls.
Sensitive data is not placed in URLs — query strings end up in access logs, referrer headers, and browser history.
Personal and sensitive fields are encrypted at rest where the threat model demands it — with keys held outside the database.
Error responses are generic and stack traces are never returned to the client — detailed errors map your framework, versions, and file layout for an attacker.
Caching headers prevent sensitive responses being stored by browsers or intermediaries.
Uploaded files are validated by content, stored outside the web root, and served with a fixed content type — and never with a user-controlled filename or extension.

8. Dependencies and configuration

A software bill of materials is generated and dependencies are scanned on every build — with a policy for how quickly a critical finding must be fixed.
Framework and server configurations are hardened for production — debug mode off, directory listing off, admin consoles unreachable from the internet.
Client-side third-party scripts are inventoried and justified — every tag manager script runs with full access to your DOM and cookies.
Secrets are loaded from a secret manager, not from the repository or the container image — verify with a history scan, not just a look at the current tree.
Security-relevant events are logged with enough context to investigate — authentication outcomes, authorisation failures, and privileged actions, with no credentials in the log.

Sign-off

Area	Reviewer	Date	Outcome
Authentication			Pass / Pass with actions / Fail
Session management			Pass / Pass with actions / Fail
Authorisation and access control			Pass / Pass with actions / Fail
Input handling and injection			Pass / Pass with actions / Fail
Output encoding and browser defences			Pass / Pass with actions / Fail
Business logic and abuse resistance			Pass / Pass with actions / Fail
Data protection and privacy			Pass / Pass with actions / Fail
Dependencies and configuration			Pass / Pass with actions / Fail

Every finding that is not a clean pass needs a severity, an owner, and a fix date agreed before release.

Cloud Security Posture

Mon, 01 Jan 0001 00:00:00 +0000

Cloud breaches rarely involve breaking the provider. They involve an over-permissive role, a storage bucket that was made public for a demo, or a long-lived access key committed to a repository. This checklist reviews the posture of a cloud account or subscription across identity, network, data, logging, and workload configuration. It is written to apply to AWS, Azure, and Google Cloud; substitute the local name for each concept as you go.

Security Code Review

Mon, 01 Jan 0001 00:00:00 +0000

A security code review is a normal code review with one extra question asked of every line: what happens if the input is hostile? It is cheaper than a penetration test and catches a different class of problem — the missing check, the unsafe default, the trust boundary crossed without anyone noticing. Use this list when reviewing a change that touches authentication, authorisation, data handling, cryptography, or anything that parses untrusted input.

Secrets Management

Mon, 01 Jan 0001 00:00:00 +0000

Secrets leak the same way every time: someone needed a credential in a hurry, put it somewhere convenient, and nobody ever moved it. This checklist covers the whole lifecycle of a secret — how it is created, where it lives, how it reaches the workload that needs it, how it is rotated, and how it is destroyed. Work through it per system rather than per organisation, because one exemplary service and one hardcoded database password in the next repository still adds up to a breach.

Security Incident Response

Mon, 01 Jan 0001 00:00:00 +0000

A security incident is an operational incident with lawyers, regulators, and an adversary who reacts to what you do. The decisions that matter most — whether to isolate a host, when to rotate credentials, who tells the regulator — are terrible decisions to make for the first time at 2am. This checklist covers both readiness and the live response, in the order events actually unfold: prepare, detect, contain, eradicate, recover, and learn.

Penetration Test Readiness

Mon, 01 Jan 0001 00:00:00 +0000

A penetration test is expensive per day and the days are easily wasted. Testers regularly spend the first two of five days waiting for credentials, discovering the environment does not match production, or re-finding issues an internal scanner already reported. This checklist gets the engagement ready so the testers spend their time on the things only a human can find: chained logic flaws, authorisation gaps, and abuse of the features you are proud of.

Production Readiness Review

Mon, 01 Jan 0001 00:00:00 +0000

A production readiness review (PRR) is the gate between “it works on staging” and “it carries customer traffic at 3am while nobody is watching”. Work through this before the first real request hits the service, and again whenever ownership changes hands.

info

Who runs this: the owning team, with one reviewer from outside the team. When: at least one week before launch, so findings can actually be fixed.

1. Ownership and documentation

A named team owns the service — not an individual, and the owner is recorded somewhere machine-readable (service catalogue, CODEOWNERS, or repo metadata).
The escalation path is written down — who gets paged first, who gets paged if they do not acknowledge, and who is the business decision-maker for an outage.
A runbook exists and has been read by someone who did not write it — start-up, shut-down, common failure modes, and how to check whether the service is actually healthy.
The architecture diagram matches reality — every dependency the service calls, and every caller it serves, is on the diagram.
All upstream and downstream dependencies are listed with their criticality — mark each as hard (service fails without it) or soft (degrades gracefully).

2. Reliability and failure behaviour

Every outbound call has a timeout — a default-infinite HTTP client is the single most common cause of cascading failure.
Retries use exponential backoff with jitter and a retry budget — naive retries turn a slow dependency into a self-inflicted denial of service.
Hard dependency failures degrade rather than crash — decide per dependency whether to fail open, fail closed, or serve stale data, and make it explicit in code.
The service starts cleanly from cold — no dependency on warm caches, in-memory state from a previous instance, or manual post-start steps.
Health checks distinguish liveness from readiness — liveness must not fail because a downstream dependency is down, or the orchestrator will restart-loop a healthy process.
Graceful shutdown is implemented — the process stops accepting new work, drains in-flight requests, and exits within the orchestrator’s termination grace period.
Single points of failure are identified and accepted in writing — a shared database, a single availability zone, or one licence server all count.

warning

Blocking: a service without timeouts on outbound calls should not be launched. Everything else on this list can carry a dated follow-up ticket; this one cannot.

3. Observability

The four golden signals are on a dashboard — traffic, error rate, latency (p50/p95/p99), and saturation of the constraining resource.
Logs are structured and include a correlation ID — free-text logs are unsearchable at the exact moment you need them.
No secrets, tokens, or personal data are written to logs — grep the codebase for logging of request bodies and auth headers specifically.
Distributed tracing is wired through the request path — trace context is propagated to every downstream call, not just the first hop.
Log retention is set deliberately — long enough to investigate a slow-burning bug, short enough to satisfy the data retention policy.
A synthetic probe exercises the critical user journey — dashboards go green when nobody is using a broken feature.

4. Alerting

Alerts fire on user-visible symptoms, not on causes — “checkout error rate above 2% for 5 minutes” beats “CPU above 80%”.
Every alert links to the runbook section that resolves it — an alert with no documented response is a notification, not an alert.
Alert thresholds have been tested by deliberately breaking something — in staging, with the on-call rotation watching.
Paging alerts are distinguishable from ticket-generating alerts — if everything pages, nothing pages.
Alert volume is estimated and is under a page or two per shift — model it against the last month of staging data if you have nothing else.

5. Capacity and performance

A load test has been run against production-like infrastructure — same instance sizes, same database tier, same network topology.
The breaking point is known — the request rate at which latency degrades past the SLO, and what fails first when you get there.
Expected launch traffic is documented with a peak-to-average ratio — including any marketing spike, batch job, or scheduled import.
Autoscaling limits are set and the maximum is affordable — check the maximum against the monthly budget, not just the technical limit.
Resource requests and limits are set from measured usage — not copied from another service’s manifest.
Rate limiting or load shedding protects the service from a bad client — including your own retrying clients.

6. Data and state

Backups are configured and a restore has actually been performed — an untested backup is a hypothesis.
The recovery time and recovery point objectives are written down and achievable — measure the restore, do not estimate it.
Schema migrations are backward compatible — the previous version of the application must run against the new schema for the duration of the rollout.
Data at rest and in transit is encrypted — including backups, snapshots, and any object storage bucket.
Personal data is classified and its retention period is enforced by a job, not by intention.

7. Security

The service authenticates its callers and authorises every request — network position is not authorisation.
Secrets come from a secret manager at runtime — not from environment variables baked into an image, and never from the repository.
Dependencies have been scanned and known critical vulnerabilities are resolved or accepted in writing.
The container runs as a non-root user with a read-only root filesystem where possible.
Network access is least-privilege — the service can reach only the dependencies it declared in section 1.
An audit log records security-relevant events — authentication, authorisation failures, and privileged actions.

8. Release and rollback

Deployment is fully automated and repeatable — the same command, run by anyone on the team, from a clean checkout.
Rollback has been rehearsed and is a single documented action — and it works even when the deployment pipeline is broken.
The rollout is progressive — canary, blue/green, or percentage-based, with automated abort on error-rate regression.
Feature flags gate risky behaviour and are removed on a schedule — a permanent flag is a permanent untested code path.
Rollback is safe with respect to data — if the new version wrote data the old version cannot read, rollback is a lie.

9. Cost

The expected monthly cost is estimated and has an owner.
Every resource is tagged for cost allocation — service, team, and environment at minimum.
A budget alert is configured at a threshold that gives you time to react.

Sign-off

Area	Reviewer	Date	Outcome
Reliability and failure behaviour			Pass / Pass with actions / Fail
Observability and alerting			Pass / Pass with actions / Fail
Capacity and performance			Pass / Pass with actions / Fail
Data, backup, and recovery			Pass / Pass with actions / Fail
Security			Pass / Pass with actions / Fail
Release and rollback			Pass / Pass with actions / Fail

Record every “Pass with actions” as a dated ticket with an owner before the launch is approved.

CI/CD Pipeline Review

Mon, 01 Jan 0001 00:00:00 +0000

A pipeline is production infrastructure. It holds credentials to every environment you own, it decides what code becomes a running artefact, and when it is wrong it is wrong for every service at once. Review it the way you would review a service that has write access to production, because that is exactly what it is.

info

Who runs this: the team that owns the pipeline, with a reviewer from platform or security. When: when a new pipeline is created, after any change to its credentials or permissions, and at least annually for pipelines that deploy to production.

1. Source control and change flow

The default branch is protected against direct pushes — including for administrators, or the protection is decorative.
Merging requires a passing pipeline and at least one approving review — and approvals are dismissed when new commits are pushed, otherwise a reviewer approves one diff and a different one merges.
CODEOWNERS covers the pipeline definition itself — a change to the deploy workflow deserves more scrutiny than a change to a README.
Force-push and branch deletion are disabled on protected branches — recovering the true history after a force-push to main is far harder than preventing it.
Commits are traceable to a person — signed commits or a verified identity, so a compromised token cannot quietly author a change attributed to someone else.
Every deployable commit is reachable from a tag or release record — you must be able to answer “what exactly is running?” without guessing.

2. Build reproducibility

A clean checkout on a fresh runner produces the same artefact — any dependence on a warm cache, a pre-installed tool, or a developer’s machine is a latent build break.
All dependencies are pinned by version and, where the ecosystem supports it, by digest or hash — a lockfile that resolves floating ranges at build time is not pinning.
The lockfile is committed and the build fails if it is out of date — npm ci, pip install --require-hashes, go mod verify and equivalents rather than a permissive install.
Base images and build tool versions are pinned rather than tracking latest — an upstream tag moving under you turns a green pipeline red for reasons unrelated to your change.
The build has no network access to unpinned sources at package time — pulling a script from a URL during the build makes the artefact a function of the internet’s mood.
Build outputs are written to a fresh workspace each run — leftover files from a previous build are a classic source of artefacts that cannot be recreated.

3. Pipeline credentials and permissions

The pipeline authenticates to cloud providers with short-lived OIDC federation, not a long-lived access key stored as a repository secret — a leaked static key is valid until someone notices; a federated token expires in minutes.
Each job requests the minimum token scope it needs — a test job does not need write permission to packages or deployments.
Secrets are scoped per environment — the staging deploy job must not be able to read production credentials.
Deployment credentials are held by a protected environment with required reviewers, not by the workflow file — otherwise any contributor who can edit a workflow can reach production.
Workflows triggered by forked pull requests cannot read secrets — the pull_request_target trigger and equivalents are the single most exploited CI misconfiguration.
Secrets are masked in logs and the masking has been tested — try echoing a secret through base64 in a scratch branch and confirm it is redacted.
Credential rotation is documented and has been performed at least once — an unrotatable credential is a permanent liability.

report

Blocking: a pipeline that exposes production credentials to workflows runnable from a fork is remotely exploitable by any user of the platform. Fix this before anything else on the list.

4. Third-party actions and runners

Every third-party action or plugin is pinned to a full commit SHA, not a tag — tags are mutable, so a pinned tag is an invitation to a supply-chain compromise.
The set of allowed actions is restricted by policy — an allowlist of verified publishers and your own organisation, rather than the whole marketplace.
Self-hosted runners for public repositories are ephemeral and isolated — a persistent runner shared with untrusted pull requests leaks the previous job’s state.
Runner images are patched and rebuilt on a schedule — a runner is a long-lived machine with credentials, and it ages like any other host.
Caches are keyed so that untrusted branches cannot poison a cache used by trusted builds — cache entries are executable inputs.
The pipeline pulls container images by digest, not by mutable tag — :latest in a deployment step means you cannot say what you shipped.

5. Testing and quality gates

Unit, integration, and contract tests run on every pull request and block merge on failure — a test suite that only runs after merge is a report, not a gate.
Flaky tests are quarantined with an owner and a deadline, not retried silently — automatic retries hide real race conditions until they reach production.
Database migrations are exercised against a realistic schema in CI — a migration that has only ever run on an empty table has not been tested.
Test data contains no production personal data — synthetic or masked fixtures only, since CI logs and artefacts are widely readable.
Total pipeline duration for a pull request is measured and kept short enough that people do not batch changes — slow pipelines cause large, risky merges.
A failing pipeline cannot be bypassed by a manual override without an audit record — emergency overrides need to exist and need to leave a trace.

6. Security scanning in the pipeline

Dependency scanning runs on every build and fails on new critical vulnerabilities — with an explicit, time-limited exception process rather than a permanently ignored report.
Static analysis runs on changed code and its findings block merge at an agreed severity — a scanner whose output nobody reads is theatre.
Secret scanning runs on the diff and on the full history — push protection stops the next leak, history scanning finds the one from last year.
Container images are scanned after build and before promotion — scanning only the Dockerfile misses everything the base image contributes.
Infrastructure-as-code templates are scanned for misconfiguration — public buckets and open security groups are cheaper to catch here than in a cloud audit.
Scanner findings have an owner and an SLA by severity — an unbounded backlog of findings is indistinguishable from no scanning.

7. Artefact management and provenance

Artefacts are built once and promoted through environments unchanged — rebuilding per environment means staging never tested what production runs.
Every artefact is immutable in the registry — tag overwrites must be disabled, or a rollback can silently deliver different bytes.
A software bill of materials is generated and stored with each artefact — when the next widely used library has a critical CVE, you need to answer “are we affected?” in minutes.
Build provenance is recorded and verifiable — an attestation binding the artefact digest to the source commit and the builder, following SLSA levels.
The deployment step verifies the artefact signature or attestation before rolling out — provenance you never check is metadata, not a control.
Registry retention and cleanup policies exist — untagged layers accumulate cost, and stale images get deployed by accident.

8. Deployment and rollback

Deployment to production is a distinct, separately authorised step — not a side effect of merging.
The pipeline can deploy a specific previous artefact version on demand — rollback must not require rebuilding from an old commit.
Concurrency control prevents two deployments to the same environment at once — overlapping rollouts produce a state nobody designed.
Rollouts are progressive with automated abort on error-rate or latency regression — canary or percentage-based, with the abort criteria defined before the deploy.
A break-glass deployment path exists and is documented — including how to deploy when the CI platform itself is down.
Every deployment writes an audit record — who, what artefact digest, which environment, and when.

9. Pipeline reliability and maintenance

Pipeline failures alert the owning team, not just the commit author — a broken shared pipeline blocks everyone.
Pipeline definitions are version controlled and reviewed like application code — no click-configured jobs that exist only in the CI platform’s database.
Success rate and duration are tracked over time — a slowly degrading pipeline is a slowly degrading release cadence.
The pipeline is restorable if the CI platform account is lost — configuration in the repository, secrets in a secret manager with a documented restore.
Unused workflows, runners, and credentials are removed on a schedule — dormant automation with live credentials is a favourite foothold.

Sign-off

Area	Reviewer	Date	Outcome
Source control and change flow			Pass / Pass with actions / Fail
Build reproducibility			Pass / Pass with actions / Fail
Pipeline credentials and permissions			Pass / Pass with actions / Fail
Third-party actions and runners			Pass / Pass with actions / Fail
Testing and quality gates			Pass / Pass with actions / Fail
Security scanning in the pipeline			Pass / Pass with actions / Fail
Artefact management and provenance			Pass / Pass with actions / Fail
Deployment and rollback			Pass / Pass with actions / Fail
Pipeline reliability and maintenance			Pass / Pass with actions / Fail

Record every “Pass with actions” as a dated ticket with a named owner before the pipeline is approved for production deployments.

Container Image Hardening

Mon, 01 Jan 0001 00:00:00 +0000

Most container images in production are far larger, far more privileged, and far older than anyone intended. Each extra package is attack surface you will be asked about after an incident, and each unpinned base tag is a change you did not review. Work through this before an image is promoted beyond a development environment, and again whenever the base image changes.

info

Who runs this: the team that owns the Dockerfile, with a reviewer from platform or security for images that run in production. When: at image creation, at any base image change, and on a recurring schedule for long-lived images.

1. Base image selection

The base image comes from a known, maintained publisher — an official image, a vendor-supported distribution image, or your own internal base, rather than an arbitrary registry account.
The base image is pinned by digest, not by tag — FROM node:22-slim moves under you; FROM node:22-slim@sha256:... is a decision you can review.
The smallest viable base is used — distroless, Alpine, or a slim variant, since packages you never installed cannot be vulnerable.
The base image’s support lifecycle outlasts the planned life of the service — building on a distribution release that reaches end of life in three months guarantees unpatchable CVEs.
A rebuild cadence is defined for picking up base image patches — an image built once and never rebuilt accumulates vulnerabilities even if your own code never changes.
The architecture set is explicit — multi-architecture images are built deliberately, so an arm64 node does not silently pull an emulated amd64 image.

2. Build hygiene

A multi-stage build separates compilation from runtime — compilers, package managers, and build caches must not ship to production.
The final stage copies only the artefacts it needs — copying the whole build workspace defeats the point of multi-stage.
A .dockerignore excludes .git, local environment files, credentials, and test fixtures — build context leakage is a routine cause of secrets inside images.
Package installation and cache cleanup happen in the same layer — a cleanup in a later layer removes files from the filesystem view but not from the image size or history.
Layer ordering puts rarely changing steps first — dependency installation before source copy, so a code change does not invalidate the whole cache.
No package manager remains usable at runtime where it can be removed — an image where an attacker can apt-get install their tooling is a much friendlier environment for them.
The image builds without --privileged or a mounted Docker socket — build steps requiring the host daemon are a privilege escalation path from CI into the host.

3. Secrets and build-time data

No secret is passed as a build argument — ARG values are recorded in image history and readable by anyone who can pull the image.
Private registry or repository credentials use build secret mounts — BuildKit secret mounts are not persisted into any layer.
The image history has been inspected for accidental secret inclusion — a file deleted in a later layer is still present in the earlier one and still extractable.
No .env, .npmrc, .netrc, or cloud credential file is present in the final image — verify by listing the filesystem, not by reading the Dockerfile.
Runtime configuration comes from the environment or a mounted secret, not baked into the image — an image containing production endpoints and keys cannot be reused or safely shared.
Private base images are pulled with credentials scoped to the pipeline — not a shared personal registry token.

4. Runtime user and filesystem

The image declares a non-root USER with a fixed numeric UID — a username alone does not satisfy orchestrator policies that check runAsNonRoot.
The UID is outside the host’s reserved range and does not collide with a privileged host user — a common convention is a UID above 10000.
The application does not need to write to the root filesystem — everything writable is an explicit volume or tmpfs, so the container can run with a read-only root.
No file in the image is setuid or setgid unless justified — strip these bits during the build and record any exception.
File ownership and permissions are set at build time, not by a start-up script running as root — an entrypoint that starts privileged and drops later is weaker than never being privileged.
The image contains no shell where the workload does not require one — a distroless runtime removes the most convenient post-exploitation tooling.

5. Image content and dependencies

The package list has been reviewed and unnecessary packages removed — debugging tools, editors, and network utilities are for a debug image, not the production one.
Application dependencies are installed from a lockfile with integrity hashes — so the image content is a function of the commit, not of the registry at build time.
A software bill of materials is generated at build and stored alongside the image — this is how you answer “are we affected?” during the next ecosystem-wide CVE.
The image is scanned for operating system and language-level vulnerabilities before promotion — many scanners default to OS packages only and miss the application layer entirely.
Critical and high findings are fixed or have a recorded, dated exception — with the compensating control written down, not implied.
Licence obligations of bundled dependencies have been checked — copyleft components in a distributed image create obligations that are expensive to discover late.

warning

Common mistake: scanning the image once at build time and never again. A CVE published tomorrow applies to the image running today, so scan images in the registry on a schedule and alert on newly discovered findings in what is deployed, not only in what is being built.

6. Image metadata and identity

The image carries OCI standard labels — source repository, revision, build timestamp, and version, so a running container can be traced to a commit.
Tags are immutable in the registry — overwriting a tag makes rollback and forensics unreliable.
Deployments reference images by digest — a tag tells you the intent, a digest tells you the bytes.
A meaningful versioning scheme is used, and latest is not deployed anywhere — latest is the fastest way to lose track of what is running.
The image is signed and the signature is verifiable — with the verification wired into admission control rather than performed by hand.
Build provenance attestations are attached to the image — recording which builder, which source commit, and which parameters produced it.

7. Runtime configuration in the image

ENTRYPOINT uses exec form so the application is PID 1 and receives signals — a shell form entrypoint swallows SIGTERM and turns every deploy into a hard kill after the grace period.
Zombie process reaping is handled — either the application reaps children correctly or a minimal init process is used.
EXPOSE documents the ports actually served, and nothing listens on an undocumented port — surprises here become surprises in network policy.
Application logs go to stdout and stderr — writing to files inside the container hides logs from the platform’s collector.
A health check endpoint exists in the application and is documented — the orchestrator’s probe configuration depends on it.
The default working directory and configuration paths are documented — operators should not have to read the Dockerfile to mount a config file.

8. Registry and distribution

Push access to production repositories is limited to the pipeline identity — no individual should be able to push an image that admission control will accept.
The registry requires authentication for pulls of private images and does not permit anonymous listing — public registry repositories routinely leak internal service names and configuration.
Retention policies remove untagged and superseded images — while retaining every digest that is still referenced by a running workload or a rollback plan.
Image promotion between environments copies the same digest — never a rebuild, or staging validated something production never ran.
Registry availability is considered in the failure model — a node that cannot pull cannot recover, so pull policies and caching are chosen deliberately.
Access to the registry is audited — pushes, deletes, and permission changes are logged and reviewed.

Sign-off

Area	Reviewer	Date	Outcome
Base image selection			Pass / Pass with actions / Fail
Build hygiene			Pass / Pass with actions / Fail
Secrets and build-time data			Pass / Pass with actions / Fail
Runtime user and filesystem			Pass / Pass with actions / Fail
Image content and dependencies			Pass / Pass with actions / Fail
Image metadata and identity			Pass / Pass with actions / Fail
Runtime configuration in the image			Pass / Pass with actions / Fail
Registry and distribution			Pass / Pass with actions / Fail

Record every “Pass with actions” as a dated ticket with a named owner before the image is promoted to production.

Kubernetes Deployment

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes will happily run a badly configured workload for months and then fail it all at once, during a node upgrade or a noisy-neighbour spike. Almost every “Kubernetes outage” is really a missing probe, an absent resource request, or a disruption budget nobody set. Work through this before a workload goes to a production cluster, and again after any significant change to its manifests.

info

Who runs this: the owning team, with a platform engineer reviewing cluster-level settings. When: before the first production apply, and whenever resources, probes, or scaling behaviour change.

1. Workload definition

The correct workload kind is used — Deployment for stateless replicas, StatefulSet where stable identity and storage matter, DaemonSet for per-node agents, Job/CronJob for finite work.
Images are referenced by digest and imagePullPolicy is deliberate — a mutable tag with IfNotPresent means different nodes can run different code indefinitely.
Replica count is at least two for anything that serves traffic — a single replica means every node drain is an outage.
Labels follow a consistent scheme including app, component, version, and owner — selectors, dashboards, network policies, and cost allocation all depend on them.
The pod template has no mutable configuration embedded in it — configuration comes from ConfigMap or Secret references, so a config change is reviewable on its own.
A change to a ConfigMap triggers a rollout — either via a checksum annotation on the pod template or immutable, versioned config objects, otherwise pods keep the old values until something unrelated restarts them.
The manifest is version controlled and applied by automation — nothing reaches production through kubectl edit on someone’s laptop.

2. Resource requests and limits

Every container sets CPU and memory requests based on measured usage — requests drive scheduling, and an unset request means the scheduler is guessing.
Memory limits are set and equal to or close to the request for latency-sensitive workloads — memory is incompressible, so exceeding a limit means an immediate OOMKilled, not throttling.
CPU limits are applied only where you intend throttling — an aggressive CPU limit causes latency spikes through CFS throttling even while the node has idle capacity.
The resulting quality-of-service class is understood — BestEffort pods are evicted first under node pressure, which is rarely what you want for a production service.
Resource values have been revisited against real production usage — copied-from-another-service values are the most common source of both waste and eviction.
Namespace ResourceQuota and LimitRange allow the workload’s peak, including during a rollout — a rollout temporarily needs headroom for surge pods.
Requests are reconciled against the cluster’s node sizes — a pod requesting more than any node can offer stays Pending forever with an unhelpful message.

3. Health probes and lifecycle

Readiness probes reflect the ability to serve a real request — not a static 200 from a handler that ignores dependency state.
Liveness probes do not check downstream dependencies — a shared dependency outage otherwise restart-loops every healthy pod in the fleet and turns a partial failure into a total one.
A startup probe covers slow initialisation — otherwise the liveness probe kills a container that was simply still warming up.
Probe timeouts, periods, and failure thresholds are tuned to real timings — the defaults are frequently too aggressive for JVM and heavy framework start-ups.
terminationGracePeriodSeconds exceeds the longest in-flight request plus drain time — otherwise every deploy severs live connections.
A preStop hook gives the service mesh or load balancer time to deregister — endpoint removal and container termination are concurrent, so a short sleep prevents requests being routed to a shutting-down pod.
The application handles SIGTERM and stops accepting new work while draining — verify by deleting a pod under load and watching for errors.

4. Scheduling and availability

A PodDisruptionBudget protects the workload during node drains — without one, a cluster upgrade can evict every replica simultaneously.
The disruption budget cannot deadlock the cluster — minAvailable equal to the replica count blocks all voluntary evictions and stalls node maintenance.
Pod anti-affinity or topology spread constraints distribute replicas across nodes and zones — two replicas on the same node is one node failure away from an outage.
Tolerations and node selectors are minimal and documented — a broad toleration can land a workload on a node pool it was never meant to touch.
Priority classes are set deliberately for critical workloads — so that under node pressure the right things are evicted.
Behaviour during a cluster autoscaler scale-down has been considered — pods with local storage or restrictive budgets can block node removal indefinitely.

warning

Blocking: a production workload with no readiness probe and no PodDisruptionBudget will drop traffic during every routine node upgrade. Neither is optional for a service that carries user requests.

5. Configuration and secrets

Secrets come from a secret manager through an operator or CSI driver, or are encrypted in git via a sealed-secrets style tool — plain base64 in a manifest is encoding, not encryption.
Etcd encryption at rest is enabled on the cluster for secret resources — confirm with the platform team rather than assuming.
Secrets are mounted as files rather than environment variables where practical — environment variables leak through crash dumps, child processes, and debug endpoints.
Secret rotation does not require a manual redeploy — either the application reloads mounted files or rotation triggers a controlled rollout.
Config differences between environments live in an overlay or values file, not in duplicated manifests — divergence between copies is how staging stops predicting production.
No service account token is mounted unless the pod calls the Kubernetes API — set automountServiceAccountToken: false by default.

6. Pod and container security

The pod security context sets runAsNonRoot, a numeric runAsUser, and fsGroup where volumes are written — and the image actually supports it.
allowPrivilegeEscalation is false and all Linux capabilities are dropped, adding back only what is needed — most workloads need none.
The root filesystem is read-only with explicit emptyDir mounts for writable paths — this removes the simplest persistence mechanism for an attacker.
A seccomp profile is applied — RuntimeDefault at minimum, which is not the default in older clusters.
No host namespaces, host paths, or privileged containers are used — each of these is effectively a route to the node, and any exception needs written approval.
The namespace enforces Pod Security Admission at the restricted level where possible — a policy that only warns will be ignored.
The pod’s service account has a least-privilege Role binding — never cluster-admin, and never a wildcard verb on a wildcard resource.

7. Networking and traffic

A default-deny NetworkPolicy exists in the namespace, with explicit allow rules — without a default deny, every pod can reach every other pod in the cluster.
Egress is restricted to the dependencies the service actually needs — including DNS, which is the rule people forget and then spend an hour debugging.
The Service selector matches the pod labels and resolves to the expected endpoints — verify with kubectl get endpoints, since a typo produces a silent black hole.
Ingress terminates TLS with a certificate that renews automatically — and the renewal has been observed to work at least once.
Timeouts and retry behaviour at the ingress or mesh layer are aligned with the application’s own timeouts — mismatched layers turn one slow request into an amplified retry storm.
Session requirements are explicit — if the application needs sticky sessions or long-lived connections, the load balancing configuration reflects that.

8. Storage and state

PersistentVolumeClaim storage classes are chosen deliberately, including the reclaim policy — a Delete policy on production data is one kubectl delete from permanent loss.
Access modes match the topology — a ReadWriteOnce volume ties the pod to a single node and constrains rolling updates.
Volume expansion is supported and the procedure has been tested — a full volume at 2am is not the moment to discover the storage class forbids resizing.
Backups of persistent volumes are configured and a restore has been performed — snapshots that have never been restored are untested.
StatefulSet scale-down behaviour is understood — orphaned volumes remain and cost money, or are reclaimed and destroy data, depending on configuration.
emptyDir usage is bounded with a size limit — an unbounded emptyDir can fill the node disk and evict unrelated pods.

9. Rollout, observability, and operations

The rolling update strategy sets maxUnavailable and maxSurge deliberately — and the cluster has capacity for the surge.
A failing rollout stops rather than replacing every healthy pod — progressDeadlineSeconds is set and the rollout status is checked by the pipeline.
Rollback to the previous revision has been rehearsed — and the revision history limit retains enough revisions to do it.
Metrics, logs, and traces from the pod reach the platform’s collectors — confirmed by looking at a dashboard, not by reading configuration.
Horizontal autoscaling targets a metric that actually reflects load — CPU is a poor proxy for an IO-bound service, and scaling on the wrong signal amplifies incidents.
Autoscaler minimum and maximum replicas are set, and the maximum is affordable and schedulable.
Alerts exist for crash-looping pods, pending pods, and failed rollouts — these are the platform-level symptoms that precede user-visible failure.

Sign-off

Area	Reviewer	Date	Outcome
Workload definition			Pass / Pass with actions / Fail
Resource requests and limits			Pass / Pass with actions / Fail
Health probes and lifecycle			Pass / Pass with actions / Fail
Scheduling and availability			Pass / Pass with actions / Fail
Configuration and secrets			Pass / Pass with actions / Fail
Pod and container security			Pass / Pass with actions / Fail
Networking and traffic			Pass / Pass with actions / Fail
Storage and state			Pass / Pass with actions / Fail
Rollout, observability, and operations			Pass / Pass with actions / Fail

Record every “Pass with actions” as a dated ticket with a named owner before the workload is approved for production traffic.

Infrastructure as Code Review

Mon, 01 Jan 0001 00:00:00 +0000

Infrastructure as code turns a cloud console mistake into a reviewable diff, but only if the review is real. The dangerous changes are rarely the ones that create resources; they are the ones that quietly destroy and recreate a database, widen a security group, or drift a state file away from reality. Use this when reviewing an infrastructure change and when assessing whether a repository’s practices are sound.

info

Who runs this: the engineer proposing the change plus a reviewer who did not write it. When: on every pull request that touches infrastructure code, with the module and state sections reviewed quarterly.

1. Repository structure and module design

Environments are separated by directory or workspace with their own state, not by a single configuration with a conditional — a shared configuration means a staging change can plan destructive edits to production.
Reusable modules are versioned and consumed by version constraint, not by a branch reference — a module sourced from main changes under every consumer without a diff.
Module inputs are typed, validated, and documented — an untyped map input is where undocumented behaviour accumulates.
Modules do not hard-code account IDs, regions, or environment names — these belong in the calling configuration or the change is not reusable.
Provider and language versions are pinned with a lockfile committed to the repository — a provider minor release can change default behaviour and produce an unexpected plan.
The repository has an owner and CODEOWNERS covers production configuration — infrastructure code without an owner is applied by whoever is desperate enough.

2. State management

State is stored remotely with versioning enabled — local state on a laptop is a single point of failure for the entire environment.
State locking is enabled and has been observed to work — two concurrent applies against one state produce corruption that is painful to unwind.
The state backend is encrypted at rest and access is restricted to the automation identity and a small break-glass group — state files contain secrets in plain text, including generated passwords and keys.
State backups exist and a restore from a previous state version has been rehearsed — recovering from a bad terraform state rm depends on this.
State is split so that a blast radius is bounded — one monolithic state for an entire organisation makes every change a high-risk change and every plan slow.
Manual state manipulation is documented and audited — state rm, import, and taint are legitimate tools that also silently rewrite reality.

3. The plan output

The plan has been read line by line, not just the summary counts — the resource count tells you nothing about whether the right resources are changing.
Every destroy and every replace in the plan is justified in the pull request description — a forced replacement of a database, a load balancer, or a key is the classic accidental outage.
The cause of each replacement is understood — which attribute forces new, and whether a lifecycle block or a change of approach avoids it.
No unexplained changes appear that the diff does not account for — unexplained drift means someone changed something by hand, and applying will revert it.
The plan is generated against the same state and variables that will be applied — a plan file is saved and applied, rather than re-planning at apply time.
Data source lookups resolve to what you expect — a filter that matches the wrong AMI or the wrong subnet produces a plan that looks harmless.

report

Blocking: never approve a plan containing a resource replacement you cannot explain. Replacement of stateful resources — databases, volumes, certificates, load balancers with static addresses — is data loss or an outage, and it is the most common cause of self-inflicted infrastructure incidents.

4. Secrets and sensitive data

No credential, key, or certificate is committed in a .tf or .tfvars file — including in examples and test fixtures.
Secrets are read at apply time from a secret manager, not passed as plain variables — and the reference, not the value, lives in the repository.
Outputs containing sensitive values are marked sensitive — otherwise they are printed in logs and stored in plan artefacts.
Generated passwords are written directly to a secret manager rather than surfaced as outputs — anything in state or output is readable by anyone with state access.
Plan artefacts are treated as sensitive and have a short retention — a saved plan can contain the values being written.
Secret scanning runs on the repository history, not only on new commits — infrastructure repositories accumulate old test credentials.

5. Security and compliance of the resources

A policy-as-code check runs in the pipeline and blocks on violations — encryption at rest, no public object storage, no unrestricted ingress, mandatory tags.
No security group, firewall rule, or bucket policy allows 0.0.0.0/0 except where explicitly documented and approved — and never for administrative ports.
IAM policies are least-privilege with no wildcard action on a wildcard resource — an over-broad role created by IaC is durable in a way an ad-hoc one is not.
Encryption at rest is enabled for every storage, database, and queue resource, with the key management model chosen deliberately.
Logging and audit trails are enabled on the resources being created — a resource created without logging is invisible during an incident.
Public exposure of any new endpoint is intentional and reviewed — load balancers, managed database public accessibility, and function URLs each default differently.
Deletion protection is enabled on stateful production resources — it is the last defence against a mistaken destroy.

6. Change safety and reversibility

The rollback path is described in the pull request — reverting the commit is not always sufficient when data or DNS is involved.
prevent_destroy or the equivalent guard is set on irreplaceable resources — databases, root DNS zones, and key management keys.
Changes are applied to a non-production environment first and the result observed — an environment that never receives the change first is not a testing environment.
The change is small enough to reason about — combining a provider upgrade, a refactor, and a functional change in one pull request makes the plan unreadable.
Resource moves use a moved block or a documented state move rather than a destroy and recreate — renaming a resource in code otherwise destroys the real one.
Dependencies between changes are sequenced explicitly — if a network change must land before a workload change, say so and order the merges.
Timing is considered — infrastructure changes touching networking, DNS, or certificates should not land immediately before a peak period or a holiday.

7. Automation and access control

Apply runs in a pipeline with a federated short-lived identity, not from a workstation with a long-lived key — and the pipeline identity is the only one with production write access under normal conditions.
Production apply requires an explicit approval separate from the code review — plan on pull request, apply on merge with a gate.
Direct human write access to production cloud accounts is break-glass only, time-limited, and alerted on — otherwise drift is inevitable.
Drift detection runs on a schedule and reports differences between code and reality — drift discovered during an incident is drift discovered too late.
Formatting, validation, and linting run automatically — so reviews discuss behaviour rather than whitespace.
The pipeline fails closed — an error in the plan step must not proceed to apply.

8. Documentation and operability

The repository README explains how to run a plan, where state lives, and who to ask — onboarding cost is a reliability property.
Every resource carries required tags — owner, environment, service, and cost centre, enforced by policy rather than by convention.
Cost impact of the change is estimated for anything that adds significant resources — instance families, provisioned throughput, and inter-zone traffic in particular.
Architecture documentation is updated in the same change — a diagram that lags the code is worse than no diagram.
Deprecated resources and modules are removed rather than left commented out — commented-out infrastructure is ambiguous during an incident.
The change is recorded in the change management system where one applies — with the plan output attached as evidence.

Sign-off

Area	Reviewer	Date	Outcome
Repository structure and module design			Pass / Pass with actions / Fail
State management			Pass / Pass with actions / Fail
The plan output			Pass / Pass with actions / Fail
Secrets and sensitive data			Pass / Pass with actions / Fail
Security and compliance of the resources			Pass / Pass with actions / Fail
Change safety and reversibility			Pass / Pass with actions / Fail
Automation and access control			Pass / Pass with actions / Fail
Documentation and operability			Pass / Pass with actions / Fail

Record every “Pass with actions” as a dated ticket with a named owner before the change is applied to production.

Release Day Runbook

Mon, 01 Jan 0001 00:00:00 +0000

Release day goes badly when decisions that should have been made yesterday are made under time pressure with an audience watching. This runbook is ordered by time rather than by topic: work down it from the day before the release to the point where the release is declared done. Assign a single release conductor who owns the sequence and the go/no-go call.

info

Who runs this: a named release conductor, with the owning engineers and the on-call responder present. When: starting the day before the deploy window and continuing through the soak period.

1. T-1 day: scope and readiness

The release contents are frozen and the change list is published — every merged commit, with the pull requests and tickets it closes, so nobody discovers an unexpected change mid-incident.
Every change in the release has been through code review and the pipeline is green on the release commit — a green build from an earlier commit does not count.
The exact artefact digest to be deployed is recorded — not a tag, so the thing you tested and the thing you ship are provably identical.
The release has been running in a staging environment with production-like data and traffic for long enough to expose slow failures — memory leaks and connection pool exhaustion do not appear in a ten-minute smoke test.
Risky changes are behind feature flags that default to off — this converts a deploy decision into a separate, reversible enablement decision.
Database migrations are backward compatible and have been timed against a production-sized dataset — a migration that locks a large table for minutes is an outage regardless of how correct it is.
Any dependency on another team’s release is confirmed in writing, including its ordering.

2. T-1 day: communication and logistics

The deploy window is agreed and published — with the timezone stated explicitly and confirmed against any change freeze.
The window avoids peak traffic, the end of a shift, and the hours before a weekend or holiday — deploying at 5pm on a Friday leaves nobody to watch the soak.
Named people are assigned to each role — release conductor, deployer, verifier, and the on-call responder who will own anything that goes wrong afterwards.
Everyone in those roles has confirmed availability for the deploy window plus the soak period.
Support, customer-facing, and dependent teams have been notified with what user-visible change to expect — including anything they should not raise as a bug.
The incident channel and bridge are created in advance — creating communication channels during an incident wastes the first ten minutes.
Anything requiring a maintenance window or a status page notice has been scheduled and drafted.

3. T-1 day: rollback preparation

The rollback procedure is written down as concrete commands, not as a description — the person running it may not be the person who wrote it.
The previous artefact version and its digest are recorded and confirmed to still exist in the registry — retention policies have deleted rollback targets before.
Rollback has been rehearsed in staging within the last release cycle — an unrehearsed rollback is a hypothesis.
The data implications of a rollback are understood — if the new version writes data or schema the old version cannot read, rollback is not actually available and you need a forward-fix plan instead.
The rollback trigger conditions are written down in advance — specific thresholds such as error rate above a stated value for a stated duration, not a judgement call made while stressed.
The maximum acceptable time to decide on rollback is agreed — typically minutes, and stated before anyone is emotionally invested in the release.
A feature flag kill switch is tested if the release relies on one — verify it takes effect without a redeploy.

4. Go/no-go, immediately before the window

The pipeline is still green and no new commits have landed on the release branch since the freeze.
No active incident is in progress in this service or its critical dependencies — check the incident tracker rather than assuming.
Dependent services and infrastructure are healthy — a deploy into a degraded platform makes attribution impossible when something breaks.
No conflicting change is being applied at the same time — infrastructure, DNS, certificate, or database maintenance from another team.
Current error rate, latency, and traffic levels are captured as the pre-deploy baseline — you cannot judge a regression without the number from before.
All named roles are present and acknowledge the go decision explicitly — silence is not consent.
The conductor states the abort criteria out loud before starting — so everyone is holding the same decision rule.

warning

No-go conditions: an active incident anywhere in the critical dependency chain, a rollback path that has not been verified, or a key person unavailable for the soak period. Any one of these is enough to postpone. Postponing costs a day; a bad release with nobody watching costs far more.

5. Deploy window: execution

Every step is announced in the release channel as it starts and as it completes — with timestamps, because the timeline is what you will reconstruct later.
Backward-compatible schema migrations run and complete before the application deploy — expand first, contract in a later release.
Migration duration and lock behaviour are watched live — abort criteria for the migration are separate from those for the deploy.
The deploy uses the recorded artefact digest through the normal automated pipeline — no manual steps, no local builds, no shortcuts because it is a special release.
The rollout is progressive and paused at the first stage for verification — canary, one instance, or a small traffic percentage before proceeding.
Canary health is compared against the rest of the fleet, not against the baseline alone — a fleet-wide change in traffic pattern otherwise looks like a canary regression.
Deploy progress is watched actively, not started and left — the person who ran the command stays until the rollout completes.
If anything unexpected occurs, the rollout is paused rather than pushed through — a paused rollout is cheap; a completed bad rollout is not.

6. Immediately after rollout: verification

The running version is confirmed on every instance or replica — a partially completed rollout serving two versions is a real and frequently missed state.
The critical user journey is exercised end to end by a human, not only by a synthetic probe — log in, complete the primary action, confirm the result persisted.
Error rate, latency percentiles, and traffic are compared against the pre-deploy baseline — looking at absolute values without the baseline hides small but real regressions.
Application logs are checked for new error signatures — a new stack trace at low volume is often the first sign of a problem that grows with traffic.
Dependency health is checked from the caller’s perspective — connection pool saturation, downstream error rates, and queue depth.
Background jobs, scheduled tasks, and consumers are confirmed to be running and keeping up — these fail quietly because no user is waiting on them.
Any feature flags intended to be enabled with this release are turned on one at a time, with verification between each.
Cache and configuration invalidation has taken effect where required — including content delivery network caches for front-end changes.

7. Soak period: monitoring

A defined soak duration is agreed before the release is declared done — long enough to cover at least one full traffic cycle for the service, and never less than the time it takes for slow failure modes to appear.
A named person is actively watching dashboards for the soak, not relying on alerts alone — alerts catch what you predicted; a human catches what you did not.
Resource saturation trends are watched — memory growth, file descriptors, thread counts, and connection pools reveal leaks that error rates do not.
Latency at the tail is watched, not just the average — p99 regressions affect the users most likely to complain publicly.
Business metrics are watched alongside technical ones — sign-ups, orders, or messages sent, because a technically healthy release can still break the product.
Support and customer-facing channels are monitored for reports that dashboards will not show — a broken payment flow for one card type is invisible in aggregate.
The on-call responder is briefed on what changed and what to do about it before the conductor stands down.

8. Rollback decision

The pre-agreed trigger conditions are applied as written, without renegotiation — the whole purpose of setting them in advance is to remove judgement under pressure.
Rollback is the default response to an unexplained regression — diagnose after service is restored, not before.
The decision is made within the agreed time limit — an hour spent debugging a live regression is an hour of user impact you chose to accept.
If rollback is not possible because of data or schema changes, the forward-fix path is executed with the same discipline — including its own verification and abort criteria.
Rollback execution is verified the same way the deploy was — version confirmed everywhere, critical journey exercised, metrics compared to baseline.
The decision, its reasoning, and its timeline are recorded in the release channel as they happen.
If rolling back, feature flags enabled during the release are also reverted — a flag left on against an old binary is an untested combination.

9. Release closure

The release is explicitly declared complete or rolled back in the release channel — an ambiguous ending leaves the on-call responder unsure of the current state.
The deployed version, artefact digest, and timestamp are recorded in the change log or change management system.
Stakeholders and support teams are notified of the outcome — including anything that behaves differently than announced.
Temporary measures are reverted — paused autoscaling, silenced alerts, extended timeouts, and any maintenance page.
Feature flags that are now permanent are scheduled for removal — a flag that outlives its purpose is a permanently untested code path.
Anything that went wrong, including near misses, is recorded for the retrospective — a release that worked out despite a scare still contains the lesson.
Improvements to this runbook are made while the details are fresh — the best time to fix a runbook is the day you used it.

Sign-off

Area	Reviewer	Date	Outcome
T-1 day: scope and readiness			Pass / Pass with actions / Fail
T-1 day: communication and logistics			Pass / Pass with actions / Fail
T-1 day: rollback preparation			Pass / Pass with actions / Fail
Go/no-go decision			Pass / Pass with actions / Fail
Deploy window: execution			Pass / Pass with actions / Fail
Post-rollout verification			Pass / Pass with actions / Fail
Soak period monitoring			Pass / Pass with actions / Fail
Rollback decision			Pass / Pass with actions / Fail
Release closure			Pass / Pass with actions / Fail

Record every “Pass with actions” as a dated ticket with a named owner, and treat any unresolved item in the go/no-go or rollback preparation sections as a no-go rather than a follow-up.

Cloud Landing Zone Setup

Mon, 01 Jan 0001 00:00:00 +0000

A landing zone is the part of your cloud estate that nobody wants to rebuild later. Account boundaries, identity, network address space, and the audit trail are all decisions that become progressively more expensive to change once real workloads depend on them. This checklist is written primarily against AWS, with the Azure and Google Cloud equivalents called out where the concept maps cleanly, and assumes you are building the foundation before the first production workload rather than retrofitting one.

Cloud Cost Optimisation

Mon, 01 Jan 0001 00:00:00 +0000

Most cloud cost programmes fail in the same way: someone is asked to cut 30%, they turn off a few instances, and six months later spend is back where it started. Durable savings come from attribution first, then waste removal, then rightsizing, then commitments — in that order, because buying a three-year commitment for an oversized fleet locks in the waste. Work through this in sequence rather than jumping to the discount instruments.

Cloud Migration Cutover

Mon, 01 Jan 0001 00:00:00 +0000

Migration projects rarely fail at the technology; they fail in the hour when traffic actually moves. This checklist is ordered as a timeline rather than by topic — assessment, dependency mapping, target build, rehearsal, freeze, the cutover window itself, validation, and rollback — so it can be worked through as the plan takes shape and then read top to bottom on the day. Adapt the timings to your change window, but do not reorder the phases.

Serverless Readiness

Mon, 01 Jan 0001 00:00:00 +0000

Serverless removes the servers, not the distributed systems problems. What it changes is which problems bite you: concurrency limits instead of instance counts, at-least-once delivery instead of a request-response call, and a cost model where an infinite retry loop bills by the millisecond. This checklist covers function-as-a-service workloads on AWS Lambda, Azure Functions, and Google Cloud Run functions, along with the event sources that trigger them.

info

Who runs this: the owning engineering team, with a reviewer who has operated an event-driven system before. When: before the workload takes production traffic, and again after any change to its event sources or concurrency configuration.

1. Workload fit

The workload’s execution profile fits within the platform’s timeout and payload limits — with headroom, because a job that takes 12 minutes against a 15-minute ceiling will time out the first time a dependency is slow.
Long-running or steady high-throughput work has been compared against containers on cost and latency — serverless wins on spiky and low-duty-cycle workloads; a service at constant high utilisation is usually cheaper and faster on provisioned compute.
The function does one thing — a single function fronting a dozen routes with a switch statement inherits the union of every route’s permissions, memory profile, and blast radius.
Orchestration of multi-step workflows uses a workflow service, not chained function invocations — Step Functions, Durable Functions, or Workflows give you state, retries, and visibility that a function calling a function does not.
State lives outside the function — anything cached in the execution environment survives only until the platform recycles it, which it will do without warning and at the worst time.
Dependencies that hold connection pools have a strategy — a relational database behind a function that scales to a thousand concurrent executions needs a proxy or a data API, or it will exhaust connections.

2. Configuration and packaging

Memory is tuned by measurement, not left at the default — CPU is allocated proportionally to memory, so more memory often reduces both duration and total cost; run a power-tuning sweep rather than guessing.
Timeout is set to a realistic ceiling per function, not the platform maximum — a long timeout means a stuck invocation bills for minutes and holds a concurrency slot the whole time.
The deployment package excludes development dependencies, tests, and the provider SDK where the runtime supplies it — package size directly affects cold start.
Shared code is in a layer or an internal package with a pinned version — a layer that changes underneath deployed functions is an untracked production change.
The runtime version is current and there is a plan for its deprecation date — providers force-migrate deprecated runtimes on a published schedule, and being surprised by it is avoidable.
Configuration comes from environment variables or a parameter store, and secrets from a secret manager — with the fetch cached outside the handler so it does not run on every invocation.
Ephemeral storage size and usage are known — the writable temporary directory is shared across invocations in the same environment and does not empty itself.

3. Cold starts and latency

Cold start latency has been measured, not assumed — measure the p99 of the initialisation phase separately from execution, because averages hide it entirely.
Initialisation work happens outside the handler — SDK clients, database connections, and configuration loading belong in the module scope so they are reused across warm invocations.
The cold start contribution to the user-facing latency SLO is quantified — for an asynchronous consumer it may be irrelevant; for a synchronous API path it may be the dominant term.
Provisioned or minimum-instance concurrency is configured where cold start would breach the SLO — and its cost is understood, since it bills whether or not it is used.
Functions in a VPC have been checked for the network initialisation penalty and only use a VPC when they need private resources.
Ahead-of-time compilation or a lighter runtime has been considered for latency-critical paths — startup-optimised runtimes and native images cut initialisation by an order of magnitude for JVM and .NET workloads.

4. Concurrency, scaling, and downstream protection

The account-level concurrency limit is known and the headroom across all functions is calculated — concurrency is a shared account quota, so one runaway function can starve every other function in the account.
Reserved concurrency is set on critical functions to guarantee capacity — and on risky ones to cap their blast radius.
Downstream capacity has been checked against the function’s maximum concurrency — a function scaling to 1,000 concurrent executions against a database that accepts 100 connections is a self-inflicted outage.
Burst scaling behaviour is understood for the region — platforms scale in bursts then at a rate limit, so a traffic spike produces throttling before it produces capacity.
Throttling is monitored and alerted on — a throttled synchronous invocation is a user-facing error; a throttled asynchronous one is a delayed or eventually dropped event.
Batch size and batching window for stream and queue sources are tuned deliberately — larger batches improve throughput and cost but increase the amount of work lost or retried on a single failure.
A load test has driven the function to its concurrency ceiling — the interesting failures all happen at the limit, not below it.

5. Event sources, delivery, and idempotency

Delivery semantics are known per event source and written down — at-least-once for most queue and stream sources means duplicate invocations are normal operation, not an incident.
Every handler with a side effect is idempotent — deduplicate on an event identifier in a store with a TTL, or make the write naturally idempotent; retries will otherwise double-charge a customer.
Ordering assumptions are explicit — only partitioned stream sources preserve order, and only within a partition; standard queues do not.
Partition or shard key choice distributes load evenly — a hot key serialises an entire workload behind one consumer regardless of how much the platform scales.
Event schemas are versioned and consumers tolerate unknown fields — a producer adding a field should never break a consumer.
Queue visibility timeout is at least the function timeout, usually a multiple of it — a shorter visibility timeout means the message is redelivered while the first invocation is still processing it.
Stream consumers handle a poison record — without bisect-on-error or a failure destination, one unparseable record blocks its shard until the retention period expires.
Fan-out patterns are checked for accidental recursion — a function writing to the bucket that triggers it is the classic infinite loop, and it bills continuously until someone notices.

report

Blocking: a function that writes to its own trigger source without a guard is a recursive invocation loop. Verify the write path cannot re-trigger the function, and set reserved concurrency as a hard ceiling before deploying.

6. Failure handling

Every asynchronous invocation path has a dead letter queue or an on-failure destination — without one, an event that fails all retries is silently discarded and you will never know it existed.
The dead letter queue has an alarm on depth greater than zero and a documented owner — an unmonitored DLQ is a data loss mechanism with extra steps.
A tested procedure exists for inspecting and replaying dead-lettered events — replay is not automatic and writing the tooling during an incident is too late.
Retry counts and backoff are configured deliberately per source — the platform defaults are rarely right, and retrying a permanent validation failure twice is wasted spend plus delay.
Transient and permanent errors are distinguished in code — a malformed payload should go straight to the DLQ; a downstream timeout should retry.
Partial batch failure reporting is implemented for batched sources — otherwise one bad record in a batch of a hundred causes all hundred to be reprocessed.
Downstream calls have explicit timeouts shorter than the function timeout — a default-infinite SDK client will burn the entire timeout and then fail with no useful error.
Circuit-breaking or shedding protects a failing downstream — thousands of concurrent executions all retrying a struggling dependency will keep it down.

7. Security

Each function has its own execution role scoped to the specific resources it uses — a shared role across functions grants every function the union of all permissions.
Wildcard resource ARNs have been eliminated or justified — Resource: "*" on a data action is the finding every audit will raise.
Resource policies restrict who can invoke the function — the invoker principal, source ARN, and source account should all be constrained.
Input from event sources is validated and treated as untrusted — an event body from a queue is user input that has travelled through one more hop.
Secrets are fetched at runtime from a secret manager, cached with a bounded TTL, and never logged — environment variables are visible to anyone with read access to the function configuration.
Dependencies are scanned in the pipeline and the package is rebuilt on a schedule — a function deployed a year ago is still running a year-old dependency tree.
Functions in a VPC use security groups that permit only the required egress — and a function that needs no private resource stays out of the VPC.
API-fronted functions have authentication, authorisation, and rate limiting at the gateway — the gateway is the only place you can reject traffic before it costs you an invocation.

8. Observability

Structured JSON logging is used with a correlation identifier propagated from the event — free-text logs across hundreds of short-lived invocations are unsearchable.
Log retention is set explicitly on every log group — the default in several platforms is to never expire, and high-volume function logs become a significant line item.
Distributed tracing is enabled and covers the asynchronous hops — an event-driven system where the trace stops at the queue tells you nothing about end-to-end latency.
Alarms cover errors, throttles, duration approaching timeout, and DLQ depth — duration approaching timeout is the leading indicator that catches the failure before it happens.
Business-level metrics are emitted from the function — events processed, events rejected, and end-to-end age of the oldest processed event.
Iterator age or queue age is alerted on for stream and queue consumers — a consumer keeping up at a steadily growing lag is the failure mode invisible to error-rate alerts.
A method exists to reproduce an invocation locally from a captured event payload — debugging by redeploying is unbearably slow.

9. Deployment, cost, and operations

Infrastructure and functions are deployed together as code from a pipeline — a function edited in the console is a change nobody can review or reproduce.
Deployments are progressive with automatic rollback on an error-rate alarm — weighted aliases or traffic-splitting revisions make this cheap, so there is no reason not to.
Function versions are immutable and aliases point at a specific version — deploying to a mutable latest pointer removes your ability to roll back precisely.
Cost per million invocations has been estimated including the event sources and log ingestion — the function is often the cheapest part of the bill; the queue, gateway, and logs are not.
A budget alarm and a concurrency ceiling limit the cost of a runaway loop — serverless removes the natural ceiling that a fixed fleet used to provide.
A runbook covers how to stop the workload — setting reserved concurrency to zero or disabling the event source mapping is the emergency brake, and the on-call engineer must know which one applies.

Sign-off

Area	Reviewer	Date	Outcome
Workload fit			Pass / Pass with actions / Fail
Configuration and packaging			Pass / Pass with actions / Fail
Cold starts and latency			Pass / Pass with actions / Fail
Concurrency, scaling, and downstream protection			Pass / Pass with actions / Fail
Event sources, delivery, and idempotency			Pass / Pass with actions / Fail
Failure handling			Pass / Pass with actions / Fail
Security			Pass / Pass with actions / Fail
Observability			Pass / Pass with actions / Fail
Deployment, cost, and operations			Pass / Pass with actions / Fail

Record every “Pass with actions” as a dated ticket with an owner before the workload is approved for production traffic.

Code Review

Mon, 01 Jan 0001 00:00:00 +0000

Code review is the cheapest place to catch a defect and the most expensive place to argue about brace style. This checklist is what a reviewer should hold in their head while reading a change: does it do what it claims, will it survive contact with production, and can the next person understand it. It also covers the author’s side, because most slow reviews are caused by a pull request that was never made reviewable.

REST API Design

Mon, 01 Jan 0001 00:00:00 +0000

An HTTP API is the hardest kind of code to change, because the people who depend on it do not deploy when you do. Most of the cost of a bad API is paid years later, in compatibility shims and support tickets. Work through this before the first consumer outside your team integrates, and again before any version is declared stable.

info

Who runs this: the owning team plus one reviewer who will have to consume the API. When: at design review, before the first endpoint ships, and again before the API is published externally.

1. Resources and URL structure

Paths name resources, not actions — POST /orders/42/refunds beats POST /refundOrder, because the resource model composes and the verb list does not.
Collections are plural and consistently cased — pick one convention for multi-word segments and apply it everywhere; mixed user_profiles and userProfiles guarantees integration bugs.
Nesting stops at one level — /customers/{id}/orders is useful, /customers/{id}/orders/{id}/lines/{id}/taxes forces callers to know an identifier hierarchy they should not need.
Identifiers in URLs are opaque to the client — exposing an auto-incrementing primary key leaks your record count and invites enumeration.
A resource has exactly one canonical URL — multiple aliases fragment caching, logging, and authorisation rules.
Genuinely non-CRUD operations are modelled as subordinate resources — a state transition can be a POST to /subscriptions/{id}/cancellations rather than an RPC verb bolted onto the path.
Trailing slash, casing, and encoding behaviour is defined and consistent — decide whether /Orders redirects or 404s, and make the router enforce it.

2. HTTP semantics

Method semantics are respected — GET and HEAD never change state, PUT and DELETE are idempotent, and POST is the only method allowed to be neither safe nor idempotent.
GET requests carry no request body and no side effects — intermediaries are entitled to retry, cache, and prefetch them.
Status codes distinguish the failure classes callers must handle differently — 400 for a malformed request, 401 for missing or invalid authentication, 403 for authenticated but not permitted, 404 for absent, 409 for a state conflict, 422 for semantically invalid content, 429 for rate limiting.
201 responses carry a Location header — the caller should not have to guess the URL of what it just created.
204 is used only when there is genuinely no body — returning 200 with an empty body forces every client to special-case parsing.
5xx is reserved for server faults — returning 500 for a validation failure trains callers to retry something that will never succeed and pollutes your error budget.
Long-running work returns 202 with a status resource — a synchronous request that blocks for minutes will be killed by some proxy in the path.
Partial updates use a defined patch format — PATCH with an ad-hoc merge dialect makes clearing a field to null indistinguishable from omitting it.

3. Request and response payloads

One media type is the default and it is negotiated properly — honour Accept, and return 415 for an unsupported Content-Type rather than silently attempting to parse it.
Field naming is consistent across every endpoint — one casing convention, and the same concept has the same field name everywhere.
Timestamps are RFC 3339 in UTC with an explicit offset — a bare local datetime is the most common cross-team integration defect.
Monetary amounts carry an explicit currency and use a string or minor-unit integer — a JSON number for money will be parsed as a float by somebody.
Enumerated values are documented and clients are told to tolerate unknown members — otherwise adding a value becomes a breaking change.
Top-level responses are objects, never bare arrays — an object leaves room to add pagination metadata later without breaking parsers.
Absent, null, and empty are distinguished deliberately — document what each means for every optional field, because callers will infer something.
Request bodies have a maximum size enforced before parsing — unbounded parsing is a trivially exploitable denial of service.

4. Errors

Errors use a single machine-readable structure across the whole API — RFC 9457 problem details is a reasonable default and saves every client writing a bespoke parser.
Every error carries a stable, documented error code — clients must be able to branch on something other than the human-readable message, which you will want to reword.
Validation errors identify the offending field — a 400 saying only “invalid request” turns integration into guesswork.
Error messages do not leak internals — stack traces, SQL fragments, internal hostnames, and library versions are reconnaissance material.
Errors include a correlation identifier that also appears in your logs — so a support ticket can be traced without asking the caller to reproduce.
Retryable and non-retryable failures are distinguishable — pair 429 and 503 with Retry-After so well-behaved clients back off correctly.

 HTTP/1.1 422 Unprocessable Content
Content-Type: application/problem+json

{
 "type": "https://api.example.com/problems/invalid-field",
 "title": "Validation failed",
 "status": 422,
 "code": "order.line_items.empty",
 "detail": "An order must contain at least one line item.",
 "instance": "/orders",
 "trace_id": "3f9a1c8e"
}

5. Pagination, filtering, and sorting

Collection endpoints are paginated by default with a documented maximum page size — an unpaginated collection is a time bomb attached to your slowest table.
Pagination uses a stable cursor rather than an offset — with offsets, rows inserted or deleted mid-scan cause results to shift, duplicate, or be skipped entirely.
The cursor is opaque and its encoding is not part of the contract — clients that decode it will break when you change the sort key.
A total count is either omitted or explicitly approximate — an exact count over a large filtered set is often more expensive than the page itself.
Sorting is limited to indexed fields — arbitrary sort parameters let a caller table-scan your production database.
Filter syntax is simple and closed — a bespoke query language on a query string becomes an injection surface and an optimiser problem.
Deep pagination is bounded — either cap the total traversable depth or require a narrower filter, rather than letting page 10,000 melt the database.

6. Versioning and evolution

The versioning strategy is decided and documented before launch — retrofitting versioning onto a live API is far more expensive than choosing wrongly at the start.
What counts as a breaking change is written down — removing or renaming a field, tightening validation, changing a status code, or adding a required request field all break existing callers.
Additive changes are safe because clients are documented as tolerant readers — state explicitly that unknown response fields must be ignored.
Deprecation has a published policy with a minimum notice period — and deprecated endpoints emit Deprecation and Sunset headers so callers can find them programmatically.
Usage per endpoint and per client is measured — you cannot retire anything you cannot prove nobody is calling.
Old versions have a defined end-of-life date, not an open-ended one — versions you never remove are versions you maintain forever.

warning

Common mistake: treating a change as non-breaking because your own client still works. Tightening a validation rule, narrowing an enum, or making an optional response field disappear will break somebody, even though your tests stay green.

7. Security and access control

Every endpoint requires authentication unless it is deliberately public — the default in the router should be deny, with public routes as explicit exceptions.
Authorisation is enforced per object, not per route — object-level authorisation failure is the most commonly exploited API vulnerability, and it always looks fine in a route table.
TLS is required and plaintext requests are rejected, not redirected — a redirect has already exposed the credential.
Tokens are short-lived and scoped — a long-lived token with full account scope turns a single leaked log line into a breach.
Rate limits exist per client and are communicated in response headers — including the limit, the remaining quota, and the reset time.
Mass assignment is prevented by an explicit allowlist of writable fields — binding a request body straight onto a model lets callers set role or account_id.
CORS policy names specific origins — a wildcard combined with credentials is a mistake that survives review because it makes the browser errors stop.
Responses never include fields the caller is not entitled to see — filtering in the client is not filtering.

8. Documentation and contract

A machine-readable specification exists and is generated from or verified against the implementation — hand-written OpenAPI drifts within one sprint.
Every endpoint documents its error responses, not just the success case — the error contract is the part integrators actually need.
Examples are real and executable — copy a request from the integration test suite rather than writing one by hand.
Authentication is documented end to end — how to obtain a credential, how to refresh it, and what happens when it expires mid-request.
Rate limits, quotas, and pagination defaults are stated in the documentation — not discovered in production by a caller who then files a support ticket.
A changelog records every change with its date and its compatibility impact.

9. Operability

Idempotency keys are supported on non-idempotent operations that create money movement or side effects — clients will retry a timed-out POST, and without a key you will charge twice.
Cache behaviour is explicit on every response — set Cache-Control deliberately, and use ETag with conditional requests where responses are expensive.
Request and response sizes, latency, and status code distribution are instrumented per endpoint — aggregate API metrics hide the one endpoint that is failing.
Timeouts are defined at every layer and the client-facing timeout is documented — so callers can set their own budget below yours rather than above it.
A load test has established the per-endpoint breaking point — particularly for the endpoints that fan out to other services.
Contract tests run in CI against the published specification — so a schema-breaking change fails the build rather than a customer’s integration.

Sign-off

Area	Reviewer	Date	Outcome
Resources and URL structure			Pass / Pass with actions / Fail
HTTP semantics			Pass / Pass with actions / Fail
Request and response payloads			Pass / Pass with actions / Fail
Errors			Pass / Pass with actions / Fail
Pagination, filtering, and sorting			Pass / Pass with actions / Fail
Versioning and evolution			Pass / Pass with actions / Fail
Security and access control			Pass / Pass with actions / Fail
Documentation and contract			Pass / Pass with actions / Fail
Operability			Pass / Pass with actions / Fail

Record each “Pass with actions” as a dated ticket with an owner; anything unresolved in the versioning or security sections should block publication.

Database Schema Migration

Mon, 01 Jan 0001 00:00:00 +0000

A schema migration is the one deployment step that is not trivially reversible. Code can be rolled back in seconds; a dropped column cannot. The two failure modes that cause almost every migration incident are the same every time: a lock held long enough to stall the application, and a schema change deployed in the wrong order relative to the code that reads it. This checklist is built around avoiding both.

New Service Bootstrap

Mon, 01 Jan 0001 00:00:00 +0000

Everything a service is missing on its first day it will still be missing two years later, because nobody gets funded to retrofit a health endpoint. The point of a bootstrap checklist is to make the boring scaffolding a precondition of the first commit rather than a cleanup task. This covers the repository, the pipeline, and the operational hooks; the go-live decision itself belongs to the production readiness review.

info

Who runs this: the engineer creating the service, with a platform or infrastructure reviewer. When: in the first week of the repository existing, well before any traffic. See also the

info

Who runs this: the maintaining team, with sign-off from legal or open source programme office and a security reviewer. When: at least two weeks before the intended publication date, because licence and history findings are slow to resolve.

1. Legal and licensing

The licence is chosen deliberately and approved by whoever owns that decision — permissive and copyleft licences make materially different commitments, and changing later requires the agreement of every contributor.
A LICENSE file with the full, unmodified licence text is at the repository root — package managers, scanners, and GitHub’s licence detection all key off it.
Every dependency’s licence is compatible with your chosen licence — a copyleft dependency inside a permissively licensed project is the finding that most often stops a release.
Vendored, copied, or generated third-party code retains its original licence and attribution — including code pasted from an answer site or produced from a licensed template.
A NOTICE file exists where any dependency licence requires attribution — Apache-2.0 in particular obliges you to carry it forward.
Contributions from outside the organisation are covered by a CLA or a DCO sign-off — decide before the first pull request, not after.
Trademarks, product names, and logos are used according to policy — the code licence does not grant trademark rights, and the distinction confuses almost everyone.
Patent implications have been reviewed if the project implements anything patented in-house.

2. History and secret scrubbing

The entire git history has been scanned for secrets, not just the current tree — a key deleted in a later commit is still trivially retrievable from the history the moment the repository is public.
Automated secret scanning has been run with more than one tool — detectors have different rule sets, and the cost of a second run is minutes.
Any credential that ever appeared in the history is rotated regardless of scrub outcome — assume it is compromised; a rewritten history does not help if a fork or a mirror already exists.
Internal hostnames, IP ranges, internal URLs, and infrastructure identifiers are removed — these are reconnaissance material and are rarely caught by secret scanners.
Customer names, personal data, and real datasets are removed from fixtures, tests, and documentation — test fixtures are the most overlooked source of a personal data disclosure.
Internal commit messages and branch names have been reviewed — they routinely reference ticket systems, incidents, security findings, and individuals by name.
If the history cannot be cleaned, publish from a squashed initial commit — losing history is preferable to publishing a leak, and it is the honest default for code extracted from a monorepo.
Push protection and secret scanning are enabled on the public repository before the first push — the protection has to exist before the commit that would need it.

report

Blocking: never publish before rotating every credential that has appeared anywhere in the repository history. Public repositories are indexed by automated scrapers within minutes, and a rewritten history does not un-publish anything that was already fetched or forked.

3. Documentation

The README states what the project does and who it is for in the first three lines — a visitor decides whether to keep reading in about ten seconds.
Installation and a minimal working example are copy-pasteable and have been tested on a clean machine — by someone who did not write them.
Supported versions, platforms, and runtimes are stated explicitly — otherwise every unsupported combination becomes an issue in your tracker.
The project’s scope and non-goals are documented — the cheapest way to decline a feature request is to have said no to that category in advance.
CONTRIBUTING.md covers how to build, test, and submit a change, and what makes a change acceptable — including whether you want unsolicited feature pull requests at all.
A CODE_OF_CONDUCT.md is present with a working, monitored reporting address — a reporting alias nobody reads is worse than none.
Documentation is versioned with the code — documentation describing an unreleased API is a support burden from day one.

4. Code and repository quality

The code builds from a clean clone with only the documented prerequisites — no internal package registry, no internal base image, no company-only tooling.
All internal dependencies have been removed or replaced with public equivalents — an internal utility library referenced in go.mod or package.json will break every external build immediately.
Comments, identifiers, and error messages are in the project’s stated language and free of internal jargon — internal codenames make code unreadable to outsiders.
Tests run in CI on the public repository, on the platforms you claim to support — a badge that reflects an internal pipeline nobody can see is not evidence.
CI on pull requests from forks is configured safely — workflows that run untrusted code with access to repository secrets are a well-understood supply chain attack.
The default branch is protected and direct pushes are disabled, including for maintainers.
Dependency update automation and vulnerability alerts are enabled — public projects are scanned by everyone, and a stale vulnerable dependency will be reported publicly.

5. Versioning and releases

The project follows semantic versioning and says so — and the meaning of the major, minor, and patch positions for this project is documented.
The initial version signals stability honestly — publishing 1.0.0 commits you to a stable public interface; use 0.x if you are not ready for that.
What constitutes the public API is defined — which packages, modules, and symbols are covered by the compatibility promise, and which are internal.
A CHANGELOG.md is maintained in a keep-a-changelog style with the changes grouped by type — generated commit logs are not a changelog because they are written for the wrong audience.
Releases are tagged, signed, and immutable — and the tag matches the artefact published to the package registry.
The release process is automated and documented — a release only one person can perform is a bus-factor problem and an inconsistency generator.
Build provenance or an SBOM is published with the artefacts — consumers increasingly require it, and generating it later for old releases is impossible.
The package registry namespace and account are owned by the organisation, not by an individual’s personal account.

6. Security posture

A SECURITY.md states how to report a vulnerability and what response time to expect — without it, reporters disclose publicly by default.
A private disclosure channel exists and is monitored by more than one person — a single monitored inbox means a report sits unread while the reporter’s disclosure clock runs.
An embargo and coordinated disclosure process is agreed before you need it — the first report is a bad time to invent a process.
Static analysis and dependency scanning run on the public repository — outsiders will scan it anyway, and it is better to find the issue before they file it publicly.
Publishing credentials for the package registry are scoped, stored in a secret manager, and protected by multi-factor authentication — a compromised publish token lets an attacker ship malware under your name.
Maintainer accounts with publish rights all have phishing-resistant multi-factor authentication enabled.
The threat model of running this code is documented if it processes untrusted input — users need to know what guarantees you do and do not make.

7. Governance and sustainability

Maintainers are named and there is more than one with full access — a single-maintainer project with a single publish key is one lost laptop from being abandoned.
The decision-making process for accepting changes is written down — even if it is “the maintainers decide”, stating it prevents a lot of friction.
Expected response times for issues and pull requests are stated — under-promising is fine; silence is what drives contributors away.
The support commitment is explicit — which versions receive fixes, and for how long.
The internal cost of maintenance has an owner and allocated time — an open source project without funded maintenance time degrades into an unanswered issue tracker.
An archival or transfer plan exists for when the project is no longer maintained — deciding in advance is far kinder to users than silent abandonment.

8. Launch and after

A pre-publication dry run has been done on a private repository with the same settings — including a full release to a registry’s test channel where one exists.
Issue templates and a pull request template are configured — they measurably improve the quality of what arrives.
Repository topics, description, and homepage are set — this is how the project is found.
A rollback plan exists for the release itself — know your registry’s rules on unpublishing and yanking before you need them, because most do not allow a version to be replaced.
Someone is on hand for the first week to answer issues — the first days after publication determine whether the project attracts contributors or a backlog.
Internal teams that depend on the code know it is now public — so they stop making internal-only assumptions in their pull requests.

Sign-off

Area	Reviewer	Date	Outcome
Legal and licensing			Pass / Pass with actions / Fail
History and secret scrubbing			Pass / Pass with actions / Fail
Documentation			Pass / Pass with actions / Fail
Code and repository quality			Pass / Pass with actions / Fail
Versioning and releases			Pass / Pass with actions / Fail
Security posture			Pass / Pass with actions / Fail
Governance and sustainability			Pass / Pass with actions / Fail
Launch and after			Pass / Pass with actions / Fail

Legal and secret-scrubbing sections must be a clear pass; everything else may carry a dated follow-up ticket with a named owner.

Frontend Performance

Mon, 01 Jan 0001 00:00:00 +0000

Frontend performance is decided by a small number of choices — how much JavaScript ships, when fonts and images load, and what blocks the main thread — and then eroded by a thousand small additions nobody measured. This checklist is organised around what the browser actually does with your page, and around the Core Web Vitals as the shared measurement, not as a scoring game. Measure on a mid-range Android device on a throttled connection, because that is what most of your users have.

Web Accessibility (WCAG)

Mon, 01 Jan 0001 00:00:00 +0000

Automated tools catch roughly a third of accessibility defects, and almost none of the ones that actually stop somebody completing a task. This checklist is organised around how the interface is used — keyboard, screen reader, zoom, forms — rather than around the numbering of the specification, and it covers the success criteria added in WCAG 2.2 that most existing components fail. Target level AA unless a legal obligation says otherwise.

Observability

Mon, 01 Jan 0001 00:00:00 +0000

Monitoring answers questions you thought of in advance. Observability is whether you can answer a question you have never asked before, at 3am, without deploying anything. This checklist covers the instrumentation, the pipeline that carries it, and the dashboards and alerts built on top — in the order you would build them.

info

Who runs this: the owning team, reviewed by whoever is on call for the service. When: before a production readiness review, and again after any incident where the answer was we could not tell.

1. Signal coverage

The four golden signals exist for every user-facing entry point — traffic, error rate, latency distribution, and saturation of the constraining resource, per endpoint rather than only in aggregate.
Latency is recorded as a histogram, not an average — averages hide the tail, and the tail is what your users are complaining about.
Errors are split by cause — client error, dependency failure, timeout, and unhandled exception behave differently and need different responses.
Queue depth and consumer lag are instrumented for every asynchronous path — an async system fails silently by falling behind, not by returning errors.
Business-level counters exist alongside technical ones — orders placed, sign-ups completed, payments settled; a drop here is real even when every technical metric looks green.
Batch and scheduled jobs emit start, finish, duration, and record counts — a cron job that stopped running produces no signal at all unless you explicitly watch for its absence.
Dependency calls are instrumented on the client side — a downstream service that reports itself healthy can still be unreachable from your network position.

2. Metrics

Metric names follow one documented convention — unit suffix, consistent prefix, and the same label spelling across services, or cross-service queries silently return nothing.
Cardinality is bounded and reviewed — never label a metric with user ID, request ID, URL path with embedded IDs, or anything unbounded; this is the single most common way to take down a metrics backend.
Counters are monotonic and reset-safe — use rate functions over counters rather than gauges that your code decrements, so a restart does not produce a negative spike.
Scrape or push interval is short enough to see the incidents you care about — a 60-second interval cannot show you a 30-second outage.
Recording rules pre-compute the expensive queries used by dashboards and alerts — a dashboard that takes 40 seconds to load will not be opened during an incident.
Retention is tiered deliberately — high resolution for recent debugging, downsampled long-term series for capacity and trend work.

3. Logs

All logs are structured — JSON or an equivalent key-value format, so that fields can be filtered and aggregated rather than grepped.
Every log line carries a trace ID and a request or job ID — without a correlation key, a distributed request becomes a hundred unlinked lines across a dozen services.
Log levels are used consistently and documented — if everything is logged at ERROR, error-rate panels and log-based alerts are worthless.
No secrets, credentials, tokens, or personal data reach the log pipeline — audit request-body logging and auth-header logging specifically, and add a redaction filter at the collector as a backstop.
Log volume per request is bounded — a debug-level loop inside a hot path can cost more than the service itself and will be discovered on the invoice.
Retention satisfies both the investigation need and the data retention policy — long enough to chase a slow-burning bug, short enough to be defensible.
Log ingestion failure is itself monitored — a full disk or a rejected batch means you are flying blind precisely when things are going wrong.

4. Distributed tracing

Trace context propagates across every hop — HTTP, gRPC, message queues, and background workers; a trace that stops at the first hop tells you nothing you did not already know.
Spans are created for outbound calls, database queries, and cache lookups — the point of tracing is attributing latency to a component, which needs spans at component boundaries.
Sampling is head-based with a tail-based override for errors and slow requests — uniform 1% sampling reliably discards the exact traces you want.
Span attributes include the things you would filter on — tenant, region, endpoint, and version, without embedding unbounded values.
Instrumentation uses OpenTelemetry or another vendor-neutral SDK — so that changing backend does not mean re-instrumenting every service.
Trace IDs are surfaced to users or in error responses — a support ticket that quotes a trace ID turns a two-hour investigation into a two-minute one.

5. Dashboards

Every service has one overview dashboard that fits on a single screen — golden signals plus dependency health, with no scrolling required.
Dashboards are ordered top-down — user-visible symptoms first, then service internals, then infrastructure, matching how you actually diagnose.
Each panel has a stated purpose and a known-good range — a graph nobody can interpret under pressure is decoration.
Deploy and configuration-change markers are overlaid on the time axis — most incidents correlate with a change, and this makes the correlation instant.
Dashboards are defined as code and version-controlled — a hand-edited dashboard is lost the first time someone else edits it.
A cross-service view exists for the critical user journey — end-to-end latency and success rate for the journey, not just for individual services.

6. Alerting

Alerts fire on user-visible symptoms, not on causes — high CPU is only worth paging about if it is degrading something a user can perceive.
Every alert that pages links to the runbook section that resolves it — an alert with no documented response should be downgraded to a ticket.
Paging alerts and ticketing alerts are separate channels with separate severities — if everything pages, nothing pages.
Alert conditions include a duration so transient blips do not page — and the duration is short enough that the alert still fires inside the error budget window.
Absence of data triggers an alert — a crashed exporter looks exactly like a perfectly healthy system on most dashboards.
Alert rules are tested by deliberately breaking something in staging — an untested alert rule is a hypothesis about a query language.
Alert volume per shift is measured and reviewed monthly — track the noisiest alerts and either fix, retune, or delete them.

warning

Common mistake: alerting on every metric that has a threshold. Each new page must be justified against the question what would the responder do differently in the next five minutes? If there is no answer, it is a dashboard panel, not an alert.

7. SLOs and error budgets

Each critical user journey has an SLI defined as good events divided by valid events — with an explicit definition of what counts as good and what requests are excluded.
SLO targets are set from what users need, not from what the system currently does — and they are achievable, since an SLO that is permanently breached is ignored.
The error budget and its burn rate are visible on the service dashboard — a burn-rate alert catches fast-burning incidents far earlier than a raw threshold.
Multi-window multi-burn-rate alerting is configured — a fast window for outages and a slow window for chronic degradation, so both are caught without false pages.
A policy exists for what happens when the budget is exhausted — typically freezing feature releases in favour of reliability work, agreed with the product owner in advance.

8. Synthetic monitoring and the pipeline itself

A synthetic probe exercises the critical user journey from outside the network — real user traffic can drop to zero while every internal metric stays green.
Probes run from more than one region or provider — so that a probe failure is distinguishable from an outage in the probing infrastructure.
The telemetry pipeline has its own monitoring and its own alerting path — collector health, drop rates, and backend ingestion latency, alerted through a channel that does not depend on the pipeline.
Observability cost is attributed per service and reviewed — telemetry regularly becomes a top-three line item and is best trimmed before finance asks.
Data survives the loss of the primary region — if the observability stack lives only in the region that just failed, you will debug the outage from memory.
On-call engineers have read access to every signal without raising a request — an access-request queue during an incident is an outage extender.

Sign-off

Area	Reviewer	Date	Outcome
Signal coverage			Pass / Pass with actions / Fail
Metrics			Pass / Pass with actions / Fail
Logs			Pass / Pass with actions / Fail
Distributed tracing			Pass / Pass with actions / Fail
Dashboards			Pass / Pass with actions / Fail
Alerting			Pass / Pass with actions / Fail
SLOs and error budgets			Pass / Pass with actions / Fail
Synthetic monitoring and the pipeline itself			Pass / Pass with actions / Fail

Record every “Pass with actions” as a dated ticket with a named owner, and re-review any area marked Fail before the service is treated as observable.

On-Call Handover

Mon, 01 Jan 0001 00:00:00 +0000

Most on-call failures are not heroic technical failures. They are an ongoing issue that the outgoing engineer knew about, forgot to mention, and that paged the incoming engineer at 2am with no context. A handover is a short, structured conversation with a written artefact attached; this checklist covers what each side owes the other and what has to be true before the pager actually moves.

info

Who runs this: the outgoing and incoming on-call engineers together, with the team lead as escalation if something cannot be handed over cleanly. When: at a fixed time at the end of every shift, before the rotation flips in the paging tool.

1. Before the handover meeting

The handover happens at a scheduled time, not whenever the rotation silently flips — an automatic rotation change with no conversation is how context is lost.
The outgoing engineer has written the handover note before the meeting starts — the meeting is for questions and clarification, not for composing the note live.
The handover note lives in a persistent, searchable place — a ticket, a channel with retention, or the runbook repository, not a direct message that nobody else can find.
Both engineers have the shift open in front of them — the alert history, the incident tracker, and the change log for the period, so nothing is recalled from memory alone.
The meeting is time-boxed to fifteen minutes — if it needs longer, that is a signal to escalate an unresolved issue rather than to talk for an hour.
A handover is still held when the shift was quiet — the absence of pages is itself information, and skipping the ritual is how the ritual dies.

2. What the outgoing engineer hands over

Every page received during the shift, with what was done about it — including pages that were acknowledged and ignored, which are the most likely to recur.
Open incidents with current status, severity, and who else is involved — never hand over an active incident by pager alone; brief the incoming engineer directly.
Any temporary mitigation still in place — a scaled-up cluster, a disabled feature flag, a paused consumer, or a manually restarted job, each with the condition for reverting it.
Silenced or suppressed alerts, with the expiry time for each silence — a permanent silence created at 3am is a future outage nobody will see coming.
Anything that looked odd but did not page — an unexplained latency step, a growing queue, a new error signature; these are the early warning that the next shift will need.
Manual work in flight — a half-finished migration, a partially applied configuration change, or a rollback that was started and not finished.
Tickets raised during the shift, with links — so follow-up work does not have to be reconstructed from the chat history.

3. What the incoming engineer confirms

The pager actually reaches them — send a test notification and confirm it arrives on the phone that will be in the room, with the ringer on and do-not-disturb bypassed.
Access works before it is needed — production console, cluster credentials, VPN, the incident tool, and any break-glass procedure, all exercised now rather than mid-incident.
They can reach the escalation path — the secondary on-call, the incident commander pool, and the vendor support contact, with numbers that have been dialled at least once.
They have read the handover note and asked their questions — the handover is complete when the incoming engineer can restate the open items, not when the outgoing engineer stops talking.
They know which changes are planned during their shift — releases, migrations, certificate rotations, and third-party maintenance windows all raise the probability of a page.
They confirm their own availability constraints — travel, poor signal, or personal commitments must be flagged now so cover can be arranged, not discovered when a page goes unanswered.

report

Blocking: do not flip the rotation until the incoming engineer has confirmed a test page arrived and that production access works. A silent pager is indistinguishable from a healthy system until the outage is already an hour old.

4. System state review

The service dashboards are walked through together — current error rate and latency versus the same time last week, so a slow drift is visible rather than normalised.
Error budget consumption for the shift is reported — a large burn during a quiet shift means something is degrading without paging.
Capacity headroom is checked — disk, database connections, queue depth, and any resource that is consumed monotonically and cannot be recovered without action.
Certificate and credential expiries within the next fourteen days are listed — expiry-driven outages are entirely predictable and entirely avoidable.
Backlogs and dead letter queues are reported with their trend — a dead letter queue that grew during the shift is a customer-impacting bug that has not surfaced yet.
Known-degraded dependencies are named — including third-party providers currently on a status-page incident, so the next page is not misdiagnosed.

5. Alerting and paging hygiene

Alerts that fired more than twice this shift are flagged as noisy — repeated pages for the same non-actionable condition are the main driver of on-call attrition.
Every silence created during the shift has an owner and an expiry — and silences that are about to expire are transferred explicitly.
New alerts added during the shift are reviewed for a runbook link — an alert authored at 3am usually has no documented response yet.
False positives are ticketed, not just tolerated — if it is not written down, the same alert will wake five more people before anyone fixes it.
The paging tool’s schedule shows the correct people for the next fourteen days — including holiday cover, and with an override rather than a private arrangement.

6. Documentation and runbooks

Any runbook used during the shift was corrected where it was wrong — the moment you discover a stale step is the only moment you will reliably remember to fix it.
Any incident resolved without a runbook produced one — even a rough set of commands is better than the next responder starting from nothing.
New or changed escalation contacts are recorded in the shared directory — not in one engineer’s phone.
Links in the handover note resolve — dashboards, tickets, and runbooks should be checked, since a dead link during an incident costs minutes you do not have.
Manual steps performed during the shift are captured as automation candidates — the third time a human runs the same recovery command, it should become a script.
The runbook index reflects any service added or retired since the last shift — responders will search the index first, and a missing entry reads as no procedure exists.

7. Escalation and follow-up ownership

Each open item has exactly one named owner after the handover — either the incoming engineer or a specific person on the team, never “on-call” as an abstraction.
Items that need daytime engineering work are moved into the team backlog, not carried in the pager — on-call is for response; recurring toil belongs in sprint planning.
Anything unresolved for more than two consecutive shifts is escalated to the team lead — repeated handover of the same problem is the signal that on-call cannot fix it alone.
Incidents from the shift have a postmortem owner assigned where the severity threshold was met — postmortems left unassigned at handover are rarely written.
Follow-up actions from previous postmortems that are now overdue are surfaced — the same incident recurring is almost always an unimplemented action item.

8. Shift health and sustainability

Overnight pages are recorded with their timestamps — sleep interruption is the metric that predicts burnout, and it is invisible unless it is counted.
The outgoing engineer takes compensating rest after a disrupted night — agreed as team policy, so that taking it does not require asking permission.
The number of actionable versus non-actionable pages is tracked per shift — the ratio, not the raw count, tells you whether alerting is working.
On-call load is reviewed at least monthly against a target — a commonly used ceiling is around two pages per shift; sustained breaches justify pausing feature work.
No single person holds the pager for consecutive rotations without agreement — informal cover arrangements hide load from everyone who could fix it.

Sign-off

Area	Reviewer	Date	Outcome
Before the handover meeting			Pass / Pass with actions / Fail
What the outgoing engineer hands over			Pass / Pass with actions / Fail
What the incoming engineer confirms			Pass / Pass with actions / Fail
System state review			Pass / Pass with actions / Fail
Alerting and paging hygiene			Pass / Pass with actions / Fail
Documentation and runbooks			Pass / Pass with actions / Fail
Escalation and follow-up ownership			Pass / Pass with actions / Fail
Shift health and sustainability			Pass / Pass with actions / Fail

Attach the completed table to the handover note and raise a dated ticket for every item marked “Pass with actions” before the rotation flips.

Incident Management

Mon, 01 Jan 0001 00:00:00 +0000

An incident process exists so that nobody has to invent one under pressure. This checklist follows the shape of a real incident in order — detection, triage, roles, mitigation, communication, resolution, and the handover to a postmortem — and is meant to be worked through top to bottom while the incident is live, not read afterwards.

info

Who runs this: the incident commander, supported by the ops lead and comms lead. When: from the moment an incident is declared until the postmortem owner has accepted the handover.

1. Detection and declaration

The incident has an explicit declaration, not a gradual realisation — someone says the words and starts the clock, because an undeclared incident has no commander and no communications.
Anyone can declare an incident without asking permission — the cost of a false declaration is a short call; the cost of a delayed one is measured in hours of customer impact.
Declaration creates the incident record automatically — a ticket, a dedicated channel, and a bridge, so the responders spend their first minutes diagnosing rather than setting up tooling.
The detection source is recorded — alert, synthetic probe, customer report, or internal observation; detection by customer report is itself a finding for the postmortem.
Time of first customer impact is estimated early and refined later — it drives severity, SLA obligations, and any regulatory reporting clock.
Recent changes are pulled up immediately — deploys, feature flag flips, configuration and infrastructure changes in the last few hours account for the large majority of incidents.

2. Triage and severity

Severity is assigned from a written matrix, not by feel — the matrix should key on customer impact and scope, so that two different commanders reach the same answer.
Severity is set within the first few minutes and revised openly when it changes — under-declaring to avoid waking people is the most expensive mistake in this process.
Blast radius is scoped explicitly — which customers, which regions, which features, and whether the impact is total or partial degradation.
Data loss, data corruption, and security compromise are checked for by name — each of these changes the response entirely and cannot be discovered late.
Each severity level maps to a defined response — who is paged, how often updates go out, and whether executives and legal are notified.
Regulatory and contractual notification clocks are identified at triage — breach notification and SLA credit windows start at impact, not at your convenience.

3. Roles

An incident commander is named out loud and acknowledged by everyone on the call — the commander coordinates and decides; they do not debug, because a commander with their head in a terminal is not commanding.
An ops lead owns the hands-on-keyboard work — all changes to production during the incident go through this person, so that two responders never fight over the same system.
A comms lead owns all outward communication — status page, customer messaging, and internal stakeholder updates, so responders are not interrupted to answer questions.
A scribe records a timestamped log of observations, decisions, and actions — memory is unreliable and the postmortem timeline is impossible to reconstruct afterwards.
Roles are handed over explicitly when someone tires or leaves — with a verbal summary and confirmation, never by simply going quiet.
Subject matter experts are pulled in on request and released when done — an open call full of idle observers degrades signal for everyone remaining.
The commander role is separated from seniority — the most senior person in the room is often the most useful debugging, and command is a distinct skill.

warning

Common failure: nobody takes the commander role because everyone assumes someone else has it. If ten minutes pass with no named commander, the most recently paged responder takes it by default and says so in the channel.

4. Mitigation

Restoring service takes priority over understanding the cause — the investigation continues in parallel, but mitigation is not blocked on a diagnosis.
Rollback is considered as the first option for any change-correlated incident — reverting to a known-good state is faster and far less risky than fixing forward under pressure.
One change is made at a time, announced before and confirmed after — simultaneous changes make it impossible to tell what worked and can compound the damage.
Every mitigating action is logged with its timestamp and its effect — including actions that made no difference, which are just as important in the postmortem.
Emergency changes bypassing normal review are recorded for retrospective approval — break-glass access should be usable and auditable, not blocked.
Temporary mitigations are ticketed for reversal at the moment they are applied — an emergency capacity increase or disabled feature that nobody revisits becomes permanent cost or permanent breakage.
Evidence is preserved before destructive remediation — capture logs, heap dumps, and a snapshot of the failing node before restarting or replacing it, or the cause is gone forever.

5. Communication cadence

The status page is updated within the response window defined for the severity — customers who cannot tell whether the problem is yours will flood support and assume the worst.
Updates go out on a fixed cadence even when there is nothing new — an update saying the investigation continues and the next update is in thirty minutes prevents the escalation phone calls.
Customer-facing messages describe impact and workaround, not internal architecture — say which features are affected and what a user can do, and avoid speculating about cause.
Internal and external communications are consistent — leaked contradictions between an internal channel and a public status page cost more trust than the outage did.
Support and account teams get a briefing they can paste to customers — otherwise each one improvises a different and probably wrong answer.
One channel is designated as the source of truth — side conversations in direct messages fragment the record and hide decisions from the scribe.
Executives receive a summary on a separate track from the response channel — so that stakeholder questions do not interrupt the responders.

6. Resolution

Recovery is verified from the customer’s perspective before declaring resolution — check the synthetic probe and a real user journey, not just that the error graph came down.
Backlogs, queues, and retry storms are drained and confirmed healthy — a restored service can immediately fall over again under the queued load released at recovery.
Data written during the incident is checked for correctness — partially processed transactions, duplicated messages, and skipped records need explicit reconciliation.
The all-clear is announced in every channel where the incident was announced — including the status page, which is commonly left showing a resolved incident as ongoing.
Monitoring is watched for an agreed stabilisation period before standing down — resolving too early and re-declaring twenty minutes later damages credibility more than waiting.
Total impact is quantified before the call ends — duration, affected users or requests, and error budget consumed, while the numbers are still easy to query.

7. Handover to postmortem

A postmortem owner is named before the incident call ends — an unassigned postmortem is a postmortem that does not get written.
The severity threshold that requires a postmortem is written policy — so it is not renegotiated case by case by whoever is tired.
The scribe’s timeline, the chat log, and relevant graphs are attached to the incident record immediately — dashboards roll off and chat retention expires faster than postmortems get written.
Temporary mitigations still in place are listed explicitly for the postmortem to track to reversal.
A due date for the postmortem draft is set, typically within five working days — accuracy of recall drops sharply after the first few days.
Open follow-up work is ticketed now rather than deferred to the postmortem — anything genuinely urgent should not wait for a document to be written.

8. Process health

The incident process is rehearsed, not only exercised in production — run game days or tabletop exercises so that first-time commanders are not learning during a real outage.
New joiners shadow an incident before commanding one — and shadowing is scheduled rather than left to chance.
Incident metrics are tracked over time — time to detect, time to acknowledge, time to mitigate, and how many incidents were customer-reported rather than alert-detected.
Repeat incidents are counted separately — a rising repeat rate means postmortem actions are not being completed.
The process documentation is short enough to be read during an incident — a forty-page policy is not an incident process, it is an audit artefact.

Sign-off

Area	Reviewer	Date	Outcome
Detection and declaration			Pass / Pass with actions / Fail
Triage and severity			Pass / Pass with actions / Fail
Roles			Pass / Pass with actions / Fail
Mitigation			Pass / Pass with actions / Fail
Communication cadence			Pass / Pass with actions / Fail
Resolution			Pass / Pass with actions / Fail
Handover to postmortem			Pass / Pass with actions / Fail
Process health			Pass / Pass with actions / Fail

Review this table at the postmortem and raise a dated ticket with a named owner for every item that did not pass.

Postmortem

Mon, 01 Jan 0001 00:00:00 +0000

A postmortem is not a report you file to prove the incident was handled. It is the mechanism by which an organisation converts one expensive outage into knowledge that prevents the next several. This checklist covers the writing, the analysis, the action items, and the review meeting — in the order the document is produced.

info

Who runs this: the postmortem owner named at the end of the incident, with the incident commander and responders as contributors. When: draft within five working days of resolution, reviewed within ten.

1. Scope, ownership, and timing

A single named owner is responsible for the document — shared ownership reliably produces no document at all.
The trigger threshold for writing a postmortem is written policy — typically any customer-visible incident above a defined severity, any data loss, and any incident that recurred.
The draft is written within five working days — recall degrades quickly and responders move on to other work.
Contributors are given time to write it as part of their normal workload — a postmortem written entirely in someone’s evenings is a postmortem that gets skipped next time.
A near miss can be written up voluntarily — the cheapest lessons come from incidents that were caught before customers noticed, and these only surface if writing them is welcomed.
The document lives in a searchable shared archive — the value compounds only if a future engineer can find the one from eighteen months ago.

2. Blameless writing

The document describes systems and decisions, not people — name roles or teams rather than individuals, since the goal is a system that tolerates ordinary human error.
Every human action is framed with the information available at the time — someone acted reasonably given what their dashboard showed, and the fix belongs to the dashboard.
Counterfactuals are removed from the analysis — sentences beginning “if only they had” describe a world that did not exist and produce no actionable change.
The words “human error” and “operator mistake” do not appear as conclusions — they are the point at which analysis stops rather than the point at which it should start.
Hindsight-loaded language is edited out — “obviously”, “clearly”, and “should have known” all signal that the writer is judging rather than explaining.
The review confirms that responders would be comfortable if the document were read widely — if people would hesitate to be named, the culture is not yet blameless and the next incident will be reported late or not at all.

report

This is the load-bearing item on the page. The moment a postmortem is used in a performance conversation, engineers start hiding incidents, and every subsequent postmortem becomes fiction. Blamelessness is not politeness; it is what makes the data trustworthy.

3. Timeline construction

The timeline is built from sources, not from memory — chat transcripts, alert history, deploy logs, and the scribe’s notes, each entry linked to its evidence.
Every entry carries an absolute timestamp with a stated timezone — mixed local times across a distributed team make the sequence unreconstructable.
The timeline starts before the incident — include the change, the configuration edit, or the traffic shift that set up the failure, which is often days earlier.
Key markers are called out explicitly — first customer impact, first alert, first human acknowledgement, mitigation applied, and full recovery.
What responders believed at each point is recorded alongside what was actually true — the gap between the two is usually the richest source of findings.
Detection delay and mitigation delay are stated as numbers — these are the two levers you can actually pull, and they need to be measurable to be improved.
Actions that had no effect are included — the dead ends explain where diagnosis time went and often point at missing signals.

4. Impact

Customer impact is quantified, not described — affected users or requests, duration, and which functionality was degraded versus fully unavailable.
Error budget consumed is reported against the relevant SLO — this connects the incident to the reliability decisions the team has already agreed.
Data impact is stated explicitly — records lost, duplicated, or corrupted, and whether reconciliation was completed or is still outstanding.
Internal cost is recorded — responder hours, work displaced, and the number of people who lost sleep, since this is what justifies the prevention work.
Financial and contractual consequences are noted where they apply — SLA credits, lost transactions, and any regulatory notification that was triggered.

5. Contributing factors and root cause

Multiple contributing factors are identified, not a single root cause — complex systems fail through combinations, and a single-cause narrative closes the analysis prematurely.
Each contributing factor is categorised — trigger, latent condition, missing safeguard, detection gap, or response impediment, since each category attracts a different kind of fix.
Latent conditions that existed long before the incident are named — the missing timeout, the unbounded queue, or the untested failover was there for months and will still be there tomorrow.
The analysis asks why the safeguards did not work, not only why the failure happened — every serious incident passed through defences that were supposed to stop it.
Detection and response are analysed separately from causation — an incident that would have lasted two minutes with better alerting is primarily a detection problem.
Any recurrence of a previous incident is linked to it — and the action items from that earlier postmortem are checked for completion, which is usually where the answer is.
Speculation is labelled as speculation — where the cause could not be conclusively determined, say so rather than committing to a plausible guess that ends the investigation.

6. Action items

Every action item has one named individual owner and a due date — a team name is not an owner and “next quarter” is not a date.
Action items are tracked in the normal backlog, not only inside the document — work that lives in a wiki page is work that does not get scheduled.
Each item is classified by whether it prevents, detects, or mitigates — a set of items that are all detection improvements means nothing is actually being fixed.
Items are specific enough to be verifiably complete — “improve monitoring” cannot be closed honestly, whereas “add a burn-rate alert on the checkout SLO” can.
The list is prioritised and deliberately short — twenty low-priority items produce nothing; three completed high-priority items prevent the recurrence.
Items intentionally not being done are recorded as accepted risk with a decision-maker — an explicit acceptance is far better than an item that quietly ages out.
Reverting temporary mitigations from the incident appears as an action item — the emergency capacity increase and the disabled feature flag both need an owner.
Overdue action items are escalated on a schedule — a monthly review of open postmortem actions catches the drift before the incident repeats.

7. The review meeting

The draft is circulated for reading before the meeting — reading the document aloud consumes the whole session and prevents discussion.
Everyone who responded is invited, plus someone from outside the team — an outsider asks the questions insiders have stopped noticing.
The meeting focuses on the analysis and the action items, not on re-narrating the timeline — the timeline should be settled in the draft, with corrections collected in writing.
Disagreements about contributing factors are recorded in the document rather than resolved by seniority — a documented disagreement is more useful than a false consensus.
Action items are agreed with their owners present and consenting — an owner assigned in absentia has not accepted anything.
The meeting is time-boxed and ends with an explicit decision to publish — postmortems that stay in draft indefinitely deliver none of their value.

8. Publication and organisational learning

The final document is published to the whole engineering organisation — the audience that learns most is the teams who were not involved.
A short summary is written for non-engineering stakeholders where impact warrants it — support, sales, and leadership need the impact and the prevention plan, not the stack traces.
Findings that apply to other services are routed to those teams explicitly — the same missing timeout usually exists in four other places.
Runbooks and alerts are updated as part of closing the postmortem — the knowledge is only durable once it lives where the next responder will look.
Postmortems are reviewed in aggregate at least quarterly — themes across ten incidents reveal systemic problems that no single review can see.
Action item completion rate is tracked as a team metric — a low completion rate is the single best predictor that the same incident will happen again.

Sign-off

Area	Reviewer	Date	Outcome
Scope, ownership, and timing			Pass / Pass with actions / Fail
Blameless writing			Pass / Pass with actions / Fail
Timeline construction			Pass / Pass with actions / Fail
Impact			Pass / Pass with actions / Fail
Contributing factors and root cause			Pass / Pass with actions / Fail
Action items			Pass / Pass with actions / Fail
The review meeting			Pass / Pass with actions / Fail
Publication and organisational learning			Pass / Pass with actions / Fail

Do not publish the postmortem until every row reads Pass or has a dated ticket attached to its outstanding actions.

Backup and Recovery

Mon, 01 Jan 0001 00:00:00 +0000

Nobody wants backups; everybody wants restores. The distinction matters because a backup job that reports success every night proves only that a job ran, not that the data inside it is readable, complete, or recoverable within the window the business assumes. This checklist works from what you are protecting, through how it is stored and secured, to the only test that counts.

info

Who runs this: the service owner together with whoever operates the data platform, reviewed by someone outside the team. When: before go-live, after any change to storage or retention, and at least once a year regardless.

1. Scope and inventory

Every data store the service depends on is listed — databases, object storage, search indexes, message queues with durable state, and the caches that cannot be rebuilt from source.
Configuration and infrastructure state are in scope, not just user data — Terraform state, secret manager contents, DNS zones, IAM policies, and CI/CD configuration are all restore-critical.
Each dataset has a named owner and a classification — the owner decides retention and approves restores, and the classification determines encryption and access requirements.
Data that is deliberately not backed up is documented as an accepted loss — derived data and rebuildable caches are legitimate exclusions, but only when written down.
Dependencies between datasets are recorded — restoring a database without the object storage its rows reference produces a consistent-looking but broken system.
The inventory is generated or verified automatically — a hand-maintained list will silently miss the database that was created last quarter.

2. Recovery objectives

An RPO and RTO are agreed per dataset with the business, in writing — engineering cannot invent an acceptable data loss window on the business’s behalf.
The objectives are derived from actual impact, not from what the current tooling happens to deliver — a 24-hour RPO for payments data is a decision, and it should be a conscious one.
The measured recovery time is compared against the RTO after every restore test — most teams discover their real RTO is several times their stated one.
Recovery time includes everything, not just the restore command — detection, decision, provisioning, restore, replay, validation, and cutover all consume the window.
Objectives for a full regional loss are separate from those for a single-table deletion — they are different scenarios with different mechanisms and different costs.
The cost of meeting each objective is known and accepted — a one-minute RPO is achievable and expensive, and the trade-off belongs to the data owner.

3. Backup implementation

Backup frequency actually satisfies the stated RPO — a nightly snapshot cannot deliver a four-hour RPO, no matter what the policy document says.
Point-in-time recovery is enabled where the RPO is shorter than the snapshot interval — continuous transaction log archiving is what closes the gap between snapshots.
Backups are consistent, not merely copied — use the database’s own snapshot mechanism or quiesce writes, since a file-level copy of a running database is often unrecoverable.
Related datasets are captured at a coordinated point in time — otherwise restoring produces referential inconsistency that is discovered days later.
Backup jobs alert on failure and on unexpected success characteristics — a job that completes in a tenth of the usual time and produces a tenth of the usual bytes has failed silently.
Missing backups are alerted on, not just failing ones — a job that stopped being scheduled generates no failure event at all.
Backup duration and size are trended — steady growth predicts the day the job stops fitting inside its window.

4. Storage, isolation, and retention

At least one copy is in a different region or physical location — a backup stored beside the primary shares its failure domain.
At least one copy is in an account or subscription with separate credentials — an attacker or a runaway script with production credentials must not be able to delete the backups.
Object lock or immutability is enabled for the retention period — this is the control that survives ransomware and a compromised administrator, and it cannot be added after the fact.
Deletion of backups requires a second approval and is logged — and lifecycle rules that expire data are reviewed, since a misconfigured rule deletes quietly and permanently.
Retention satisfies both recovery needs and legal obligations — long enough to recover from corruption discovered late, and short enough to satisfy data protection commitments.
Retention is enforced by an automated job rather than by intention — unbounded retention is a compliance liability and an unbounded bill.
Storage costs are attributed and reviewed — backup storage grows monotonically and is a common source of budget surprises.

report

Blocking: if the credentials that run production can also delete the backups, you do not have backups. Immutable storage in a separately credentialed account is the minimum bar for anything you would be unwilling to lose.

5. Encryption and access control

Backups are encrypted at rest and in transit — including snapshots, exports, and any temporary staging bucket used during the copy.
Encryption keys are stored separately from the backups and are themselves backed up — an unrecoverable key makes an intact backup worthless, which is the most avoidable form of total data loss.
Key rotation does not orphan older backups — verify that a backup taken before the last rotation is still decryptable, by decrypting one.
Access to backups is least-privilege and audited — a full backup is the most concentrated copy of your data that exists, and it is frequently the least protected.
Restores into non-production environments anonymise or mask personal data — copying production data into a development account is a common and serious breach path.
Break-glass access to backups is documented and tested — including how to restore when the normal identity provider is the thing that is down.

6. Restore testing

A full restore has been performed end to end, into a clean environment — an untested backup is a hypothesis, and the failure rate of first restores is high.
Restore tests run on a schedule, at least quarterly, and after every major version upgrade — a database engine upgrade can silently break compatibility with older backup formats.
The restore is validated by data content, not by job exit code — check row counts, run a checksum or reconciliation query, and exercise a real application read path.
Partial restores are tested as well as full ones — the far more common real-world case is one table or one bucket prefix deleted by mistake.
Point-in-time recovery is tested by restoring to a specific timestamp — including replaying transaction logs, which is the step that usually turns out to be misconfigured.
The restore is performed by someone who did not build it, following only the runbook — this is how you find the undocumented step that lives in one person’s head.
Measured restore time is recorded and compared to the RTO — with any gap raised as a ticket rather than quietly normalised.

7. Runbooks and decision-making

A restore runbook exists per dataset with exact commands — during a data-loss incident nobody should be reading vendor documentation for the first time.
The runbook states who authorises a restore — overwriting production with a backup is itself destructive and needs a named decision-maker.
Rolling back to a backup and rolling forward are both covered — including how to reconcile writes that happened after the restore point.
The runbook is stored where it can be read while the primary systems are down — an incident guide that only exists in the wiki hosted on the failed cluster is not a guide.
Vendor support paths and account identifiers are recorded in the runbook — some restores require the provider, and finding the support contract number under pressure wastes the window.
Communication expectations during a restore are defined — restores are slow, and stakeholders need a cadence rather than silence.

8. Failover and disaster scenarios

Loss of an entire region has a documented and rehearsed response — including how DNS, certificates, and secrets are made available in the recovery region.
The recovery environment can actually be provisioned — check service quotas, instance availability, and image replication in the target region before you need them.
Failback is planned, not just failover — returning to the primary region with reconciled data is usually the harder half and is frequently unplanned.
Ransomware and malicious deletion are treated as distinct scenarios — they require immutable copies and a known-clean recovery point, not simply the most recent backup.
Dependencies on third parties are included in the disaster plan — an identity provider, payment processor, or DNS provider outage needs a documented response of its own.
A full disaster recovery exercise is run at least annually with the outcome recorded — with findings ticketed exactly as they would be after a real incident.

Sign-off

Area	Reviewer	Date	Outcome
Scope and inventory			Pass / Pass with actions / Fail
Recovery objectives			Pass / Pass with actions / Fail
Backup implementation			Pass / Pass with actions / Fail
Storage, isolation, and retention			Pass / Pass with actions / Fail
Encryption and access control			Pass / Pass with actions / Fail
Restore testing			Pass / Pass with actions / Fail
Runbooks and decision-making			Pass / Pass with actions / Fail
Failover and disaster scenarios			Pass / Pass with actions / Fail

Record the date and measured duration of the most recent successful restore alongside the sign-off, and treat any area without a completed restore test as Fail.

Capacity Planning

Mon, 01 Jan 0001 00:00:00 +0000

Capacity planning is the discipline of knowing, before your users find out, where the system breaks and how long you have until you get there. It sits between load testing and cost management: measure what you have, forecast what you will need, and make sure the difference is bought or engineered with time to spare. This checklist follows that order.

info

Who runs this: the service owner with input from finance and whoever owns the infrastructure budget. When: quarterly, before any expected demand event, and after any architectural change that alters the constraining resource.

1. Demand model

The unit of demand is defined in business terms — orders per minute, active sessions, or documents indexed, so that a product forecast translates directly into infrastructure.
The peak-to-average ratio is measured, not assumed — provisioning to the daily average guarantees failure at the daily peak, which is the only hour that matters.
Seasonal and weekly patterns are documented — month-end batch runs, payday spikes, and campaign traffic each have a shape that averages hide entirely.
Known future events are on a calendar with expected multipliers — marketing launches, partner integrations, migrations, and regulatory deadlines all have owners who can be asked.
Growth is forecast from at least twelve months of history where it exists — a trend fitted to one quarter will mislead in either direction.
Internal demand is modelled alongside customer demand — analytics jobs, backfills, and your own retrying clients frequently dominate the peak.
The forecast records its assumptions explicitly — so that when reality diverges you can tell which assumption was wrong rather than rebuilding it from scratch.

2. Current utilisation and the constraining resource

The constraining resource is identified per service — CPU, memory, disk IOPS, network, database connections, or a third-party rate limit; everything else is secondary.
Utilisation is measured at peak, not averaged over the day — an instance at 40% daily average can be saturated for two hours every evening.
Utilisation is tracked per instance as well as in aggregate — an unbalanced shard or a hot partition saturates while the fleet average looks comfortable.
Resources that are consumed monotonically have their own tracking — disk usage, table sizes, and index growth only ever go one way and produce entirely predictable outages.
Connection pools and thread pools are counted as capacity — database connection exhaustion is a far more common ceiling than raw CPU.
Provider quotas and service limits are inventoried — API rate limits, instances per region, IP addresses, and load balancer rules all become the binding constraint eventually.
Efficiency is tracked as demand per unit of resource — this is what tells you whether growth in cost is growth in traffic or growth in waste.

3. Headroom and safety margins

A target headroom is defined and justified — enough to absorb the loss of an availability zone plus the largest plausible traffic spike, not a number copied from another team.
Headroom accounts for the failure of the largest single unit — losing one of three zones means the surviving two must carry the full peak.
The time required to add capacity is measured — if provisioning takes two hours, headroom must cover two hours of growth plus the detection delay.
Alerts fire on projected exhaustion, not only on current utilisation — an alert saying disk will be full in six days is actionable; one saying disk is 95% full is an incident.
Headroom is validated after each scaling change — reducing instance count for cost reasons quietly consumes the margin that was there for zone failure.
Stateful components have separate, larger margins — adding a database replica or resharding takes hours or days, unlike adding a stateless instance.

4. Load testing and the breaking point

Load tests run against production-like infrastructure — same instance classes, same database tier, same network topology, or the results describe a system you do not operate.
The test uses a realistic traffic mix — replayed or modelled from production, since a single-endpoint benchmark tells you nothing about contention between endpoints.
The breaking point is known as a number — the request rate at which latency crosses the SLO, and which component fails first when you get there.
Behaviour beyond the breaking point is characterised — the system should degrade and shed load rather than collapse into a retry storm it cannot recover from.
Recovery from overload is tested, not just the overload itself — many systems survive the spike and then fall over on the queued backlog when traffic returns to normal.
Soak tests run long enough to reveal leaks — memory growth, file descriptor exhaustion, and connection leaks appear over hours, not over a ten-minute test.
Load test results are dated and re-run after significant changes — a benchmark from two releases ago describes a system that no longer exists.

warning

Common mistake: load testing one service in isolation. The interesting failures happen at shared dependencies — the database, the cache, the identity provider — under the combined load of every caller, which single-service tests never produce.

5. Scaling mechanisms

Autoscaling triggers on the constraining resource — scaling on CPU when the bottleneck is database connections adds load to the thing that is already failing.
Scale-up is fast and scale-down is slow — aggressive scale-down causes thrashing and leaves you short at the start of the next spike.
Maximum scaling limits are set and are affordable — check the ceiling against the monthly budget, not only against the technical limit.
The system has been observed actually scaling under load — autoscaling configuration is frequently wrong in ways only a real scale event reveals.
Cold start and warm-up time are accounted for — an instance that takes three minutes to become useful cannot respond to a sixty-second spike.
Scaling one tier does not overwhelm another — more application instances mean more database connections, and the database rarely scales on the same timescale.
Manual scaling procedures exist for when autoscaling fails or is too slow — documented, and tested by someone who is not the person who wrote them.

6. Protection under overload

Rate limiting protects the service from any single client — including your own internal callers and your own retrying clients, which are the usual culprits.
Load shedding drops the least important work first — a documented priority order lets you keep checkout working while analytics queries are rejected.
Backpressure propagates rather than being absorbed silently — an unbounded internal queue converts an overload into an unbounded latency increase and eventually a memory failure.
Retries have a budget and use exponential backoff with jitter — synchronised naive retries turn a brief slowdown into a self-inflicted denial of service.
Circuit breakers protect against a slow dependency — timeouts alone still consume every thread while waiting.
Rejection under load is fast and cheap — a request that is going to fail should fail immediately rather than consuming a connection for thirty seconds first.

7. Cost and procurement

Capacity cost per unit of demand is calculated — cost per thousand requests or per active user is the number that makes a forecast financially meaningful.
The forecast is translated into a budget and shared with finance before the spend arrives — capacity surprises are usually organisational failures rather than technical ones.
Committed-use discounts and reservations are sized against the forecast floor, not the peak — over-committing is as costly as not committing at all.
Lead times for anything not instantly available are known — GPU capacity, reserved instances, licences, and physical hardware all have procurement delays measured in weeks.
Budget alerts are configured at thresholds that leave time to react — an alert at 100% of budget is a report, not a warning.
Idle and over-provisioned resources are reviewed regularly — waste consumes the budget that would otherwise fund genuine headroom.

8. Review cadence and ownership

Capacity is reviewed on a fixed schedule with a named owner — quarterly for most services, monthly for anything growing quickly.
Forecast accuracy is measured against actuals each cycle — a forecast that is never scored never improves.
A dashboard shows current utilisation against forecast demand and the breaking point — so headroom is visible without anyone having to assemble it.
Product roadmap changes trigger a capacity review — a new feature that multiplies read volume needs to be known before it ships, not after.
Capacity findings feed the risk register with owners and dates — an exhaustion date more than one review cycle away is a plan; one inside the cycle is an incident waiting.
On-call engineers know the current headroom and where the limits are — this is the context that turns an ambiguous latency page into a fast diagnosis.

Sign-off

Area	Reviewer	Date	Outcome
Demand model			Pass / Pass with actions / Fail
Current utilisation and the constraining resource			Pass / Pass with actions / Fail
Headroom and safety margins			Pass / Pass with actions / Fail
Load testing and the breaking point			Pass / Pass with actions / Fail
Scaling mechanisms			Pass / Pass with actions / Fail
Protection under overload			Pass / Pass with actions / Fail
Cost and procurement			Pass / Pass with actions / Fail
Review cadence and ownership			Pass / Pass with actions / Fail

Record the projected exhaustion date for the constraining resource alongside the sign-off, and raise a dated ticket for every item marked “Pass with actions”.

Data Pipeline Review

Mon, 01 Jan 0001 00:00:00 +0000

Most data pipelines do not fail loudly. They succeed, quietly emit the wrong numbers, and nobody notices until a finance report disagrees with itself three weeks later. This review is about making a pipeline re-runnable, correct under late and duplicated input, and observable enough that a silent failure becomes a page. Work through it before the pipeline feeds a dashboard, a model, or a downstream team.

info

Who runs this: the owning data engineer plus one reviewer who consumes the output. When: before the pipeline is promoted to production, and again whenever its source system or partitioning scheme changes.

1. Contracts and sources

Every source has a named owner and a documented delivery expectation — what arrives, how often, and by what time, so a missing file is a broken promise rather than a surprise.
The read pattern from each source is agreed with its owner — replica versus primary, snapshot versus CDC, and whether your query load is acceptable at their peak.
A data contract exists for each critical input — column names, types, nullability, and semantic meaning, checked in with the pipeline rather than living in a wiki page nobody edits.
The pipeline pins an explicit column list rather than SELECT * — a new upstream column should not silently widen your table or reorder positional consumers.
Source-side deletes are handled deliberately — decide whether a row disappearing upstream means soft delete, hard delete, or ignore, because CDC and snapshot loads behave differently here.
Timezone and clock semantics are documented per source — whether timestamps are UTC, local, or session-dependent, and which clock (event, ingest, or processing) each column represents.

2. Idempotency and reprocessing

Every ingestion job is idempotent on its partition key — re-running a failed day must produce the same table state, not a second copy of the rows.
Writes use delete-and-insert per partition, MERGE on a stable key, or an atomic partition swap — plain INSERT into a shared table is what turns a retry into duplicated revenue.
The natural key that identifies a record is defined and enforced — a uniqueness test on that key is the cheapest duplicate detector you will ever write.
Re-running a task never depends on the outcome of a previous run — no in-place counters, no reading the target table to decide what to write, no state hidden in a scratch file.
Output is written to a temporary location and promoted atomically — a job that dies halfway must not leave a half-written table that a consumer can read.
Reprocessing a historical window is a documented, parameterised operation — not a hand-edited copy of the DAG with the dates changed.

warning

Blocking: a pipeline whose re-run duplicates rows must not go to production. Every other item here can carry a dated follow-up ticket; non-idempotent writes cannot, because the first retry silently corrupts the table.

3. Late and out-of-order data

Event time and processing time are separate columns — conflating them makes every late arrival look like it happened when you happened to read it.
The allowed lateness window is an explicit, documented number — and it is derived from measured arrival distributions, not from a round number that felt safe.
Records arriving after the lateness window are routed somewhere visible — a side-output table or dead-letter topic, counted and alerted on, never silently dropped.
Aggregates over a window are recomputed when late data lands — or the window is explicitly declared immutable and consumers are told that late facts are lost.
Watermark or high-water-mark logic is tested against out-of-order input — replay a shuffled batch in staging and assert the output matches the ordered run.
Restatement of already-published numbers has an agreed process — who is told, how the correction is versioned, and whether downstream extracts are refreshed.

4. Schema evolution

Additive schema changes flow through without a manual deploy — a new nullable upstream column should not break the load, and it should not silently vanish either.
Breaking changes are detected before the write, not after — a type change or dropped column fails the run with a clear error rather than writing nulls.
A schema registry or equivalent enforces compatibility for streaming sources — backward compatibility for consumers, forward compatibility for producers, chosen deliberately.
Column type widening is planned rather than discovered — an integer id that becomes a string upstream will otherwise fail at the worst possible time.
Schema changes are versioned in the repository and reviewed like code — including the migration for any already-written historical partitions.
Downstream consumers are notified before a column is renamed or removed — with a deprecation period long enough for them to actually act.

5. Partitioning, storage, and layout

The partition column matches the dominant query filter — partitioning by ingest date while every consumer filters on event date guarantees full scans.
Partition granularity is chosen against data volume — hourly partitions on a small table produce millions of tiny files and a metadata problem worse than the scan you avoided.
Small-file compaction runs on a schedule for streaming or micro-batch outputs — file count, not row count, is what eventually makes the table unqueryable.
File format and compression are deliberate — columnar formats such as Parquet or ORC for analytical reads, with a codec chosen for the read/write ratio you actually have.
Table statistics or manifests are refreshed after each load — a stale catalogue makes the query planner choose badly and hides newly written partitions.
Retention and archival are enforced by a job — an unbounded table is a cost problem and, where personal data is involved, a compliance problem.

6. Orchestration, retries, and dependencies

Tasks declare their data dependencies rather than relying on schedule ordering — a job that starts at 02:00 because the upstream usually finishes at 01:45 will eventually read yesterday’s data.
Retries use bounded attempts with exponential backoff — and retry only on transient errors, because retrying a schema violation twelve times just delays the alert.
A task that is already running cannot be started twice — concurrency limits and run-key locking prevent two backfills writing the same partition simultaneously.
Task timeouts are set below the point at which the run blocks the next scheduled run — a hung task with no timeout silently stops the whole schedule.
Sensors and polling have a deadline and a failure path — an infinite wait for a file that will never arrive is an outage that never pages anyone.
The DAG can be resumed from the failed task — a four-hour pipeline that must restart from step one after a step-nine failure will not be recoverable during an incident.
Credentials come from a secret manager at run time — not from connection strings in the DAG file or variables committed to the repository.

7. Backfills

A backfill runs through the same code path as the scheduled run — a separate backfill script drifts from production logic and produces history that does not match the present.
Backfills are rate-limited and resource-capped — an unthrottled 400-day backfill will saturate the warehouse and take the daily load down with it.
The backfill is tested on one partition and verified before the full range is launched — compare row counts and a few aggregates against the existing data.
Downstream consumers are informed before a backfill overwrites published history — dashboards and extracts change underneath people otherwise.
Backfills can be paused and resumed — and a partially completed backfill leaves the table in a consistent, identifiable state.
The cost of the backfill is estimated before it is run — full-history reprocessing on a metered warehouse is one of the most common surprise invoices.

8. Freshness, quality gates, and monitoring

A freshness SLA is defined per output table — the maximum acceptable age of the newest record, agreed with the consumers who depend on it.
Freshness is monitored from the table, not from the scheduler — a green DAG run that wrote zero rows is the failure mode this catches.
Row-count anomaly detection is in place for each load — an absolute floor plus a deviation band against the trailing average catches both empty and duplicated loads.
Quality tests run as a gate before publication, not as a report afterwards — failing tests should stop the swap into the consumer-visible table.
Null rate, distinct count, and range checks are asserted on business-critical columns — the ones that feed revenue, headcount, or a regulatory report.
Pipeline logs are structured and carry the run id and partition — so an investigation starts from a query rather than from scrolling.
Alerts distinguish a failed run from a stale table — they have different causes and different responders, and one can happen without the other.
Column-level lineage is available for critical outputs — when a number looks wrong, tracing it to the source column should take minutes, not a day.

9. Cost and efficiency

The cost per run is measured and attributed to the pipeline — query tags, job labels, or a dedicated warehouse make this attributable rather than a shared mystery.
Incremental processing is used wherever the source supports it — full-table reloads scale linearly with history and eventually stop finishing overnight.
Queries prune partitions as intended — check the query plan or bytes-scanned figure rather than assuming the filter was pushed down.
Compute is sized against the job rather than the largest job on the platform — and it scales down or shuts off when idle.
A budget alert exists for the pipeline’s warehouse or cluster — set at a threshold that leaves time to react before month end.

Sign-off

Area	Reviewer	Date	Outcome
Contracts and sources			Pass / Pass with actions / Fail
Idempotency and reprocessing			Pass / Pass with actions / Fail
Late and out-of-order data			Pass / Pass with actions / Fail
Schema evolution			Pass / Pass with actions / Fail
Partitioning, storage, and layout			Pass / Pass with actions / Fail
Orchestration, retries, and dependencies			Pass / Pass with actions / Fail
Backfills			Pass / Pass with actions / Fail
Freshness, quality gates, and monitoring			Pass / Pass with actions / Fail
Cost and efficiency			Pass / Pass with actions / Fail

Record every “Pass with actions” as a dated ticket with a named owner before the pipeline is scheduled in production.

Data Quality

Mon, 01 Jan 0001 00:00:00 +0000

Data quality work fails in a predictable way: a team writes a hundred tests, they all run after publication, half of them are noisy, and everyone learns to ignore the channel they post to. This checklist is about the opposite — a small set of tests that block publication, clear ownership of what “correct” means, and a way to tell whether trust in a dataset is going up or down. Use it when adopting a new critical dataset, or when a set of numbers has lost the confidence of the people who use it.

ML Model Deployment

Mon, 01 Jan 0001 00:00:00 +0000

A model that scores well offline can still be a bad deployment: the features it sees in production differ from the ones it trained on, nobody can reproduce the artefact six months later, and there is no way back once traffic is on it. This checklist treats the model as a production service with unusual failure modes — silent degradation, feedback loops, and correctness that depends on data you do not control. Work through it before the model serves real decisions.

Data Warehouse Migration

Mon, 01 Jan 0001 00:00:00 +0000

Warehouse migrations rarely fail on the technical load. They fail because nobody knew who used the seventeen-year-old table, because the two systems round differently and the numbers never quite matched, or because the old platform stayed switched on for three years after cutover. This checklist follows the migration in the order it actually happens — inventory, build, dual-run, reconcile, cut over, decommission — and assumes you will need to prove equivalence to people who do not trust the new system yet.

Network Change

Mon, 01 Jan 0001 00:00:00 +0000

Network changes are unusual among production changes because the thing you are modifying is the same thing carrying your management session. A bad line in an access list, a VLAN typo, or a routing policy that withdraws the wrong prefix can remove your own path to the device a fraction of a second after you press enter. This checklist is a change-window runbook: capture a baseline, understand the blast radius, arrange a way back in, make the change, verify it, and know exactly how to undo it.

DNS Migration

Mon, 01 Jan 0001 00:00:00 +0000

DNS migrations are unforgiving because the mistake is invisible at the moment you make it and becomes visible to users hours later, spread unevenly across the internet as caches expire. The work that determines whether a migration is boring happens days before the cutover: lowering TTLs, inventorying every record, and proving the new provider answers identically to the old one. This checklist is ordered in time, from roughly a week out to a week after.

TLS Certificate Management

Mon, 01 Jan 0001 00:00:00 +0000

Almost every certificate outage has the same shape: something was issued once by hand, the person who did it moved on, and the calendar reminder went to a mailbox nobody reads. The rest are caused by an incomplete chain that works in a browser and fails in every non-browser client. This checklist covers issuance and key handling, chain and SAN correctness, automated renewal and the monitoring that proves it is working, the TLS configuration itself, and what to do when a certificate expires anyway.

Load Balancer Configuration

Mon, 01 Jan 0001 00:00:00 +0000

A load balancer is the component most likely to be blamed for an outage and least likely to have caused it on its own. Its defaults are chosen to be safe for a generic workload, and almost every one of them — health check interval, idle timeout, draining period, stickiness — interacts with something specific about your backend. This checklist works through those interactions, with particular attention to the relationship between the load balancer’s timeouts and the backend’s own.

GDPR Readiness

Mon, 01 Jan 0001 00:00:00 +0000

GDPR compliance is not a document exercise. A supervisory authority asking questions after a complaint will want to see the record of processing, the retention job that actually deletes data, the ticket trail behind a subject access request, and the transfer assessment behind the analytics vendor in the United States. This checklist covers the engineering and operational work that makes those artefacts real rather than aspirational, and is aimed at teams that build and run the systems where personal data lives.

ISO/IEC 27001 ISMS

Mon, 01 Jan 0001 00:00:00 +0000

An ISO/IEC 27001 certification audit does not test whether your security is good. It tests whether you have a management system that identifies risk, treats it deliberately, checks itself, and corrects itself — and whether you can prove all of that with records. Most first-time failures are not control failures; they are missing evidence that the management cycle actually ran. This checklist walks the clauses and the Annex A implementation work in the order an auditor tends to open them.

SOC 2 Audit Readiness

Mon, 01 Jan 0001 00:00:00 +0000

A SOC 2 Type II report is an opinion on whether your controls operated effectively over a period, and the auditor forms that opinion by sampling evidence from across the window. The window is unforgiving: a control that started working in month five leaves four months of exceptions that cannot be repaired afterwards. This checklist is about getting the control set, the narratives, and the evidence pipeline in place before the clock starts, and keeping them intact while it runs.

Vendor Security Assessment

Mon, 01 Jan 0001 00:00:00 +0000

Most organisations lose control of their data at the third hop: a vendor you assessed uses a sub-processor you never saw, running in a region you did not approve. Third-party assessment is worth doing properly only if the effort is proportionate to what the vendor can actually reach, and only if the assessment carries through into the contract, the monitoring, and the day the relationship ends. This checklist covers that full lifecycle for engineering and procurement teams.

Employee IT Onboarding

Mon, 01 Jan 0001 00:00:00 +0000

Onboarding is the moment an organisation decides how much access a stranger gets, usually under time pressure and usually by copying somebody else’s permissions. Done well it is boring: the account, the device, and the entitlements all exist before the person does, and each one is traceable to an approved request. Work through this in time order — before day one, day one, first week, first month — and treat the manager attestation at the end as the real completion criterion, not the laptop handover.

Employee IT Offboarding

Mon, 01 Jan 0001 00:00:00 +0000

Offboarding is judged on two axes: speed and completeness. Speed matters because the risk window is the gap between the person’s last moment of goodwill and the moment their last credential stops working. Completeness matters because attackers and disgruntled leavers do not use the front door you remembered to lock — they use the forgotten local account, the personal access token, the shared credential nobody rotated. Run this the same way every time, whether the departure is amicable or not.

Change Management

Mon, 01 Jan 0001 00:00:00 +0000

Change management exists to stop two failure modes: the risky change that nobody reviewed, and the review process so heavy that people route around it. The balance comes from classification — most changes should be pre-approved standard changes that flow through automation, leaving human attention for the genuinely novel ones. Use this checklist for a normal change heading to a change advisory board, and use the emergency section when something is already on fire.

Disaster Recovery Drill

Mon, 01 Jan 0001 00:00:00 +0000

A disaster recovery plan that has never been executed is a document, not a capability. The point of a drill is to convert assumptions into measurements: how long recovery actually takes, how much data is actually lost, and which step in the runbook turns out to reference a server that was decommissioned two years ago. Run this in time order — choose the scenario, bound the blast radius, tell the right people, execute, measure, and write it up — and resist the urge to pick a scenario you already know you will pass.

IT Asset Management

Mon, 01 Jan 0001 00:00:00 +0000

Asset management is unglamorous until the moment it matters: a vulnerability advisory lands and you need to know which machines are affected, a vendor opens a licence audit, or a leaver fails to return a laptop that nobody recorded issuing. The register is not the goal — an accurate register that reconciles against the systems that actually see the assets is. Work through this as a periodic review of the asset management practice rather than as a one-off inventory exercise.

IT Checklists

Web Application Security Review

1. Authentication link

2. Session management link

3. Authorisation and access control link

4. Input handling and injection link

5. Output encoding and browser-side defences link

6. Business logic and abuse resistance link

7. Data protection and privacy link

8. Dependencies and configuration link

Sign-off link

Cloud Security Posture

Security Code Review

Secrets Management

Security Incident Response

Penetration Test Readiness

Production Readiness Review

1. Ownership and documentation link

2. Reliability and failure behaviour link

3. Observability link

4. Alerting link

5. Capacity and performance link

6. Data and state link

7. Security link

8. Release and rollback link

9. Cost link

Sign-off link

CI/CD Pipeline Review

1. Source control and change flow link

2. Build reproducibility link

3. Pipeline credentials and permissions link

4. Third-party actions and runners link

5. Testing and quality gates link

6. Security scanning in the pipeline link

7. Artefact management and provenance link

8. Deployment and rollback link

9. Pipeline reliability and maintenance link

Sign-off link

Container Image Hardening

1. Base image selection link

2. Build hygiene link

3. Secrets and build-time data link

4. Runtime user and filesystem link

5. Image content and dependencies link

6. Image metadata and identity link

7. Runtime configuration in the image link

8. Registry and distribution link

Sign-off link

Kubernetes Deployment

1. Workload definition link

2. Resource requests and limits link

3. Health probes and lifecycle link

4. Scheduling and availability link

5. Configuration and secrets link

6. Pod and container security link

7. Networking and traffic link

8. Storage and state link

9. Rollout, observability, and operations link

Sign-off link

Infrastructure as Code Review

1. Repository structure and module design link

2. State management link

3. The plan output link

4. Secrets and sensitive data link

5. Security and compliance of the resources link

6. Change safety and reversibility link

7. Automation and access control link

8. Documentation and operability link

Sign-off link

Release Day Runbook

1. T-1 day: scope and readiness link

2. T-1 day: communication and logistics link

3. T-1 day: rollback preparation link

4. Go/no-go, immediately before the window link

5. Deploy window: execution link

6. Immediately after rollout: verification link

7. Soak period: monitoring link

8. Rollback decision link

9. Release closure link

Sign-off link

1. Authentication

2. Session management

3. Authorisation and access control

4. Input handling and injection

5. Output encoding and browser-side defences

6. Business logic and abuse resistance

7. Data protection and privacy

8. Dependencies and configuration

Sign-off

1. Ownership and documentation

2. Reliability and failure behaviour

3. Observability

4. Alerting

5. Capacity and performance

6. Data and state

7. Security

8. Release and rollback

9. Cost

Sign-off

1. Source control and change flow

2. Build reproducibility

3. Pipeline credentials and permissions

4. Third-party actions and runners

5. Testing and quality gates

6. Security scanning in the pipeline

7. Artefact management and provenance

8. Deployment and rollback

9. Pipeline reliability and maintenance

Sign-off

1. Base image selection

2. Build hygiene

3. Secrets and build-time data

4. Runtime user and filesystem

5. Image content and dependencies

6. Image metadata and identity

7. Runtime configuration in the image

8. Registry and distribution

Sign-off

1. Workload definition

2. Resource requests and limits

3. Health probes and lifecycle

4. Scheduling and availability

5. Configuration and secrets

6. Pod and container security

7. Networking and traffic

8. Storage and state

9. Rollout, observability, and operations

Sign-off

1. Repository structure and module design

2. State management

3. The plan output

4. Secrets and sensitive data

5. Security and compliance of the resources

6. Change safety and reversibility

7. Automation and access control

8. Documentation and operability

Sign-off

1. T-1 day: scope and readiness

2. T-1 day: communication and logistics

3. T-1 day: rollback preparation

4. Go/no-go, immediately before the window

5. Deploy window: execution

6. Immediately after rollout: verification

7. Soak period: monitoring

8. Rollback decision

9. Release closure

Sign-off

1. Workload fit

2. Configuration and packaging

3. Cold starts and latency

4. Concurrency, scaling, and downstream protection

5. Event sources, delivery, and idempotency

6. Failure handling

7. Security

8. Observability

9. Deployment, cost, and operations

Sign-off

1. Resources and URL structure

2. HTTP semantics

3. Request and response payloads