Applied Testing Strategies

• 1: Pre-Ship Checklist
• 2: Cross-Cutting Concerns

1 - Pre-Ship Checklist

Use this as a set of prompts for a quick self-audit, not a list of gates that must all pass. Items that don’t apply to a component can be ignored; items the list doesn’t mention but your component clearly needs should be added. Walk back to the pattern or cross-cutting concern that needs attention for any item that prompts a “we should fix that.”

• The bulk of the suite is sociable unit tests that exercise how behaviors collaborate to deliver a domain operation. Solitary unit tests are reserved for genuinely complex pure logic.
• Tests are organized around domain operations, not around classes or methods. Test names read as something a stakeholder would recognize.
• Every public-interface contract (inbound and outbound) has a contract test running in the pipeline.
• Classes are tested through their public methods only. No reflection, no test-only visibility relaxations, no asserting on private state.
• Every consumed external dependency is wrapped in a gateway the team owns; doubles are of the gateway, not of the third-party library.
• Every boundary adapter has an adapter integration test against the real dependency or a high-fidelity stand-in (testcontainer, WireMock with provider fixtures).
• The bulk of testing runs in-band in the pipeline and gates the build; out-of-band checks against real systems run on a schedule and trigger review on failure, never a build break.
• Every test double has a corresponding non-deterministic check that exercises the real dependency on a schedule or post-deploy.
• Every documented failure mode has a negative test.
• Every error response has a test that verifies the error envelope, status code, and any side effects (or absence thereof).
• Time, randomness, and the network are injected, not called directly. No sleep in tests. Use bounded polling or a fake clock.
• All deterministic tests run pre-commit and in CI Stage 1, and fail the build on failure.
• All post-deploy integration checks run out of pipeline and trigger review on failure, never blocking a commit.
• Pipeline gates map to defect sources from the Systemic Defect Fixes catalog. If a defect category has no automated check, that’s a known risk.
• Authn and authz are tested across every protected endpoint, not as one-offs per feature.
• Database migrations are tested forward, backward (where supported), and on representative data volume against the production engine.
• Fixtures are generated from the schema or built through Object Mother / builder helpers, not inline literals.
• Failure-path tests assert on observability (metric incremented, structured log emitted with correlation ID), not just the response.
• Per-endpoint perf budgets exist for hot paths; load tests gate production promotion; soak tests run out of pipeline.
• Flaky tests are quarantined with a dated owner and time-boxed remediation. No permanent quarantine list.
• The deterministic suite respects the pattern’s time budget (under 5 to 8 minutes per component, under 10 minutes total).

2 - Cross-Cutting Concerns

The patterns describe testing organized by component shape. The concerns below cut across all patterns and deserve dedicated coverage in any non-trivial system.

Authn and authz testing

Authentication and authorization deserve dedicated, exhaustive coverage. They are a major source of high-impact incidents and the failure modes are predictable:

• Tenant isolation: tenant A’s queries never return tenant B’s data. Test every read path. Multi-tenant SaaS bugs are almost always missing isolation tests.
• Scope or role escalation: a token with read:orders cannot perform write:orders. Test the matrix of scope and endpoint.
• Expired tokens: rejected even if cached locally. Clock-skew tolerance is a property of the verifier, not a license to skip the test.
• Forged tokens: signature validation actually validates. The classic JWT alg: none bug still ships periodically.
• Missing auth: every protected endpoint returns 401, never 500 (information leak) and never 200 (catastrophic).
• Service-to-service auth: machine identities respected, mTLS validated, token-swapping attacks detected.

The pattern: a parameterized test that takes (endpoint, method, expected-status-when-no-token, expected-status-when-wrong-scope) and runs across every endpoint in the OpenAPI or schema definition. New endpoints are covered automatically.

Database migrations

Migrations have their own discipline. For every migration:

• Forward on representative data: produces the expected schema and data.
• Backward (where supported): returns to the previous schema with no data loss. Expand-contract migrations may not roll back; that’s a design choice the test pins.
• Forward + backward + forward: idempotent.
• Time on production-scale data: budget assertion. A 30-minute migration on a 50M-row table needs a different deploy strategy than a 30-second one.
• Under traffic: the expand-contract pattern doesn’t break in-flight transactions.

Test against the real production database engine and version using testcontainers. SQLite-against-Postgres is a frequent source of “passed in CI, broke at 02:00 in prod” incidents.

Test data and fixtures

Fixtures rot faster than the code that uses them. Two principles keep them honest:

1. Generate fixtures from the schema, not by hand. When the schema is the source of truth (Avro, OpenAPI, SQL DDL, Protobuf), generate fixture builders from it. A type change breaks the build, not production.
2. Use Object Mother or builder patterns, not raw inline literals. A test that says placeOrder(buildValidOrder().withItem("A1", 2).build()) survives a schema change because the builder updates centrally. A test with 30 lines of raw JSON inline does not.

Avoid shared global fixtures that tests mutate. Each test creates the state it needs, names what is essential about that state, and discards the rest.

Observability as a tested artifact

Logs, metrics, and traces are part of a service’s contract with operators. If an alert depends on a metric, the test for the failure path should assert the metric is emitted. If a runbook depends on a structured log line, the test should assert the line is produced with the right fields and correlation ID.

The pattern: in component tests, attach a metrics collector and a log capture to the assembled component. Failure-path tests assert three things at once:

1. The response status is correct.
2. The error metric is incremented with the right labels.
3. The structured log line is emitted with correlation ID, error code, and any fields the runbook depends on.

This prevents silent regressions where the code “works” but the operator can’t see what’s happening when it doesn’t.

Accessibility testing

For any pattern that renders a user interface, accessibility is a functional requirement, not a finishing touch, and it belongs in the same in-band / out-of-band split as every other concern on this page. The dividing line is the one the whole test architecture uses: deterministic checks gate the build; subjective judgment runs continuously and never blocks.

The governing rule: automate the deterministic rules, reserve human judgment for the rest. A large share of WCAG success criteria are machine-checkable - missing alt attributes, invalid or contradictory ARIA, unlabeled form controls, insufficient color contrast, broken heading hierarchy, a missing document language. Those are deterministic and belong in the pipeline. The remainder - whether alt text is meaningful, whether the screen-reader narrative makes sense, whether a flow is actually operable with a keyboard or a switch device - cannot be settled by a tool and must not be faked with one.

Three tiers, mapped to pipeline placement:

1. Static analysis (in-band, blocks build). Accessibility linting catches structural violations in source without rendering: missing alt text, ARIA misuse, label associations, heading order. It runs in the IDE, pre-commit, and CI, exactly like any other static check. Cheapest and fastest; treat high-severity findings as build-breaking, the same as a security finding.
2. Component tests against the rendered DOM (in-band, blocks build). Some violations exist only in the rendered output: contrast computed after CSS resolves, focus order, dynamic ARIA state, keyboard operability. A scanner assertion inside a component test (expect(results).toHaveNoViolations()) plus explicit keyboard-navigation assertions cover these deterministically, on every commit. The user interface pattern shows the full shape.
3. Manual audit and assistive-technology testing (out-of-band, never a gate). Real screen-reader passes, keyboard-only walkthroughs, and expert review of whether the experience is coherent. This is continuous and informs the backlog; like exploratory testing, it is not a pass/fail checkpoint and must not gate a deploy.

The caveat that keeps tiers 1 and 2 honest: automated checks detect only a fraction of WCAG success criteria - industry estimates commonly land between a third and a half, depending on the tool and the page. A green automated scan means “no detectable violations,” not “accessible.” Wiring a scanner into the build is necessary and high-value, but a team that reads a passing scan as proof of accessibility has the same false-confidence problem as a team that reads high line coverage as proof of correctness. The deterministic tiers shrink the manual surface; they do not remove it.

This mirrors observability as a tested artifact above: the machine-verifiable part of a human contract gets pinned in the deterministic suite, and the judgment part stays with people.

Performance and load testing

Three classes of perf tests, each with a different home in the pipeline:

1. Per-endpoint perf budgets in component tests. Simple latency assertion under no load (assertThat(p99).isLessThan(50ms)). Catches algorithmic regressions cheaply. Fits in CI Stage 1 if the assertions are tight and the runtime is stable.
2. Load tests in acceptance. k6, Gatling, or Locust against a deployed instance. Validate p99 latency, throughput, and error rate at expected production load. Gates production promotion.
3. Soak tests out of pipeline. Long-running load to catch memory leaks, file handle leaks, and slow drift. Scheduled, non-blocking.

A perf regression that breaches a documented budget should block deploy. A regression within budget but worse than baseline should generate a finding for review, not a build failure: noisy alerts get ignored.

Mutation testing

Coverage % tells you what code ran. Mutation testing tells you whether the tests would have failed if the code had been wrong. Tools (Stryker for JS, PIT for Java) systematically change operators, return values, and conditionals, then re-run the test suite. Surviving mutants are tests that didn’t catch the mutation.

Each surviving mutant is one of three things:

• A real test gap. Add a flow-oriented test that would have failed when the mutation was applied.
• An equivalent mutant, semantically identical to the original. Mark and move on.
• A trivially equivalent mutant (logging change, assertion message tweak). Configure the tool to skip.

Mutation testing is too slow to run on every commit. Run it nightly or weekly on the highest-value modules. Treat it as a periodic audit of test quality, not a gating check.

Flake handling protocol

A flaky test is a known unknown. Three rules keep flakes from rotting the suite:

1. Quarantine on detection. First flake gets the test moved to a quarantine lane that doesn’t block the build. Don’t ignore it; don’t keep failing builds for unrelated reasons.
2. Time-boxed remediation. Quarantined tests have a deadline (e.g., five business days) and an owner. After the deadline, fix or delete. No silent quarantine.
3. Track the cause. Most flakes share root causes: timing, shared state, network, ordering. The fix is usually structural (eliminate the timing dependency) rather than local (add a longer sleep).

A suite with a permanent quarantine list has lost its CD-ready quality. See also Tests Randomly Pass or Fail.

Cost and time budgets

Empirical starting points for in-band test budgets, based on typical service complexity. Adjust for your codebase, language, framework, and the size of the component under test.

The total CD pipeline in-band suite under 10 minutes is the gating constraint at the team level. The first lever for hitting that budget is parallel execution: the suite should fan out across cores or runners, not run serially. Parallelism only works when tests are independent of each other - no shared mutable state, no ordering dependencies, no global fixtures that one test mutates and another reads. Decoupling tests is a prerequisite for speed, not an optimization on top of it.

If a component’s tests still can’t fit the budget after the suite is running in parallel, the goal is to remediate the underlying cause - slow component startup, oversize fixtures, expensive setup duplicated per test, hidden serialization through a shared resource - not to declare the budget unreachable. While the remediation is underway, moving the offending tests out-of-band on a schedule is a reasonable stopgap so the in-band suite stays fast. Out-of-band placement here is a temporary mitigation, not the destination: those tests should come back in-band once the underlying speed issue is fixed.