Phase 1: Foundations

• 1: Trunk-Based Development
• 1.1: TBD Migration Guide
• 2: Evolutionary Coding Techniques
• 2.1: Dark Code
• 2.2: Branch by Abstraction
• 2.3: Parallel Run
• 2.4: Expand and Contract
• 3: Testing Fundamentals
• 3.1: Test Architecture
• 3.2: What to Test - and What Not To
• 3.3: Pipeline Test Strategy
• 3.4: Getting Started
• 3.5: Defect Feedback Loop
• 3.6: Test Types
• 3.6.1: Component Tests
• 3.6.2: Contract Tests
• 3.6.3: End-to-End Tests
• 3.6.4: Integration Tests
• 3.6.5: Static Analysis
• 3.6.6: Smoke Tests
• 3.6.7: Unit Tests
• 3.7: Patterns
• 3.7.1: API Provider
• 3.7.2: API Consumer
• 3.7.3: Scheduled Job
• 3.7.4: User Interface
• 3.7.5: Event Consumer
• 3.7.6: Event Producer
• 3.7.7: CLI Tool or Library
• 3.7.8: Stateful Service
• 3.8: Applied Testing Strategies
• 3.8.1: Pre-Ship Checklist
• 3.8.2: Cross-Cutting Concerns
• 3.9: Test Feedback Speed
• 3.10: Testing Antipatterns
• 3.11: Testing Glossary
• 4: Build Automation
• 5: Work Decomposition
• 6: Code Review
• 7: Working Agreements
• 8: Everything as Code

1 - Trunk-Based Development

What Is Trunk-Based Development?

Trunk-based development (TBD) is a branching strategy where all developers integrate their work into a single shared branch - the trunk - at least once per day. The trunk is always kept in a releasable state.

This is a non-negotiable prerequisite for continuous delivery. If your team is not integrating to trunk daily, you are not doing CI, and you cannot do CD. There is no workaround.

“If it hurts, do it more often, and bring the pain forward.”

• Jez Humble, Continuous Delivery

What TBD Is Not

• It is not “everyone commits directly to main with no guardrails.” You still test, review, and validate work - you just do it in small increments.
• It is not incompatible with code review. It requires review to happen quickly.
• It is not reckless. It is the opposite: small, frequent integrations are far safer than large, infrequent merges.

What Trunk-Based Development Improves

Two Migration Paths

There are two valid approaches to trunk-based development. Both satisfy the minimum CD requirement of daily integration. Choose the one that fits your team’s current maturity and constraints.

Path 1: Short-Lived Branches

Developers create branches that live for less than 24 hours. Work is done on the branch, reviewed quickly, and merged to trunk within a single day.

How it works:

1. Pull the latest trunk
2. Create a short-lived branch
3. Make small, focused changes
4. Open a pull request (or use pair programming as the review)
5. Merge to trunk before end of day
6. The branch is deleted after merge

Best for teams that:

• Currently use long-lived feature branches and need a stepping stone
• Have regulatory requirements for traceable review records
• Use pull request workflows they want to keep (but make faster)
• Are new to TBD and want a gradual transition

Key constraint: The branch must merge to trunk within 24 hours. If it does not, you have a long-lived branch and you have lost the benefit of TBD.

Path 2: Direct Trunk Commits

Developers commit directly to trunk. Quality is ensured through pre-commit checks, pair programming, and strong automated testing.

How it works:

1. Pull the latest trunk
2. Make a small, tested change locally
3. Run the local build and test suite
4. Push directly to trunk
5. CI validates the commit immediately

Best for teams that:

• Have strong automated test coverage
• Practice pair or mob programming (which provides real-time review)
• Want maximum integration frequency
• Have high trust and shared code ownership

Key constraint: This requires excellent test coverage and a culture where the team owns quality collectively. Without these, direct trunk commits become reckless.

How to Choose Your Path

Ask these questions:

1. Do you have automated tests that catch real defects? If no, start with Path 1 and invest in testing fundamentals in parallel.
2. Does your organization require documented review approvals? If yes, use Path 1 with rapid pull requests.
3. Does your team practice pair programming? If yes, Path 2 may work immediately - pairing is a continuous review process.
4. How large is your team? Teams of 2-4 can adopt Path 2 more easily. Larger teams may start with Path 1 and transition later.

Both paths are valid. The important thing is daily integration to trunk. Do not spend weeks debating which path to use. Pick one, start today, and adjust.

Essential Supporting Practices

Trunk-based development does not work in isolation. These practices make daily integration safe:

• Evolutionary coding techniques: Dark code, branch by abstraction, parallel run, and expand and contract let you merge incomplete work without exposing it to users, in most cases without a flag at all.
• Feature flags: The last-resort technique in that hierarchy, for cases the others can’t cover.
• Small, atomic commits: Each commit is a single logical change that leaves trunk releasable.
• TDD/ATDD: Tests written before code provide the safety net for frequent integration.

The Evolutionary Coding Techniques section and the TBD Migration Guide cover each practice in detail with code examples.

Getting Started

Start by shortening branch lifetimes, then tighten to daily integration. The TBD Migration Guide walks through each step with team agreements, metrics, and retrospective checkpoints.

Common Pitfalls

Teams migrating to TBD commonly stumble on slow CI builds, incomplete feature flags, and treating branch renaming as real integration. See Common Pitfalls to Avoid for detailed guidance and fixes.

Measuring Success

Track these metrics to verify your TBD adoption:

Next Step

Once your team is integrating to trunk daily, build the test suite that makes that integration trustworthy. Continue to Testing Fundamentals.

Related Content

• Evolutionary Coding Techniques - Dark code, branch by abstraction, parallel run, and expand and contract, ordered from least to most costly to maintain
• TBD Migration Guide - Detailed scenarios including regulated environments, multi-team environments, and advanced pitfalls
• Trunk-Based Development - Practice definition and minimum criteria
• trunkbaseddevelopment.com - Comprehensive reference by Paul Hammant
• Painful Merges - Symptom eliminated by integrating to trunk daily
• Merge Freeze - Symptom caused by long-lived branches and infrequent integration
• No Fast Feedback - Symptom that daily integration and CI address directly
• Long-Lived Feature Branches - Anti-pattern that TBD replaces
• Integration Deferred - Anti-pattern where integration is postponed until late in development
• Integration Frequency - Key metric for tracking TBD adoption

1.1 - TBD Migration Guide

This guide walks you through migrating from GitFlow or long-lived branches to trunk-based development. It covers two paths (short-lived branches and direct trunk commits), essential practices, regulated-environment compliance, and common pitfalls.

Long-lived branches hide problems. TBD exposes them early, which is why it is the first step toward continuous integration.

Why Move to Trunk-Based Development?

Long-lived branches hide problems. TBD exposes them early, when they are cheap to fix.

Think of long-lived branches like storing food in a bunker: it feels safe until you open the door and discover half of it rotting. With TBD, teams check freshness every day.

To do CI, teams need:

• Small changes integrated at least daily
• Automated tests giving fast, deterministic feedback
• A single source of truth: the trunk

If your branches live for more than a day or two, you aren’t doing continuous integration. You’re doing periodic integration at best. True CI requires at least daily integration to the trunk.

The First Step: Stop Letting Work Age

The biggest barrier isn’t tooling. It’s habits.

The first meaningful change is simple:

Stop letting branches live long enough to become problems.

Your first goal isn’t true TBD. It’s shorter-lived branches: changes that live for hours or a couple of days, not weeks.

That alone exposes dependency issues, unclear requirements, and missing tests, which is exactly the point. The pain tells you where improvement is needed.

Before You Start: What to Measure

You cannot improve what you don’t measure. Before changing anything, establish baseline metrics, so you can track actual progress.

Essential Metrics to Track Weekly

Start with just two or three of these. Don’t let measurement become its own project.

Path 1: Moving from Long-Lived Branches to Short-Lived Branches

When GitFlow habits are deeply ingrained, this is usually the least-threatening first step.

1. Collapse the Branching Model

Stop using:

• develop
• release branches that sit around for weeks
• feature branches lasting a sprint or more

Move toward:

• A single main (or trunk)
• Temporary branches measured in hours or days

2. Integrate Every Few Days, Then Every Day

Set an explicit working agreement:

“Nothing lives longer than 48 hours.”

Once this feels normal, shorten it:

“Integrate at least once per day.”

If a change is too large to merge within a day or two, the problem isn’t the branching model. The problem is the decomposition of work.

3. Test Before You Code

Branch lifetime shortens when you stop guessing about expected behavior. Bring product, QA, and developers together before coding:

• Write acceptance criteria collaboratively
• Turn them into executable tests
• Then write code to make those tests pass

You’ll discover misunderstandings upfront instead of after a week of coding.

This approach is called Behavior-Driven Development (BDD), a collaborative practice where teams define expected behavior in plain language before writing code. BDD bridges the gap between business requirements and technical implementation by using concrete examples that become executable tests.

Key BDD resources:

• Behavior-Driven Development - Dojo Consortium - Comprehensive guide to BDD practices
• “Specification by Example” by Gojko Adzic - Foundational text on collaborative specification

How to Run a Three Amigos Session

Participants: Product Owner, Developer, Tester (15-30 minutes per story)

Process:

1. Product describes the user need and expected outcome
2. Developer asks questions about edge cases and dependencies
3. Tester identifies scenarios that could fail
4. Together, write acceptance criteria as examples

Example:

Feature: User password reset

Scenario: Valid reset request
  Given a user with email "user@example.com" exists
  When they request a password reset
  Then they receive an email with a reset link
  And the link expires after 1 hour

Scenario: Invalid email
  Given no user with email "nobody@example.com" exists
  When they request a password reset
  Then they see "If the email exists, a reset link was sent"
  And no email is sent

Scenario: Expired link
  Given a user has a reset link older than 1 hour
  When they click the link
  Then they see "This reset link has expired"
  And they are prompted to request a new one

These scenarios become your automated acceptance tests before you write any implementation code.

From Acceptance Criteria to Tests

Turn those scenarios into executable tests in your framework of choice:

// Example using Jest and Supertest
describe('Password Reset', () => {
  it('sends reset email for valid user', async () => {
    await createUser({ email: 'user@example.com' });

    const response = await request(app)
      .post('/password-reset')
      .send({ email: 'user@example.com' });

    expect(response.status).toBe(200);
    expect(emailService.sentEmails).toHaveLength(1);
    expect(emailService.sentEmails[0].to).toBe('user@example.com');
  });

  it('does not reveal whether email exists', async () => {
    const response = await request(app)
      .post('/password-reset')
      .send({ email: 'nobody@example.com' });

    expect(response.status).toBe(200);
    expect(response.body.message).toBe('If the email exists, a reset link was sent');
    expect(emailService.sentEmails).toHaveLength(0);
  });
});

Now you can write the minimum code to make these tests pass. This drives smaller, more focused changes.

4. Invest in Contract Tests

Most merge pain isn’t from your code. It’s from the interfaces between services. Define interface changes early and codify them with provider/consumer contract tests.

This lets teams integrate frequently without surprises.

Path 2: Committing Directly to the Trunk

This is the cleanest and most powerful version of TBD. It requires discipline, but it produces the most stable delivery pipeline and the least drama.

If the idea of committing straight to main makes people panic, that’s a signal about your current testing process, not a problem with TBD.

How to Choose Your Path

Use this rule of thumb:

• If your team fears “breaking everything,” start with short-lived branches.
• If your team collaborates well and writes tests first, go straight to trunk commits.

Both paths require the same skills:

• Smaller work
• Better requirements
• Shared understanding
• Automated tests
• A reliable pipeline

The difference is pace.

Essential TBD Practices

These practices apply to both paths, whether you’re using short-lived branches or committing directly to trunk.

Use Feature Flags the Right Way

Feature flags are one of several evolutionary coding techniques that allow you to integrate incomplete work safely, and the one to reach for last. See Evolutionary Coding Techniques for the full decision hierarchy, including dark code, branch by abstraction, parallel run, and expand and contract.

Feature flags are not a testing strategy. They are a release strategy.

Every commit to trunk must:

• Build
• Test
• Deploy safely

Flags let you deploy incomplete work without exposing it prematurely. They don’t excuse poor test discipline.

Start Simple: Boolean Flags

You don’t need a sophisticated feature flag system to start. Begin with environment variables or simple config files.

Simple boolean flag example:

// config/features.js
module.exports = {
  newCheckoutFlow: process.env.FEATURE_NEW_CHECKOUT === 'true',
  enhancedSearch: process.env.FEATURE_ENHANCED_SEARCH === 'true',
};

// In your code
const features = require('./config/features');

app.get('/checkout', (req, res) => {
  if (features.newCheckoutFlow) {
    return renderNewCheckout(req, res);
  }
  return renderOldCheckout(req, res);
});

This is enough for most TBD use cases.

Testing Code Behind Flags

Critical: You must test both code paths, flag on and flag off.

describe('Checkout flow', () => {
  describe('with new checkout flow enabled', () => {
    beforeEach(() => {
      features.newCheckoutFlow = true;
    });

    it('shows new checkout UI', () => {
      // Test new flow
    });
  });

  describe('with new checkout flow disabled', () => {
    beforeEach(() => {
      features.newCheckoutFlow = false;
    });

    it('shows legacy checkout UI', () => {
      // Test old flow
    });
  });
});

If you only test with the flag on, you’ll break production when the flag is off.

Keep Flags Short-Lived

For TBD, most flags are temporary release flags: they hide incomplete work during integration and get removed once the feature is stable (typically 1-4 weeks). Set a removal date when you create each flag, assign an owner, and treat unremoved flags as technical debt.

For a deeper taxonomy of flag types (release flags vs. permanent configuration flags) and lifecycle management practices, see the feature flag glossary entry.

Commit Small and Commit Often

If a change is too large to commit today, split it.

Large commits are failed design upstream, not failed integration downstream.

Use TDD and ATDD to Keep Refactors Safe

Refactoring must not break tests. If it does, you’re testing implementation, not behavior. Behavioral tests are what keep trunk commits safe.

Prioritize Interfaces First

Always start by defining and codifying the contract:

• What is the shape of the request?
• What is the response?
• What error states must be handled?

Interfaces are the highest-risk area. Drive them with tests first. Then work inward.

Getting Started: A Tactical Guide

The initial phase sets the tone. Focus on establishing new habits, not perfection.

Step 1: Team Agreement and Baseline

• Hold a team meeting to discuss the migration
• Agree on initial branch lifetime limit (start with 48 hours if unsure)
• Document current baseline metrics (branch age, merge frequency, build time)
• Identify your slowest-running tests
• Create a list of known integration pain points
• Set up a visible tracker (physical board or digital dashboard) for metrics

Step 2: Test Infrastructure Audit

Focus: Find and fix what will slow you down.

• Run your test suite and time each major section
• Identify slow tests
• Look for:
  • Tests with sleeps or arbitrary waits
  • Tests hitting external services unnecessarily
  • Integration tests that could be contract tests
  • Flaky tests masking real issues

Fix or isolate the worst offenders. You don’t need a perfect test suite to start, just one fast enough to not punish frequent integration.

Step 3: First Integrated Change

Pick the smallest possible change:

• A bug fix
• A refactoring with existing test coverage
• A configuration update
• Documentation improvement

The goal is to validate your process, not to deliver a feature.

Execute:

1. Create a branch (if using Path 1) or commit directly (if using Path 2)
2. Make the change
3. Run tests locally
4. Integrate to trunk
5. Deploy through your pipeline
6. Observe what breaks or slows you down

Step 4: Retrospective

Gather the team:

What went well:

• Did anyone integrate faster than before?
• Did you discover useful information about your tests or pipeline?

What hurt:

• What took longer than expected?
• What manual steps could be automated?
• What dependencies blocked integration?

Ongoing commitment:

• Adjust branch lifetime limit if needed
• Assign owners to top 3 blockers
• Commit to integrating at least one change per person

The initial phase won’t feel smooth. That’s expected. You’re learning what needs fixing.

Getting Your Team On Board

Technical changes are easy compared to changing habits and mindsets. Here’s how to build buy-in.

Acknowledge the Fear

When you propose TBD, you’ll hear:

• “We’ll break production constantly”
• “Our code isn’t good enough for that”
• “We need code review on branches”
• “This won’t work with our compliance requirements”

These concerns are valid signals about your current system. Don’t dismiss them.

Instead: “You’re right that committing directly to trunk with our current test coverage would be risky. That’s why we need to improve our tests first.”

Start with an Experiment

Don’t mandate TBD for the whole team immediately. Propose a time-boxed experiment:

The Proposal:

“Let’s try this for two weeks with a single small feature. We’ll track what goes well and what hurts. After two weeks, we’ll decide whether to continue, adjust, or stop.”

What to measure during the experiment:

• How many times did we integrate?
• How long did merges take?
• Did we catch issues earlier or later than usual?
• How did it feel compared to our normal process?

After two weeks: Hold a retrospective. Let the data and experience guide the decision.

Pair on the First Changes

Don’t expect everyone to adopt TBD simultaneously. Instead:

1. Identify one advocate who wants to try it
2. Pair with them on the first trunk-based changes
3. Let them experience the process firsthand
4. Have them pair with the next person

Knowledge transfer through pairing works better than documentation.

Address Code Review Concerns

“But we need code review!” Yes. TBD doesn’t eliminate code review.

Options that work:

• Pair or mob programming (review happens in real-time)
• Commit to trunk, review immediately after, fix forward if issues found
• Very short-lived branches (hours, not days) with rapid review SLA
• Pairing on code review and review change

The goal is fast feedback, not zero review.

Handle Skeptics and Blockers

You’ll encounter people who don’t want to change. Don’t force it.

Instead:

• Let them observe the experiment from the outside
• Share metrics and outcomes transparently
• Invite them to pair for one change
• Let success speak louder than arguments

Some people need to see it working before they believe it.

Get Management Support

Managers often worry about:

• Reduced control
• Quality risks
• Slower delivery (ironically)

Address these with data:

• Show branch age metrics before/after
• Track cycle time improvements
• Demonstrate faster feedback on defects
• Highlight reduced merge conflicts

Frame TBD as a risk reduction strategy, not a risky experiment.

Working in a Multi-Team Environment

Migrating to TBD gets complicated when you depend on teams still using long-lived branches. Here’s how to handle it.

The Core Problem

You want to integrate daily. Your dependency team integrates weekly or monthly. Their API changes surprise you during their big-bang merge.

You can’t force other teams to change. But you can protect yourself.

Strategy 1: Consumer-Driven Contract Tests

Define the contract you need from the upstream service and codify it in tests that run in your pipeline.

Example using Pact:

// Your consumer test
const { pact } = require('@pact-foundation/pact');

describe('User Service Contract', () => {
  it('returns user profile by ID', async () => {
    await provider.addInteraction({
      state: 'user 123 exists',
      uponReceiving: 'a request for user 123',
      withRequest: {
        method: 'GET',
        path: '/users/123',
      },
      willRespondWith: {
        status: 200,
        body: {
          id: 123,
          name: 'Jane Doe',
          email: 'jane@example.com',
        },
      },
    });

    const user = await userService.getUser(123);
    expect(user.name).toBe('Jane Doe');
  });
});

This test runs against your expectations of the API, not the actual service. When the upstream team changes their API, your contract test fails before you integrate their changes.

Share the contract:

• Publish your contract to a shared repository
• Upstream team runs provider verification against your contract
• If they break your contract, they know before merging

Strategy 2: API Versioning with Backwards Compatibility

If you control the shared service:

// Support both old and new API versions
app.get('/api/v1/users/:id', handleV1Users);
app.get('/api/v2/users/:id', handleV2Users);

// Or use content negotiation
app.get('/api/users/:id', (req, res) => {
  const version = req.headers['api-version'] || 'v1';
  if (version === 'v2') {
    return handleV2Users(req, res);
  }
  return handleV1Users(req, res);
});

Migration path:

1. Deploy new version alongside old version
2. Update consumers one by one
3. After all consumers migrated, deprecate old version
4. Remove old version after deprecation period

Strategy 3: Strangler Fig Pattern

When you depend on a team that won’t change:

1. Create an anti-corruption layer between your code and theirs
2. Define your ideal interface in the adapter
3. Let the adapter handle their messy API

// Your ideal interface
class UserRepository {
  async getUser(id) {
    // Your clean, typed interface
  }
}

// Adapter that deals with their mess
class LegacyUserServiceAdapter extends UserRepository {
  async getUser(id) {
    const response = await fetch(`https://legacy-service/users/${id}`);
    const messyData = await response.json();

    // Transform their format to yours
    return {
      id: messyData.user_id,
      name: `${messyData.first_name} ${messyData.last_name}`,
      email: messyData.email_address,
    };
  }
}

Now your code depends on your interface, not theirs. When they change, you only update the adapter.

Strategy 4: Feature Toggles for Cross-Team Coordination

When multiple teams need to coordinate a release:

1. Each team develops behind feature flags
2. Each team integrates to trunk continuously
3. Features remain disabled until coordination point
4. Enable flags in coordinated sequence

This decouples development velocity from release coordination.

When You Can’t Integrate with Dependencies

If upstream dependencies block you from integrating daily:

Short term:

• Use contract tests to detect breaking changes early
• Create adapters to isolate their changes
• Document the integration pain as a business cost

Long term:

• Advocate for those teams to adopt TBD
• Share your success metrics
• Offer to help them migrate

You can’t force other teams to change. But you can demonstrate a better way and make it easier for them to follow.

TBD in Regulated Environments

Regulated industries face legitimate compliance requirements: audit trails, change traceability, separation of duties, and documented approval processes. These requirements often lead teams to believe trunk-based development is incompatible with compliance. This is a misconception.

TBD is about integration frequency, not about eliminating controls. You can meet compliance requirements while still integrating at least daily.

The Compliance Concerns

Common regulatory requirements that seem to conflict with TBD:

Audit Trail and Traceability

• Every change must be traceable to a requirement, ticket, or change request
• Changes must be attributable to specific individuals
• History of what changed, when, and why must be preserved

Separation of Duties

• The person who writes code shouldn’t be the person who approves it
• Changes must be reviewed before reaching production
• No single person should have unchecked commit access

Change Control Process

• Changes must follow a documented approval workflow
• Risk assessment before deployment
• Rollback capability for failed changes

Documentation Requirements

• Changes must be documented before implementation
• Testing evidence must be retained
• Deployment procedures must be repeatable and auditable

Short-Lived Branches: The Compliant Path to TBD

Path 1 from this guide (short-lived branches) directly addresses compliance concerns while maintaining the benefits of TBD.

Short-lived branches mean:

• Branches live for hours to 2 days maximum, not weeks or months
• Integration happens at least daily
• Pull requests are small, focused, and fast to review
• Review and approval happen within the branch lifetime

This approach satisfies both regulatory requirements and continuous integration principles.

How Short-Lived Branches Meet Compliance Requirements

Audit Trail:

Every commit references the change ticket:

git commit -m "JIRA-1234: Add validation for SSN input

Implements requirement REQ-445 from Q4 compliance review.
Changes limited to user input validation layer."

Modern Git hosting platforms (GitHub, GitLab, Bitbucket) automatically track:

• Who created the branch
• Who committed each change
• Who reviewed and approved
• When it merged
• Complete diff history

Separation of Duties:

Use pull request workflows:

1. Developer creates branch from trunk
2. Developer commits changes (same day)
3. Second person reviews and approves (within 24 hours)
4. Automated checks validate (tests, security scans, compliance checks)
5. Merge to trunk after approval
6. Automated deployment with gates

This provides stronger separation of duties than long-lived branches because:

• Reviews happen while context is fresh
• Reviewers can actually understand the small changeset
• Automated checks enforce policies consistently

Change Control Process:

Branch protection rules enforce your process:

# Example GitHub branch protection for trunk
required_reviews: 1
required_checks:
  - unit-tests
  - security-scan
  - compliance-validation
dismiss_stale_reviews: true
require_code_owner_review: true

This ensures:

• No direct commits to trunk (except in documented break-glass scenarios)
• Required approvals before merge
• Automated validation gates
• Audit log of every merge decision

Documentation Requirements:

Pull request templates enforce documentation:

## Change Description
[Link to Jira ticket]

## Risk Assessment
- [ ] Low risk: Configuration only
- [ ] Medium risk: New functionality, backward compatible
- [ ] High risk: Database migration, breaking change

## Testing Evidence
- [ ] Unit tests added/updated
- [ ] Integration tests pass
- [ ] Manual testing completed (attach screenshots if UI change)
- [ ] Security scan passed

## Rollback Plan
[How to rollback if this causes issues in production]

What “Short-Lived” Means in Practice

Hours, not days:

• Simple bug fixes: 2-4 hours
• Small feature additions: 4-8 hours
• Refactoring: 1-2 days

Maximum 2 days: If a branch can’t merge within 2 days, the work is too large. Decompose it further or use feature flags to integrate incomplete work safely.

Daily integration requirement: Even if the feature isn’t complete, integrate what you have:

• Behind a feature flag if needed
• As internal APIs not yet exposed
• As tests and interfaces before implementation

Compliance-Friendly Tooling

Modern platforms provide compliance features built-in:

Git Hosting (GitHub, GitLab, Bitbucket):

• Immutable audit logs
• Branch protection rules
• Required approvals
• Status check enforcement
• Signed commits for authenticity

Pipeline Platforms:

• Deployment approval gates
• Audit trails of every deployment
• Environment-specific controls
• Automated compliance checks

Feature Flag Systems:

• Change deployment without code deployment
• Gradual rollout controls
• Instant rollback capability
• Audit log of flag changes

Secrets Management:

• Vault, AWS Secrets Manager, Azure Key Vault
• Audit log of secret access
• Rotation policies
• Environment isolation

Example: Compliant Short-Lived Branch Workflow

Monday 9 AM: Developer creates branch feature/JIRA-1234-add-audit-logging from trunk.

Monday 9 AM to 2 PM: Developer implements audit logging for user authentication events. Commits reference JIRA-1234. Automated tests run on each commit.

Monday 2 PM: Developer opens pull request:

• Title: “JIRA-1234: Add audit logging for authentication events”
• Description includes risk assessment, testing evidence, rollback plan
• Automated checks run: tests, security scan, compliance validation
• Code owner automatically assigned for review

Monday 3 PM: Code owner reviews (5-10 minutes; change is small and focused). Suggests minor improvement.

Monday 3:30 PM: Developer addresses feedback, pushes update.

Monday 4 PM: Code owner approves. All automated checks pass. Developer merges to trunk.

Monday 4:05 PM: Pipeline deploys to staging automatically. Automated smoke tests pass.

Monday 4:30 PM: Deployment gate requires manual approval for production. Tech lead approves based on risk assessment.

Monday 4:35 PM: Automated deployment to production. Audit log captures: what deployed, who approved, when, what checks passed.

Total time: 7.5 hours from branch creation to production.

Full compliance maintained. Full audit trail captured. Daily integration achieved.

When Long-Lived Branches Hide Compliance Problems

Ironically, long-lived branches often create compliance risks:

Stale Reviews: Reviewing a 3-week-old, 2000-line pull request is performative, not effective. Reviewers rubber-stamp because they can’t actually understand the changes.

Integration Risk: Big-bang merges after weeks introduce unexpected behavior. The change that was reviewed isn’t the change that actually deployed (due to merge conflicts and integration issues).

Delayed Feedback: Problems discovered weeks after code was written are expensive to fix and hard to trace to requirements.

Audit Trail Gaps: Long-lived branches often have messy commit history, force pushes, and unclear attribution. The audit trail is polluted.

Regulatory Examples Where Short-Lived Branches Work

Financial Services (SOX, PCI-DSS):

• Short-lived branches with required approvals
• Automated security scanning on every PR
• Separation of duties via required reviewers
• Immutable audit logs in Git hosting platform
• Feature flags for gradual rollout and instant rollback

Healthcare (HIPAA):

• Pull request templates documenting PHI handling
• Automated compliance checks for data access patterns
• Required security review for any PHI-touching code
• Audit logs of deployments
• Environment isolation enforced by the pipeline

Government (FedRAMP, FISMA):

• Branch protection requiring government code owner approval
• Automated STIG compliance validation
• Signed commits for authenticity
• Deployment gates requiring authority to operate
• Complete audit trail from commit to production

What Will Hurt (At First)

When you migrate to TBD, you’ll expose every weakness you’ve been avoiding:

• Slow tests
• Unclear requirements
• Fragile integration points
• Architecture that resists small changes
• Gaps in automated validation
• Long manual processes in the value stream

This is not a regression. This is the point.

Problems you discover early are problems you can fix cheaply.

Common Pitfalls to Avoid

Teams migrating to TBD often make predictable mistakes. The table below summarizes all ten; the three most critical are expanded afterward.

Pitfall 1: Treating TBD as Just a Branch Renaming Exercise

The mistake: Renaming develop to main and calling it TBD.

Why it fails: You’re still doing long-lived feature branches, just with different names. The fundamental integration problems remain.

What to do instead: Focus on integration frequency, not branch names. Measure time-to-merge, not what you call your branches.

Pitfall 2: Merging Daily Without Actually Integrating

The mistake: Committing to trunk every day, but your code doesn’t interact with anyone else’s work. Your tests don’t cover integration points.

Why it fails: You’re batching integration for later. When you finally connect your component to the rest of the system, you discover incompatibilities.

What to do instead: Ensure your tests exercise the boundaries between components. Use contract tests for service interfaces. Integrate at the interface level, not just at the source control level.

Pitfall 5: Keeping Flags Forever

The mistake: Creating feature flags and never removing them. Your codebase becomes a maze of conditionals.

Why it fails: Every permanent flag doubles your testing surface area and increases complexity. Eventually, no one knows which flags do what.

What to do instead: Set a removal date when creating each flag. Track flags like technical debt. Remove them aggressively once features are stable.

When to Pause or Pivot

Sometimes TBD migration stalls or causes more problems than it solves. Here’s how to tell if you need to pause and what to do about it.

Signs You’re Not Ready Yet

Red flag 1: Your test suite takes hours to run If developers can’t get feedback in minutes, they can’t integrate frequently. Forcing TBD now will just slow everyone down.

What to do: Pause the TBD migration. Invest 2-4 weeks in making tests faster. Parallelize test execution. Remove or optimize the slowest tests. Resume TBD when feedback takes less than 10 minutes.

Red flag 2: More than half your tests are flaky If tests fail randomly, developers will ignore failures. You’ll integrate broken code without realizing it.

What to do: Stop adding new features. Spend one sprint fixing or deleting flaky tests. Track flakiness metrics. Only resume TBD when you trust your test results.

Red flag 3: Production incidents increased significantly If TBD caused a spike in production issues, something is wrong with your safety net.

What to do: Revert to short-lived branches (48-72 hours) temporarily. Analyze what’s escaping to production. Add tests or checks to catch those issues. Resume direct-to-trunk when the safety net is stronger.

Red flag 4: The team is in constant conflict If people are fighting about the process, frustrated daily, or actively working around it, you’ve lost the team.

What to do: Hold a retrospective. Listen to concerns without defending TBD. Identify the top 3 pain points. Address those first. Resume TBD migration when the team agrees to try again.

Signs You’re Doing It Wrong (But Can Fix It)

Yellow flag 1: Daily commits, but monthly integration You’re committing to trunk, but your code doesn’t connect to the rest of the system until the end.

What to fix: Focus on interface-level integration. Ensure your tests exercise boundaries between components. Use contract tests.

Yellow flag 2: Trunk is broken often If trunk is red more than 5% of the time, something’s wrong with your testing or commit discipline.

What to fix: Make “fix trunk immediately” the top priority. Consider requiring local tests to pass before pushing. Add pre-commit hooks if needed.

Yellow flag 3: Feature flags piling up If you have more than 5 active flags, you’re not cleaning up after yourself.

What to fix: Set a team rule: “For every new flag created, remove an old one.” Dedicate time each sprint to flag cleanup.

How to Pause Gracefully

If you need to pause:

1. Communicate clearly: “We’re pausing TBD migration for two weeks to fix our test infrastructure. This isn’t abandoning the goal.”

2. Set a specific resumption date: Don’t let “pause” become “quit.” Schedule a date to revisit.

3. Fix the blockers: Use the pause to address the specific problems preventing success.

4. Retrospect and adjust: When you resume, what will you do differently?

Pausing isn’t failure. Pausing to fix the foundation is smart.

What “Good” Looks Like

You know TBD is working when:

• Branches live for hours, not days
• Developers collaborate early instead of merging late
• Product participates in defining behaviors, not just writing stories
• Tests run fast enough to integrate frequently
• Deployments are boring
• You can fix production issues with the same process you use for normal work

When your deployment process enables emergency fixes without special exceptions, you’ve reached the real payoff: lower cost of change, which makes everything else faster, safer, and more sustainable.

Concrete Examples and Scenarios

Theory is useful. Examples make it real. Here are practical scenarios showing how to apply TBD principles.

Scenario 1: Breaking Down a Large Feature

Problem: You need to build a user notification system with email, SMS, and in-app notifications. Estimated: 3 weeks of work.

Old approach (GitFlow): Create a feature/notifications branch. Work for three weeks. Submit a massive pull request. Spend days in code review and merge conflicts.

TBD approach:

The interface and its implementations are new code with no existing caller, so they ship as dark code first. Only the final send behavior, which has a real user-facing effect, needs a feature flag.

First commit: Define notification interface, commit to trunk

// notifications/NotificationService.js
// Contract: all implementations must provide send(userId, message)
// message shape: { title, body, priority } where priority is 'low', 'normal', or 'high'

class NotificationService {
  async send(userId, message) {
    throw new Error('Not implemented');
  }
}

This compiles but doesn’t do anything yet. That’s fine.

Next commit: Add in-memory implementation for testing

class InMemoryNotificationService extends NotificationService {
  constructor() {
    super();
    this.notifications = [];
  }

  async send(userId, message) {
    this.notifications.push(message);
  }
}

Now other teams can use the interface in their code and tests.

Then: Implement email notifications behind a feature flag

class EmailNotificationService extends NotificationService {
  async send(userId, message) {
    if (!features.emailNotifications) {
      return; // No-op when disabled
    }
    // Real email sending implementation
  }
}

Commit daily. Deploy. Flag is off in production.

Continue iterating:

• Add SMS notifications (same pattern: interface, implementation, feature flag)
• Enable email notifications for internal users only
• Add in-app notifications
• Roll out email and SMS to all users
• Remove flags for email once stable

Result: Integrated 12-15 times instead of once. Each integration was small and low-risk.

Scenario 2: Database Schema Change

Problem: You need to split the users.name column into first_name and last_name.

Old approach: Update schema, update all code, deploy everything at once. Hope nothing breaks.

TBD approach (expand and contract pattern):

Step 1: Expand Add new columns without removing the old one:

ALTER TABLE users ADD COLUMN first_name VARCHAR(255);
ALTER TABLE users ADD COLUMN last_name VARCHAR(255);

Commit and deploy. Application still uses name column. No breaking change.

Step 2: Dual writes Update write path to populate both old and new columns:

async function createUser(name) {
  const [firstName, lastName] = name.split(' ');
  await db.query(
    'INSERT INTO users (name, first_name, last_name) VALUES (?, ?, ?)',
    [name, firstName, lastName]
  );
}

Commit and deploy. Now new data populates both formats.

Step 3: Backfill Migrate existing data in the background:

async function backfillNames() {
  const users = await db.query('SELECT id, name FROM users WHERE first_name IS NULL');
  for (const user of users) {
    const [firstName, lastName] = user.name.split(' ');
    await db.query(
      'UPDATE users SET first_name = ?, last_name = ? WHERE id = ?',
      [firstName, lastName, user.id]
    );
  }
}

Run this as a background job. Commit and deploy.

Step 4: Read from new columns Update read path behind a feature flag:

async function getUser(id) {
  const user = await db.query('SELECT * FROM users WHERE id = ?', [id]);
  if (features.useNewNameColumns) {
    return {
      firstName: user.first_name,
      lastName: user.last_name,
    };
  }
  return { name: user.name };
}

Deploy and gradually enable the flag.

Step 5: Contract Once all reads use new columns and flag is removed:

ALTER TABLE users DROP COLUMN name;

Result: Five deployments instead of one big-bang change. Each step was reversible. Zero downtime.

Scenario 3: Refactoring Without Breaking the World

Problem: Your authentication code is a mess. You want to refactor it without breaking production.

TBD approach:

Characterization tests Write tests that capture current behavior (warts and all):

describe('Current auth behavior', () => {
  it('accepts password with special characters', () => {
    // Document what currently happens
  });

  it('handles malformed tokens by returning 401', () => {
    // Capture edge case behavior
  });
});

These tests document how the system actually works. Commit.

Branch by abstraction Create new implementation alongside old one:

class LegacyAuthService {
  // Existing messy code (don't touch it)
}

class ModernAuthService {
  // Clean implementation
}

class AuthServiceRouter {
  constructor(legacy, modern) {
    this.legacy = legacy;
    this.modern = modern;
  }

  async authenticate(credentials) {
    if (features.modernAuth) {
      return this.modern.authenticate(credentials);
    }
    return this.legacy.authenticate(credentials);
  }
}

Commit with flag off. Old behavior unchanged.

Migrate piece by piece Enable modern auth for one endpoint at a time:

if (features.modernAuth && endpoint === '/api/users') {
  return modernAuth.authenticate(credentials);
}

Commit daily. Monitor each endpoint.

Remove old code Once all endpoints use modern auth and it has been stable:

class AuthService {
  async authenticate(credentials) {
    // Just the modern implementation
  }
}

Delete the legacy code entirely.

Result: Continuous refactoring without a “big rewrite” branch. Production was never at risk.

Scenario 4: Working with External API Changes

Problem: A third-party API you depend on is changing their response format next month.

TBD approach:

Adapter pattern Create an adapter that normalizes both old and new formats:

class PaymentAPIAdapter {
  async getPaymentStatus(orderId) {
    const response = await fetch(`https://api.payments.com/orders/${orderId}`);
    const data = await response.json();

    // Handle both old and new format
    if (data.payment_status) {
      // Old format
      return {
        status: data.payment_status,
        amount: data.total_amount,
      };
    } else {
      // New format
      return {
        status: data.status.payment,
        amount: data.amounts.total,
      };
    }
  }
}

Commit. Your code now works with both formats.

After the API migration: Simplify adapter to only handle new format:

async getPaymentStatus(orderId) {
  const response = await fetch(`https://api.payments.com/orders/${orderId}`);
  const data = await response.json();
  return {
    status: data.status.payment,
    amount: data.amounts.total,
  };
}

Result: No coupling between your deployment schedule and the external API migration. Zero downtime.

References and Further Reading

• trunkbaseddevelopment.com - Comprehensive guide by Paul Hammant
• “Continuous Delivery” by Jez Humble and David Farley - Foundational text on CD practices
• Martin Fowler on Feature Toggles - Deep dive into feature flag patterns
• “Specification by Example” by Gojko Adzic - Collaborative test writing
• Pact Documentation - Consumer-driven contract testing
• “Working Effectively with Legacy Code” by Michael Feathers - Characterization tests and strangler patterns
• Evolutionary Database Design - Martin Fowler on expand-contract migrations
• “Accelerate” by Nicole Forsgren, Jez Humble, and Gene Kim - Data on what drives software delivery performance
• State of DevOps Reports - Annual research on delivery practices

Final Thought

Migrating from GitFlow to TBD isn’t a matter of changing your branching strategy. It’s a matter of changing your thinking.

Stop optimizing for isolation. Start optimizing for feedback.

Small, tested, integrated changes, delivered continuously, will always outperform big batches delivered occasionally.

That’s why teams migrate to TBD. Not because it’s trendy, but because it’s the only path to real continuous integration and continuous delivery.

2 - Evolutionary Coding Techniques

Deployment Is Not Release

Deployment is a technical action: pushing code to production. Release is a business decision: making a capability available to users. Evolutionary coding techniques are how you deploy continuously while controlling release independently.

Feature flags are the best-known way to make that separation, which is why teams reach for them first. But a runtime if (flag) branch is conditional complexity: every flag combination has to be tested, and the flag has to be deleted later or it becomes permanent debt. Most incomplete work does not need a flag at all. It needs to be structured so the unfinished parts are inert until they are ready.

Use the least intrusive technique that solves the problem. Reach for a flag only when nothing simpler applies.

In This Section

The Hierarchy of Techniques

Work down this list. Each technique carries more long-term maintenance cost than the one above it.

How to Choose

Ask these questions in order. Stop at the first “yes.”

1. Can the change be introduced additively, with nothing pointing to it yet? Use dark code.
2. Does an existing interface already isolate this behavior, or can you extract one first? Use branch by abstraction.
3. Do you need to prove the new logic produces the same answer as the old logic before you trust it? Use parallel run.
4. Are you changing a shared contract, like a database schema or an API, that other code or services depend on? Use expand and contract.
5. Are you replacing a whole subsystem or service, not a single implementation? Use the strangler fig pattern at the routing layer.
6. Is this strictly a business release timing decision, such as a coordinated launch, a kill switch, an entitlement, or an experiment, that none of the above can express? Use a feature flag, and only at the edge of the system.

Reaching question 6 is a legitimate reason to flag. Reaching for a flag at question 1 is not.

Rules If You Do Reach for a Flag

• Flag at the edge, not in domain logic. Put the check in a controller, router, or top-level entry point, never buried inside business logic or a data-access layer.
• Give every flag an expiration date at creation time. No date means the flag is permanent, and permanent release flags are debt.
• Create the removal ticket in the same pull request that adds the flag. Not later.
• Never nest flags. If capability B depends on capability A, extend A’s flag instead of stacking a second flag on top of it.

See Feature Flags for the full flag lifecycle, from creation through removal.

Key Pitfalls

1. “We used a flag because it was the tool we already knew”

Familiarity is not a reason to skip the hierarchy. A flag left in place after launch is technical debt that dark code or branch by abstraction would never have created in the first place.

2. “We picked branch by abstraction, but nothing was calling the old code through an interface yet”

Extract the interface first, as its own zero-behavior-change commit, before starting the swap. Introducing an abstraction and a new implementation in the same change makes the abstraction hard to review on its own merits.

3. “We treated a database migration like a code refactor”

A shared schema or API is a contract other people’s code depends on. Use expand and contract rather than branch by abstraction for anything consumed outside your own codebase.

Next Step

These techniques are what make trunk-based development safe for incomplete work. Once your team can integrate daily without flag sprawl, continue building the test architecture that backs it.

Related Content

• Trunk-Based Development - the practice these techniques make safe
• TBD Migration Guide - worked scenarios using these techniques
• Feature Flags - full lifecycle guidance for when a flag is the right tool
• Architecture Decoupling - the strangler fig pattern for replacing whole subsystems
• Long-Lived Feature Branches - the anti-pattern these techniques replace
• Small Batches - the batch-sizing discipline these techniques support

2.1 - Dark Code

What Is Dark Code?

Dark code is new logic that is fully built, tested, and deployed to production, but not yet reachable. No route, UI trigger, or message consumer points to it. It sits inert in the running binary until the final commit connects it.

This is sometimes called “connect tests last” or a “dark launch,” because the implementation is complete; only the wiring is missing.

What Dark Code Is Not

• It is not dead code left behind after a change. Dark code is temporary and has a defined moment it becomes live.
• It is not a feature flag. There is no runtime check and no conditional branch. The only cleanup is the wiring commit itself, which is not cleanup at all.
• It is not untested. The code has full unit and integration test coverage before it deploys; only production traffic hasn’t reached it yet.

What Dark Code Improves

Building Behind Dark Code

Step 1: Build the implementation

Write the domain logic, service, or component with its own unit and integration tests, exactly as you would if it were shipping today. Commit and deploy continuously as you go.

Step 2: Deploy without wiring

Each commit ships to production. The code compiles, is tested, and runs inside the deployed artifact, but nothing calls it. There is no behavior change for users, because there is no path to the new code.

// discountEngine.js - deployed, tested, unreferenced
class DiscountEngine {
  calculate(cart) {
    // Fully implemented and unit tested
  }
}

module.exports = DiscountEngine;

// Nothing in the request handlers imports DiscountEngine yet.
// It ships with every deploy and does nothing until it's wired in.

Step 3: Wire it in as the final change

Once the implementation is complete and reviewed, the last pull request adds the entry point: the route, the UI trigger, or the consumer binding.

const DiscountEngine = require('./discountEngine');
const engine = new DiscountEngine();

app.post('/cart/checkout', (req, res) => {
  const discount = engine.calculate(req.body.cart);
  res.json({ discount });
});

This commit is small and easy to review, because all the risk was already tested and deployed in the commits before it.

When Dark Code Is Not Enough

Dark code works when you control every caller and can wait to wire the last one in. It does not work when:

• You need to compare the new logic against production behavior before trusting it. Use parallel run instead.
• You are replacing an implementation that already has live callers. Use branch by abstraction instead.
• The business needs the release timed independently of when the code is ready, such as a coordinated launch or a gradual percentage rollout. Use a feature flag instead.

Key Pitfalls

1. “We wired it in early to test in production”

If you need production traffic to validate the new code before trusting it, that is a parallel run, not dark code. Wiring in an unfinished path to see what happens exposes users to unfinished work.

2. “The dark code sat unwired for three months”

Dark code should be wired in within days, not months. If the entry point keeps slipping, the feature isn’t actually close to done, and calling it dark code is hiding that from the team.

Measuring Success

Next Step

When you’re replacing an implementation that already has live callers instead of adding a new one, use Branch by Abstraction.

Related Content

• Evolutionary Coding Techniques - the full decision hierarchy
• Trunk-Based Development - the practice dark code makes safe
• Feature Flags - the alternative to reach for once dark code doesn’t apply
• Small Batches - deploying each piece of a feature continuously

2.2 - Branch by Abstraction

What Is Branch by Abstraction?

Branch by abstraction introduces an interface over an existing implementation, redirects callers to that interface, then builds and switches in a new implementation behind it, all as small commits on trunk. The “branching” happens in the abstraction layer, not in version control.

The technique gets its name because it replaces a source-control branch with a branch in the code itself: an interface with two implementations, one of which is live.

What Branch by Abstraction Is Not

• It is not a long-lived feature branch with an interface added to justify it. If the work still takes weeks on a branch, the abstraction hasn’t replaced anything.
• It is not the strangler fig pattern. Branch by abstraction swaps an implementation behind an in-process interface. Strangler fig replaces a whole subsystem or service by routing traffic to it at a system boundary. Use branch by abstraction inside a codebase you own; use strangler fig when the thing being replaced is bigger than one component.
• It is not a permanent abstraction layer. Once the swap is complete, remove the old implementation, and remove the interface too if nothing else needs it.

What Branch by Abstraction Improves

Making the Swap

Step 1: Abstract

Introduce an interface over the existing code and redirect every caller to it. This is a zero-behavior-change commit: the interface wraps the current implementation and nothing else changes.

class AuthService {
  authenticate(credentials) {
    // existing implementation, moved behind the interface unchanged
  }
}

// callers now depend on AuthService, not the concrete legacy class
const auth = new AuthService();

Step 2: Implement

Build the new implementation alongside the old one, as its own class. Commit and deploy the new class in small pieces; it isn’t wired to any caller yet, so it carries the same zero risk as dark code.

class LegacyAuthService {
  authenticate(credentials) {
    // existing messy implementation, untouched
  }
}

class ModernAuthService {
  authenticate(credentials) {
    // new implementation, built and tested incrementally
  }
}

Step 3: Switch

Change the dependency injection or factory binding to instantiate the new implementation instead of the old one. This is the entire cutover: one line, one commit, easy to revert.

// container.js
container.register('AuthService', ModernAuthService); // was LegacyAuthService

If you need to de-risk the switch further, or need confidence the two implementations produce identical results first, run them side by side with a parallel run before flipping the binding.

Step 4: Prune

Delete the legacy implementation. Delete the interface too if only one implementation remains and nothing else depends on the abstraction.

class AuthService {
  authenticate(credentials) {
    // just the modern implementation now
  }
}

Cleanup here is a straightforward deletion of a class. There is no scattered if/else logic to search for, because the old and new implementations were never in the same function.

Key Pitfalls

1. “We built the new implementation and the interface in the same commit”

This makes the abstraction hard to review on its own merits, and it removes the option to ship the interface as a safe, standalone step. Extract the interface first, verify it changes nothing, then start on the new implementation.

2. “We left the old implementation in place after the switch”

The switch commit is not the finish line. If the old class is still in the codebase a month later, delete it. An unused implementation behind a working interface is exactly the kind of dead weight branch by abstraction is supposed to avoid.

3. “We used branch by abstraction to replace a whole service”

If the replacement spans multiple components, teams, or a system boundary, that’s a strangler fig problem, not an in-process interface swap.

Measuring Success

Next Step

If you need to prove the new implementation matches production behavior before switching the binding, use Parallel Run.

Related Content

• Evolutionary Coding Techniques - the full decision hierarchy
• Dark Code - the simpler technique for new, unreferenced logic
• Architecture Decoupling - the strangler fig pattern for replacing whole subsystems
• Long-Lived Feature Branches - the anti-pattern branch by abstraction replaces
• Branch By Abstraction - external reference site for the pattern

2.3 - Parallel Run

What Is a Parallel Run?

A parallel run, sometimes called shadowing or a dark launch of logic, wraps a call so both the current implementation and a candidate replacement execute against identical production input. The caller always receives the current implementation’s result. The candidate’s result is captured and compared, never returned.

This technique is best known from GitHub’s open-source Scientist library, which formalized the pattern for verifying refactors of high-risk code paths.

What a Parallel Run Is Not

• It is not a percentage rollout. Every request runs through both implementations; nothing is split between them. Percentage rollout is a feature flag concern that comes after a parallel run has already established parity.
• It is not A/B testing or hypothesis-driven development. Users never see the candidate’s output during a parallel run, so it measures technical correctness, not user response.
• It is not a permanent architecture. The comparison harness is temporary scaffolding, removed once the candidate becomes the primary path.

What a Parallel Run Improves

Running the Comparison

Step 1: Wrap the call

Introduce a thin proxy around the existing call. The caller’s contract does not change.

async function calculatePrice(order) {
  const legacyResult = await legacyPricingEngine.calculate(order);

  // Fire the candidate asynchronously; never let it affect the response
  shadowRun(() => modernPricingEngine.calculate(order), legacyResult, order);

  return legacyResult;
}

Step 2: Run the candidate and compare

Run the new implementation against the same input, in the background, and log any mismatch along with enough context to debug it.

async function shadowRun(candidateFn, legacyResult, order) {
  try {
    const candidateResult = await candidateFn();
    if (!isEquivalent(candidateResult, legacyResult)) {
      telemetry.recordMismatch('pricing-engine', {
        orderId: order.id,
        legacyResult,
        candidateResult,
      });
    }
  } catch (err) {
    telemetry.recordCandidateError('pricing-engine', { orderId: order.id, err });
  }
}

Step 3: Watch the telemetry

Track mismatch rate, candidate error rate, and performance delta over a statistically meaningful window. Investigate every mismatch; each one is either a genuine bug in the candidate or a case where the legacy behavior was wrong and needs a deliberate decision.

Step 4: Cut over and remove the harness

Once the mismatch rate holds at zero for the agreed period, switch the candidate to the primary path, typically with branch by abstraction, and delete the comparison harness. Leaving it in place after cutover is unnecessary runtime cost with no further benefit.

When a Parallel Run Is Not Enough

• The old and new implementations must not both execute, for example when the operation has side effects like sending an email or charging a card. Idempotent, side-effect-free logic (pricing, scoring, routing decisions) is what parallel run is for. For operations with side effects, use branch by abstraction with a smaller, monitored rollout instead.
• You are changing a shared schema or contract, not just an implementation. Use expand and contract instead.

Key Pitfalls

1. “We let the candidate’s exceptions bubble up to the caller”

The candidate’s failures must never affect the response. Catch and log every exception from the candidate path independently of the legacy path.

2. “We ran the comparison for a day and called it proven”

A parallel run needs enough volume and enough time to cover the input space that matters, including rare edge cases and periodic patterns like end-of-month billing. Set the comparison window based on when those cases actually occur, not a fixed number of days.

3. “We kept the shadow harness running after cutover”

The harness is temporary. Once the candidate is primary and stable, remove the shadow call entirely. Running both implementations forever doubles compute cost for no ongoing benefit.

Measuring Success

Next Step

If the change touches a shared database schema or an API contract rather than a single implementation, use Expand and Contract.

Related Content

• Evolutionary Coding Techniques - the full decision hierarchy
• Branch by Abstraction - the technique that typically performs the eventual cutover
• Change Failure Rate - the metric a parallel run protects for high-risk changes

2.4 - Expand and Contract

What Is Expand and Contract?

Expand and contract evolves a shared contract, a database schema, an event schema, or an API, without ever mutating it in place. Instead of replacing the old shape with the new one in a single change, you add the new shape alongside the old one, migrate consumers over incrementally, and only remove the old shape once nothing depends on it.

The name comes from the two ends of the sequence: you expand the contract to support both shapes at once, then contract it back down to just the new shape.

What Expand and Contract Is Not

• It is not a single migration script run during a deployment window. Each phase is its own independent, reversible deployment.
• It is not limited to databases. The same four phases apply to API fields, event schemas, and message formats: anything with more than one reader or writer.
• It is not optional for shared contracts. Branch by abstraction is enough when only your own code depends on the thing you’re changing. Once another service, another team, or stored data depends on it, use expand and contract instead.

What Expand and Contract Improves

The Four Phases

Phase 1: Expand

Add the new shape alongside the old one. Nothing reads from it yet, and nothing that currently works stops working.

ALTER TABLE users ADD COLUMN first_name VARCHAR(255);
ALTER TABLE users ADD COLUMN last_name VARCHAR(255);

Deploy this on its own. The application still reads and writes the name column exclusively. There is no behavior change.

For an API, the equivalent is adding a new field or a new endpoint version without removing the old one.

Phase 2: Dual-write and backfill

Update the write path to populate both the old and new shapes at once, then backfill existing data in the background.

async function createUser(name) {
  const [firstName, lastName] = name.split(' ');
  await db.query(
    'INSERT INTO users (name, first_name, last_name) VALUES (?, ?, ?)',
    [name, firstName, lastName]
  );
}

async function backfillNames() {
  const users = await db.query('SELECT id, name FROM users WHERE first_name IS NULL');
  for (const user of users) {
    const [firstName, lastName] = user.name.split(' ');
    await db.query(
      'UPDATE users SET first_name = ?, last_name = ? WHERE id = ?',
      [firstName, lastName, user.id]
    );
  }
}

Deploy the dual-write change, then run the backfill as its own job. Both are independently reversible: if the backfill has a problem, the application still works off the old column.

For an API, this phase is a tolerant reader: consumers ignore fields they don’t recognize, and producers populate both the old and new field until every consumer has moved.

Phase 3: Dual-read and cutover

Switch reads to consume the new shape, one caller at a time.

async function getUser(id) {
  const user = await db.query('SELECT * FROM users WHERE id = ?', [id]);
  return {
    firstName: user.first_name,
    lastName: user.last_name,
  };
}

Because Phase 2 guarantees the new columns are always populated, this switch has nothing to migrate; it’s a straightforward read-path change deployed independently of the write-path change that came before it.

Phase 4: Contract

Once every reader and writer uses the new shape exclusively, remove the old one.

ALTER TABLE users DROP COLUMN name;

For an API, this phase is deprecating and eventually removing the old field or version, once telemetry confirms no consumer still calls it.

Result: four independent deployments instead of one big-bang change. Each one is reversible on its own, and none of them requires another team or service to redeploy in the same window.

When Expand and Contract Is Not Enough

If the two shapes cannot coexist even briefly, for example a uniqueness constraint that the old and new schema can’t both satisfy, you need a more deliberate migration plan with an explicit maintenance window. That is the exception, not the default: most schema and contract changes can be expressed as expand and contract if you’re willing to take more, smaller steps.

Key Pitfalls

1. “We skipped the backfill and just changed the read path”

Reads that assume the new column is populated will fail or return nulls for any row that predates the change. Backfill before cutting over reads, not after.

2. “We dropped the old column right after the dual-write phase”

The contract phase must wait until every writer and every reader has confirmed to use the new shape only. Removing the old column while any code, including code outside your immediate deploy, still references it turns a safe migration into an outage.

3. “We coordinated the four phases into one deployment”

Collapsing the phases back into a single deployment defeats the purpose. Each phase exists so it can be deployed, verified, and rolled back independently.

Measuring Success

Next Step

Return to Evolutionary Coding Techniques to see when a broader strangler fig migration, or a feature flag for business release timing, is the more appropriate tool.

Related Content

• Evolutionary Coding Techniques - the full decision hierarchy
• Branch by Abstraction - the equivalent technique for implementations with no shared contract
• TBD Migration Guide - a full worked scenario of expand and contract on a database schema
• Application Configuration - keeping environment-specific values out of the artifacts this pattern deploys
• Rollback - why each phase of expand and contract must be independently reversible

3 - Testing Fundamentals

Testing Goals for CD

Your test suite must meet these goals before it can support continuous delivery.

In This Section

The Ice Cream Cone: What to Avoid

An inverted test distribution, with too many slow end-to-end tests and too few fast unit tests, is the most common testing barrier to CD.

The ice cream cone makes CD impossible. Manual testing gates block every release. End-to-end tests take hours, fail randomly, and depend on external systems being healthy. For the test architecture that replaces this, see Pipeline Test Strategy and the Testing reference.

Next Step

Automate your build process so that building, testing, and packaging happen with a single command. Continue to Build Automation.

Content contributed by Dojo Consortium, licensed under CC BY 4.0. Additional concepts drawn from Ham Vocke, The Practical Test Pyramid, and Toby Clemson, Testing Strategies in a Microservice Architecture.

Related Content

• Flaky Tests - Symptom of non-deterministic tests that destroy pipeline trust
• High Coverage, Ineffective Tests - Symptom where coverage metrics mask poor test quality
• Refactoring Breaks Tests - Symptom of white-box tests that assert on implementation details
• Slow Test Suites - Symptom caused by an inverted test pyramid or missing test doubles
• Environment-Dependent Failures - Symptom of tests coupled to external systems
• Inverted Test Pyramid - Anti-pattern where too many slow E2E tests replace fast unit tests
• Pressure to Skip Testing - Anti-pattern where testing is treated as optional under deadline pressure

3.1 - Test Architecture

A test architecture that lets your pipeline deploy confidently, regardless of external system availability, is a core CD capability. The Test Types pages cover each type in depth.

A CD pipeline’s job is to force every artifact to prove it is worthy of delivery. That proof only works when test changes ship with the code they validate. If a developer adds a feature but the corresponding tests arrive in a later commit, the pipeline approved an artifact it never actually verified. That is not a CD pipeline. It is a CI pipeline with a deploy step. Tests and production code must always travel together through the pipeline as a single unit of change.

Beyond the Test Pyramid

The test pyramid says: write many fast unit tests at the base, fewer integration tests in the middle, and only a handful of end-to-end tests at the top. The underlying principle is sound - lower-level tests are faster, more deterministic, and cheaper to maintain.

The principle behind the shape

The pyramid’s shape communicates a principle: prefer fast, deterministic tests that you fully control. Tests at the base are cheap to write, fast to run, and reliable. Tests at the top are slow, expensive, and depend on systems outside your control. The more weight you put at the base, the faster and more reliable your pipeline becomes - to a point. We also have the engineering goal of achieving the most functional coverage with the fewest number of tests. Every test costs money to maintain and adds time to the pipeline.

The testing trophy

The testing trophy, popularized by Kent C. Dodds, rebalances the pyramid by putting component tests at the center. Where the pyramid emphasizes unit tests at the base, the trophy argues that component tests give you the most confidence per test because they exercise realistic user behavior through a component’s public interface while still using test doubles for external dependencies.

The trophy also makes static analysis explicit as the foundation. Linting, type checking, and formatting catch entire categories of defects for free - no test code to write or maintain.

Both models agree on the principle: keep end-to-end tests few and focused, and maximize fast, deterministic coverage. The trophy simply shifts where that coverage concentrates. For teams building component-heavy applications, the trophy distribution often produces better results than a strict pyramid.

Teams often miss this underlying principle and treat either shape as a metric. They count tests by type and debate ratios - “do we have enough unit tests?” or “are our integration tests too many?” - when the real question is:

Can our pipeline determine that a change is safe to deploy without depending on any system we do not control?

A pipeline that answers yes can deploy at any time - even when a downstream service is down, a third-party API is slow, or a partner team hasn’t shipped yet. That independence is what CD requires, and it is the reason the pyramid favors the base.

What this looks like in practice

A test architecture that achieves this has three responsibilities:

1. Fast, deterministic tests - unit, component, and contract tests - run on every commit using test doubles for external dependencies. They give a reliable go/no-go signal in minutes.
2. Acceptance tests validate that a deployed artifact is deliverable. Acceptance testing is not a single test type. It is a pipeline stage that can include component tests, load tests, chaos tests, resilience tests, and compliance tests. Any test that runs after CI to gate promotion to production is an acceptance test.
3. Integration tests validate that contract test doubles still match the real external systems. They run in a dedicated test environment with versioned test data, on demand or on a schedule, providing monitoring rather than gating.

The anti-pattern: the ice cream cone

Most teams that struggle with CD have inverted the pyramid - too many slow, flaky end-to-end tests and too few fast, focused ones. Manual gates block every release. The pipeline cannot give a fast, reliable answer, so deployments become high-ceremony events.

Test Architecture

A test architecture is the deliberate structure of how different test types work together across your pipeline to give you deployment confidence. Use the table below to decide what type of test to write and where it runs. This is not a comprehensive list. It shows how common tests impact pipeline design and how teams should structure their suites. See the Pipeline Reference Architecture for a complete quality gate sequence.

The critical insight: everything that blocks merge is deterministic and under your control. Acceptance tests gate production promotion after verifying the deployed artifact. Everything that involves real external systems runs post-deployment. This is what gives you the independence to deploy any time, regardless of the state of the world around you.

Acceptance tests can include non-deterministic activities (load, chaos, resilience), but the gate decision is still deterministic: it fires on a documented pass/fail threshold - a performance budget, an error-rate ceiling, a required compliance check - not on the raw variability of the measurement. That is different from gating on a flaky test whose pass/fail flips for reasons unrelated to the change, which the Do Not list below warns against.

Pre-merge vs post-merge

The table maps to two distinct phases of your pipeline, each with different goals and constraints.

Pre-merge (before code lands on trunk): Run unit, component, and contract tests. These must all be deterministic and fast. Target: under 10 minutes total. This is the quality gate that every change must pass. If pre-merge tests are slow, developers batch up changes or skip local runs, both of which undermine continuous integration.

Post-merge (after code lands on trunk, before or after deployment): Re-run the full deterministic suite against the integrated trunk. Then run acceptance tests, E2E smoke tests, and synthetic monitoring post-deploy. Integration tests run separately in a test environment, on demand or on a schedule. Target: under 60 minutes for the full post-merge cycle.

Why re-run pre-merge tests post-merge? Two changes can each pass pre-merge independently but conflict when combined on trunk. The post-merge run catches these integration effects.

If a post-merge failure occurs, the team fixes it immediately. Trunk must always be releasable.

This post-merge re-run is what teams traditionally call regression testing: running all previous tests against the current artifact to confirm that existing behavior still works after a change. In CD, regression testing is not a separate test type or a special suite. Every test in the pipeline is a regression test. The deterministic suite runs on every commit, and the full suite runs post-merge. A green run means the artifact has been regression-tested against every behavior the suite encodes - no more and no less, which is why the suite’s coverage of prior behavior is what makes the signal trustworthy.

good practices

Do

• Run tests on every commit. If tests do not run automatically, they will be skipped.
• Keep the deterministic suite under 10 minutes. If it is slower, developers will stop running it locally.
• Fix broken tests immediately. A broken test is equivalent to a broken build.
• Delete tests that do not provide value. A test that never fails and tests trivial behavior is maintenance cost with no benefit.
• Test behavior, not implementation. Use a black box approach - verify what the code does, not how it does it. As Ham Vocke advises: “if I enter values x and y, will the result be z?” - not the sequence of internal calls that produce z. Avoid white box testing that asserts on internals.
• Use test doubles for external dependencies. Your deterministic tests should run without network access to external systems.
• Validate test doubles with contract tests. Test doubles that drift from reality give false confidence.
• Treat test code as production code. Give it the same care, review, and refactoring attention.
• Run automated accessibility checks on every commit. WCAG compliance scans are fast, deterministic, and catch violations that are invisible to sighted developers. Treat them like security scans: automate the detectable rules and reserve manual review for subjective judgment. See Accessibility testing for the full three-tier strategy and pipeline placement.

Do Not

• Do not tolerate flaky tests. Quarantine or delete them immediately.
• Do not gate your pipeline on flaky, non-deterministic test signals. E2E and integration test failures - pass/fail that flips for reasons unrelated to the change - should trigger review or alerts, not block deployment. (An acceptance gate that fires on a deterministic threshold, like a performance budget, is not this: the gate decision is stable even when the underlying measurement varies.)
• Do not couple your deployment to external system availability. If a third-party API being down prevents you from deploying, your test architecture has a critical gap.
• Do not write tests after the fact as a checkbox exercise. Tests written without understanding the behavior they verify add noise, not value.
• Do not test private methods directly. Test the public interface; private methods are tested indirectly.
• Do not share mutable state between tests. Each test should set up and tear down its own state.
• Do not use sleep/wait for timing-dependent tests. Use explicit waits, polling, or event-driven assertions.
• Do not let unit or component tests depend on a shared or external database or service. A real engine the team controls and isolates per test - a per-test testcontainer, or a transaction that rolls back at teardown - is fine in-band and stays deterministic. A shared, mutable database, or any service the team does not control, is not: that reintroduces non-determinism, so categorize the test as integration or end-to-end and run it post-deployment, not as a pre-merge gate.
• Do not make exploratory or usability testing a release gate. These activities are continuous and inform product direction; they are not a pass/fail checkpoint before deployment.

Related Content

• ACD - How acceptance criteria make testing the constraint that governs agent-generated code
• Testing Fundamentals - Establishing testing practices as part of CD migration
• High Coverage but Ineffective Tests - When tests pass but do not catch real defects

Additional concepts drawn from Ham Vocke, The Practical Test Pyramid, and Toby Clemson, Testing Strategies in a Microservice Architecture.

3.2 - What to Test - and What Not To

Three principles determine what belongs in your test suite and what does not.

If you cannot fix it, do not test for it

You should never test the behavior of services you consume. Testing their behavior is the responsibility of the team that builds them. If their service returns incorrect data, you cannot fix that, so testing for it is waste.

What you should test is how your system responds when a consumed service is unstable or unavailable. Can you degrade gracefully? Do you return a meaningful error? Do you retry appropriately? These are behaviors you own and can fix, so they belong in your test suite.

This principle directly enables the pipeline test strategy. When you stop testing things you cannot fix, you stop depending on external systems in your pipeline. Your tests become faster, more deterministic, and more focused on the code your team actually ships.

Test interfaces first

Most integration failures originate at interfaces, the boundaries where your system talks to other systems. These boundaries are the highest-risk areas in your codebase, and they deserve the most testing attention. But testing interfaces does not require integrating with the real system on the other side.

When you test an interface you consume, the question is: “Can I understand the response and act accordingly?” If you send a request for a user’s information, you do not test that you get that specific user back. You test that you receive and understand the properties you need - that your code can parse the response structure and make correct decisions based on it. This distinction matters because it keeps your tests deterministic and focused on what you control.

Use contract mocks, virtual services, or any test double that faithfully represents the interface contract. The test validates your side of the conversation, not theirs.

Frontend and backend follow the same pattern

Both frontend and backend applications provide interfaces to consumers and consume interfaces from providers. The only difference is the consumer: a frontend provides an interface for humans, while a backend provides one for machines. The testing strategy is the same.

Test frontend code the same way you test backend code: validate the interface you provide, test logic in isolation, and verify that user actions trigger the correct behavior. The only difference is the consumer (a human instead of a machine).

For a frontend:

• Validate the interface you provide. The UI contains the components it should and they appear correctly. This is the equivalent of verifying your API returns the right response structure.
• Test behavior isolated from presentation. Use your unit test framework to test the logic that UI controls trigger, separated from the rendering layer. This gives you the same speed and control you get from testing backend logic in isolation.
• Verify that controls trigger the right logic. Confirm that user actions invoke the correct behavior, without needing a running backend or browser-based E2E test.

This approach gives you targeted testing with far more control. Testing exception flows - what happens when a service returns an error, when a network request times out, when data is malformed, becomes straightforward instead of requiring elaborate E2E setups that are hard to make fail on demand.

Test Quality Over Coverage Percentage

Code coverage tells you which lines executed during tests. It does not tell you whether the tests verified anything meaningful. A test suite with 90% coverage and no assertions has high coverage and zero value.

Better questions than “what is our coverage percentage?”:

• When a test fails, does it point directly to the defect?
• When we refactor, do tests break because behavior changed or because implementation details shifted?
• Do our tests catch the bugs that actually reach production?
• Can a developer trust a green build enough to deploy immediately?

Why coverage mandates are harmful

When teams are required to hit a coverage target, they write tests to satisfy the metric rather than to verify behavior. This produces:

• Tests that exercise code paths without asserting outcomes
• Tests that mirror implementation rather than specify behavior
• Tests that inflate the number without improving confidence

The metric goes up while the defect escape rate stays the same. Worse, meaningless tests add maintenance cost and slow down the suite.

Instead of mandating a coverage number, set a coverage floor (see Getting Started) and focus team attention on test quality: mutation testing scores, defect escape rates, and whether developers actually trust the suite enough to deploy on green.

Related Content

• High Coverage, Ineffective Tests - When coverage metrics mask poor test quality
• Refactoring Breaks Tests - Tests that assert on implementation details instead of behavior
• Code Coverage Mandates - The anti-pattern of mandating coverage targets
• Test Doubles - Patterns for isolating dependencies in tests
• Contract Tests - Verifying that test doubles match reality

3.3 - Pipeline Test Strategy

Everything that blocks deployment must be deterministic and under your control. Everything that involves external systems runs asynchronously or post-deployment. This gives you the independence to deploy any time, regardless of the state of the world around you.

Tests Inside the Pipeline

These tests run on every commit and block deployment if they fail. They must be fast, deterministic, and free of external dependencies.

Every test in this pipeline uses test doubles for anything that crosses the component boundary into a system the team does not control: third-party APIs, downstream services owned by other teams, message brokers. No in-band test calls a shared or external service. A real engine the team owns and isolates per test - a database in a per-test testcontainer, for example - is permitted in-band because it stays deterministic. This means:

• A downstream outage cannot block your deployment. Your pipeline runs the same whether external systems are healthy or down.
• Tests are deterministic. The same code always produces the same result.
• The suite is fast. No network latency, no waiting for external systems to respond.

Why re-run tests post-merge?

Two changes can each pass pre-merge independently but conflict when combined on trunk. The post-merge run catches these integration effects. If a post-merge failure occurs, the team fixes it immediately. Trunk must always be releasable.

Tests Outside the Pipeline

These tests involve real external systems and are therefore non-deterministic. They never block deployment. Instead, they validate assumptions and monitor production health.

How Contract Tests Validate Test Doubles

The pipeline’s deterministic tests depend on test doubles to represent external systems. But test doubles can drift from reality. An API adds a required field, changes a response format, or deprecates an endpoint. Contract tests close this gap.

1. Pipeline tests use test doubles that encode your assumptions about external APIs - response schemas, status codes, error formats.
2. Contract tests run on a schedule and send real requests to the actual external APIs.
3. Contract tests compare the real response against what your test doubles return. They check structure and types, not specific data values.
4. When a contract test passes, your test doubles are confirmed accurate. The pipeline’s deterministic tests are trustworthy.
5. When a contract test fails, the team is alerted. They update the test doubles to match the new reality, then re-run component tests to verify nothing breaks.

This design means your pipeline never touches external systems, but you still catch when external systems change. You get both speed and accuracy.

Consumer-driven contracts

When the external API is owned by another team in your organization, you can go further with consumer-driven contracts. Instead of your team polling their API on a schedule, both teams share a contract specification (using a tool like Pact):

• You (the consumer) define the requests you send and the responses you expect.
• They (the provider) run your contract as part of their build. If a change would break your expectations, their build fails before they deploy.
• Your test doubles are generated from the contract, guaranteeing they match what the provider actually delivers.

This shifts contract validation from “detect and react” to “prevent.” See Contract Tests for implementation details.

Summary: All Stages at a Glance

The Testing reference provides detailed documentation for each test type, including code examples and anti-patterns.

Related Content

• Testing Reference - Full reference for each test type
• Test Doubles - Patterns for stubs, mocks, fakes, spies, and dummies
• Contract Tests - Consumer-driven and provider contracts in detail
• Test Feedback Speed - The cognitive science behind the 10-minute target
• Pipeline Architecture - The pipeline structure these tests feed into
• Pipeline Reference Architecture - Reference diagrams for single-team and multi-team pipelines

3.4 - Getting Started

Starting Without Full Coverage

Teams often delay adopting CI because their existing code lacks tests. This is backwards. You do not need tests for existing code to begin. You need one rule applied without exception:

Every new change gets a test. We will not go lower than the current level of code coverage.

Record your current coverage percentage as a baseline. Configure CI to fail if coverage drops below that number. This does not mean the baseline is good enough. It means the trend only moves in one direction. Every bug fix, every new feature, and every refactoring adds tests. Over time, coverage grows organically in the areas that matter most: the code that is actively changing.

Do not attempt to retrofit tests across the entire codebase before starting CI. That approach takes months and delivers no incremental value. It also produces low-quality tests written by developers who are testing code they did not write and do not fully understand.

Quick-Start Action Plan

If your test suite is not yet ready to support CD, use this focused action plan to make immediate progress.

1. Audit your current test suite

Assess where you stand before making changes.

Actions:

• Run your full test suite 3 times. Note total duration and any tests that pass intermittently (flaky tests).
• Count tests by type: unit, integration, functional, end-to-end.
• Identify tests that require external dependencies (databases, APIs, file systems) to run.
• Record your baseline: total test count, pass rate, duration, flaky test count.
• Map each test type to a pipeline stage. Which tests gate deployment? Which run asynchronously? Which tests couple your deployment to external systems?

Output: A clear picture of your test distribution and the specific problems to address.

2. Fix or remove flaky tests

Flaky tests are worse than no tests. They train developers to ignore failures, which means real failures also get ignored.

Actions:

• Quarantine all flaky tests immediately. Move them to a separate suite that does not block the build.
• For each quarantined test, decide: fix it (if the behavior it tests matters) or delete it (if it does not).
• Common causes of flakiness: timing dependencies, shared mutable state, reliance on external services, test order dependencies.
• Target: zero flaky tests in your main test suite.

3. Decouple your pipeline from external dependencies

This is the highest-leverage change for CD. Identify every test that calls a real external service and replace that dependency with a test double.

Actions:

• List every external service your tests depend on: databases, APIs, message queues, file storage, third-party services.
• For each dependency, decide the right test double approach:
  • In-memory fakes for databases (e.g., an in-memory repository, or SQLite/H2 standing in for the production engine). Fastest, but they do not exercise real SQL semantics.
  • A team-controlled real engine in a per-test testcontainer when the production query plan, constraints, or migrations matter. This is a real database, not a fake, but it stays deterministic because the team pins the version and isolates state per test, so it runs in-band.
  • HTTP stubs for external APIs the team does not control (e.g., WireMock, nock, MSW).
  • Fakes for message queues, email services, and other infrastructure.
• Replace the dependencies in your unit and component tests.
• Move the original tests that hit real services into a separate suite. These become your starting contract tests or E2E smoke tests.

Output: A test suite where everything that blocks the build is deterministic and runs without network access to external systems.

4. Add component tests for critical paths

If you do not have component tests that exercise your whole service in isolation, start with the most critical paths.

Actions:

• Identify the 3-5 most critical user journeys or API endpoints in your application.
• Write a component test for each: boot the application, stub external dependencies, send a real request or simulate a real user action, verify the response.
• Each component test should prove that the feature works correctly assuming external dependencies behave as expected (which your test doubles encode).
• Run these in CI on every commit.

Output: Component tests covering your critical paths, running in CI on every commit.

5. Set up contract tests for your most important dependency

Pick the external dependency that changes most frequently or has caused the most production issues. Set up a contract test for it.

Actions:

• Write a contract test that validates the response structure (types, required fields, status codes) of the dependency’s API.
• Run it on a schedule (e.g., every hour or daily), not on every commit.
• When it fails, update your test doubles to match the new reality and re-verify your component tests.
• If the dependency is owned by another team in your organization, explore consumer-driven contracts with a tool like Pact.

Output: One contract test running on a schedule, with a process to update test doubles when it fails.

6. Adopt TDD for new code

Once your pipeline tests are reliable, adopt TDD for all new work. TDD is the practice of writing the test before the code. It ensures every piece of behavior has a corresponding test.

The TDD cycle

1. Red: Write a failing test that describes the behavior you want.
2. Green: Write the minimum code to make the test pass.
3. Refactor: Improve the code without changing the behavior. The test ensures you do not break anything.

Why TDD matters for CD

• Every change is automatically covered by a test
• The test suite grows proportionally with the codebase
• Tests describe behavior, not implementation, making them more resilient to refactoring
• Developers get immediate feedback on whether their change works

TDD is not mandatory for CD, but teams that practice TDD consistently have significantly faster and more reliable test suites.

How to start: Pick one new feature or bug fix this week. Write the test first, watch it fail, write the code to make it pass, then refactor. Do not try to retroactively TDD your entire codebase. Apply TDD to new code and to any code you modify.

Output: Team members practicing TDD on new work, with at least one completed red-green-refactor cycle.

Related Content

• Pipeline Test Strategy - Where these tests fit in your pipeline
• Flaky Tests - Symptom of non-deterministic tests
• Test Doubles - Patterns for isolating dependencies
• Contract Tests - Verifying test doubles match reality
• No Contract Testing Between Services - The anti-pattern this action plan addresses
• A Large Codebase Has No Automated Tests - Starting testing on a brownfield codebase

3.5 - Defect Feedback Loop

Treat every test failure as diagnostic data about where your process breaks down, not just as something to fix. When you identify the systemic source of defects, you can prevent entire categories from recurring.

Two questions sharpen this thinking:

1. What is the earliest point we can detect this defect? The later a defect is found, the more expensive it is to fix. A requirements defect caught during example mapping costs minutes. The same defect caught in production costs days of incident response, rollback, and rework.
2. Can AI help us detect it earlier? AI-assisted tools can now surface defects at stages where only human review was previously possible, shifting detection left without adding manual effort.

Trace Every Defect to Its Origin

When a test catches a defect (or worse, when a defect escapes to production) ask: where was this defect introduced, and what would have prevented it from being created?

Defects do not originate randomly. They cluster around specific causes. The CD Defect Detection and Remediation Catalog documents over 30 defect types across eight categories, with detection methods, AI opportunities, and systemic fixes for each.

For the complete catalog covering all defect categories (including product and discovery, dependency and infrastructure, testing and observability gaps, and more) see the CD Defect Detection and Remediation Catalog.

Build a Defect Feedback Loop

You need a process that systematically connects test failures to root causes and root causes to systemic fixes.

1. Classify every defect. When a test fails or a bug is reported, tag it with its origin category from the tables above. This takes seconds and builds a dataset over time.
2. Look for patterns. Monthly (or during retrospectives), review the defect classifications. Which categories appear most often? That is where your process is weakest.
3. Apply the systemic fix, not just the local fix. When you fix a bug, also ask: what systemic change would prevent this entire category of bug? If most defects come from integration boundaries, the fix is not “write more integration tests.” It is “make contract tests mandatory for every new boundary.” If most defects come from untested edge cases, the fix is not “increase code coverage.” It is “adopt property-based testing as a standard practice.”
4. Measure whether the fix works. Track defect counts by category over time. If you applied a systemic fix for integration boundary defects and the count does not drop, the fix is not working and you need a different approach.

The Test-for-Every-Bug-Fix Rule

Every bug fix must include a test that reproduces the bug before the fix and passes after. This is non-negotiable for CD because:

• It proves the fix actually addresses the defect (not just the symptom).
• It prevents the same defect from recurring.
• It builds test coverage exactly where the codebase is weakest: the places where bugs actually occur.
• Over time, it shifts your test suite from “tests we thought to write” to “tests that cover real failure modes.”

Advanced Detection Techniques

As your test architecture matures, add techniques that catch defects before manual review:

For more examples of mapping defect origins to detection methods and systemic corrections, see the CD Defect Detection and Remediation Catalog.

Related Content

• Systemic Defect Fixes - Detailed reference for each defect category
• High Coverage, Ineffective Tests - When tests pass but do not catch real defects
• Refactoring Breaks Tests - Tests that break on implementation changes
• Retrospectives - Where defect pattern review fits in the improvement cycle
• Metrics-Driven Improvement - Using defect escape rate as a key metric

3.6 - Test Types

Definitions for the test types used throughout this site. Each page covers what the type is, when it runs in the pipeline, what it asserts on, and what it does not.

The list isn’t exhaustive and the boundaries between types aren’t crisp in every codebase. Use these definitions as shared vocabulary for the rest of the testing section, especially Applied Testing Strategies and Testing Antipatterns.

3.6.1 - Component Tests

Definition

Verification of a coherent structural unit (such as an entire microservice, UI component, or self-contained subsystem) against its specific contract and internal logic, while keeping interactions beyond that component’s boundary mocked or stubbed.

Scope & Boundaries

Broader than a unit test, but strictly narrower than an end-to-end (E2E) integration test. It tests the interplay of multiple internal classes/modules working together within that component. Out-of-process network calls and external downstream services are replaced by API wire-level stubs or in-memory equivalents (e.g., WireMock, MSW, ephemeral test containers).

Core Characteristics

Validates state management, internal workflows, data transformations, and edge-to-edge behavior within a bounded context without taking dependencies on third-party uptime or network latency.

Good Practices

• Mock only at boundary borders: Exercise the component’s internal routing, controllers, domain models, and data mappers together; only mock external HTTP APIs, message brokers, or remote databases.
• Use ephemeral infrastructure: Use fast, disposable local resources (e.g., local SQLite/Postgres in Docker, local WireMock) to mirror real component runtime characteristics.
• Versioned, repeatable test data.
• Verify contract-to-state workflows: Validate that boundary inputs result in the correct local state changes and expected outgoing network payloads.

Anti-Patterns

• E2E scope creep: Allowing the test to call live third-party services or dependent microservices instead of wire-level stubs.
• Re-testing granular unit logic: Writing dozens of micro-permutations of input edge cases at the component level instead of covering them in fast unit tests.
• Leaky test harness state: Failing to purge in-memory databases or reset wire stubs between runs, leading to non-deterministic test flakiness.

When to Avoid

They overlap heavily with other layers when the component is:

• Thin CRUD with no middleware to speak of. Provider contract verification against a booted app plus sociable unit tests of the domain cover most of what a component test would. Keep one per critical flow as smoke coverage; skip exhaustive component coverage.
• Utility libraries are effectively multiple small components in as a single consumable dependency.
• Pure transformation logic. Parsers, calculators, scheduling math. Unit tests give better coverage per unit of effort.

If you’re choosing between an extra component test and an extra unit test for the same behavior, the unit test is cheaper to write, run, and maintain. Component tests earn their keep at the seams between layers, not in repeating ground that unit tests already cover.

Examples

Backend Service

A component test for a REST API, exercising the full application stack with the downstream inventory service replaced by a test double:

describe("POST /orders", () => {
  it("should create an order and return 201", async () => {
    // Arrange: mock the inventory service response
    httpMock("https://inventory.internal")
      .onGet("/stock/item-42")
      .reply(200, { available: true, quantity: 10 });

    // Act: send a request through the full application stack
    const response = await request(app)
      .post("/orders")
      .send({ itemId: "item-42", quantity: 2 });

    // Assert: verify the public interface response
    expect(response.status).toBe(201);
    expect(response.body.orderId).toBeDefined();
    expect(response.body.status).toBe("confirmed");
  });

  it("should return 409 when inventory is insufficient", async () => {
    httpMock("https://inventory.internal")
      .onGet("/stock/item-42")
      .reply(200, { available: true, quantity: 0 });

    const response = await request(app)
      .post("/orders")
      .send({ itemId: "item-42", quantity: 2 });

    expect(response.status).toBe(409);
    expect(response.body.error).toMatch(/insufficient/i);
  });
});

Frontend Component

A component test exercising a login flow with a stubbed authentication service:

describe("Login page", () => {
  it("should redirect to the dashboard after successful login", async () => {
    mockAuthService.login.mockResolvedValue({ token: "abc123" });

    render(<App />);
    await userEvent.type(screen.getByLabelText("Email"), "ada@example.com");
    await userEvent.type(screen.getByLabelText("Password"), "s3cret");
    await userEvent.click(screen.getByRole("button", { name: "Sign in" }));

    expect(await screen.findByText("Dashboard")).toBeInTheDocument();
  });
});

Accessibility Verification

Component tests already exercise the UI from the actor’s perspective, making them the natural place to verify that interactions work for all users. Accessibility assertions fit alongside existing assertions rather than in a separate test suite.

This is the second of three tiers in the Accessibility testing strategy: static-analysis linting catches structural violations in source, component tests catch the rendered-only ones (computed contrast, focus order, keyboard operability), and manual audits cover the subjective remainder.

// accessibility scanner setup

describe("Checkout flow", () => {
  it("should be completable using only the keyboard", async () => {
    render(<CheckoutPage />);

    await userEvent.tab();
    expect(screen.getByLabelText("Card number")).toHaveFocus();

    await userEvent.type(screen.getByLabelText("Card number"), "4111111111111111");
    await userEvent.tab();
    await userEvent.type(screen.getByLabelText("Expiry"), "12/27");
    await userEvent.tab();
    await userEvent.keyboard("{Enter}");

    expect(await screen.findByText("Order confirmed")).toBeInTheDocument();

    const results = await accessibilityScanner(document.body);
    expect(results).toHaveNoViolations();
  });
});

Connection to CD Pipeline

Component tests run after unit tests in the pipeline, but before longer running acceptance tests, and provide the broadest fast, deterministic feedback:

1. Local development: run before committing. Deterministic scope keeps them fast enough to run locally without slowing the development loop.
2. PR verification: CI executes the full suite; failures block merge.
3. Trunk verification: the same tests run on the merged HEAD to catch conflicts.

They should always halt the CD pipeline on failure.

3.6.2 - Contract Tests

Definition

Verification that two separate systems (such as an API provider and its consumers, or a message publisher and subscriber) adhere to a shared, agreed-upon formal specification, the “contract”, without requiring both services to run simultaneously in an integrated environment.

Scope & Boundaries

Targets only the boundary interface: request payloads, query parameters, HTTP headers, response schemas, status codes, or message formats. It does not test internal business logic, database state, or deep end-to-end user journeys; it solely verifies compatibility with the interface schema (e.g., OpenAPI/Swagger, AsyncAPI, Protobuf).

Characteristics

Fast, independent execution across pipelines, prevents breaking schema changes before deployment, and eliminates the need for expensive, flaky end-to-end integration environments in applications with proper domain separation.

Good Practices:

• Strict schema adherence: Explicitly define nullability, enums, required fields, and format constraints rather than relying on loose, open schemas.
• Automated breaking-change detection: Integrate tools like oasdiff or buf breaking into CI to catch backwards-incompatible schema changes on pull requests.
• Generate stubs directly from contracts: Use contract-driven mock engines (e.g., Prism, Microcks) so mock behavior automatically updates whenever the contract changes.

Anti-Patterns:

• Testing business logic via contracts: Attempting to verify authorization rules, complex workflows, or algorithmic computations inside a contract test.
  • Example: testing that a request for a user return a specific user instead of the expected user object.
• Hand-crafted, unverified mock fixtures: Manually updating JSON response stubs in consumer code repos without validating them against the live contract artifact.
• All-or-nothing megaspecs: Coupling unrelated domains or multiple service interfaces into a single monolithic contract document that cannot be versioned or evolved independently.

Validating the Contract

A contract test only proves your code matches the contract - not that the contract still matches the real system. Integration tests close that gap by running against the real dependency. How tightly that loop closes depends on collaboration level: low collaboration means scheduled integration tests against the real system with no shared tooling; high collaboration adds a hosted specification server (Pact, Pacto), a specification fetched from a shared or provider repository at build time, or a provider webhook that triggers the consumer’s CI when the contract changes.

Consumer and Provider Perspectives

A contract has two sides asking different questions. The consumer asks whether the fields, types, and status codes it depends on still exist. Consumer tests assert only on the subset of the API the consumer actually uses, not the whole response - following Postel’s Law, be liberal in what you accept and conservative in what you send.

The provider asks whether its changes will break any consumer. A provider test runs every published consumer expectation against the real implementation, catching a removed field, a changed type, or altered error behavior before a consumer deploys and discovers the break.

Contract-First Development

The interface is defined as a formal artifact - an OpenAPI, Protobuf, or AsyncAPI spec - before either side writes an implementation. Consumer and provider teams build independently against that artifact, then verify conformance to the spec rather than to each other’s code. Works best for new APIs and parallel development, where there’s no existing implementation to write consumer-driven contracts against.

Examples

A consumer contract test using a consumer-driven contract tool:

describe("Order Service - Inventory Provider Contract", () => {
  it("should receive stock availability in the expected format", async () => {
    // Define what the consumer expects from the provider
    await contractTool.addInteraction({
      state: "item-42 is in stock",
      uponReceiving: "a request for item-42 stock",
      withRequest: { method: "GET", path: "/stock/item-42" },
      willRespondWith: {
        status: 200,
        body: {
          // Only assert on fields the consumer actually uses
          available: matchType(true),   // boolean
          quantity: matchType(10),      // integer
        },
      },
    });

    // Exercise the consumer code against the mock provider
    const result = await inventoryClient.checkStock("item-42");
    expect(result.available).toBe(true);
  });
});

A provider verification test that runs consumer expectations against the real implementation:

describe("Inventory Service - Provider Verification", () => {
  it("should satisfy all registered consumer contracts", async () => {
    await contractBroker.verifyProvider({
      provider: "InventoryService",
      providerBaseUrl: "http://localhost:3001",
      brokerUrl: "https://contract-broker.internal",
      providerVersion: process.env.GIT_SHA,
    });
  });
});

A contract-first schema validation test verifying a provider response against an OpenAPI spec:

// The OpenAPI document is the source of truth. Validate the whole response
// against the named schema rather than hand-checking individual fields - a
// field-by-field check drifts from the spec the moment the spec changes.
const validator = openApiValidator(openApiSpec);

describe("GET /stock/:id - OpenAPI contract", () => {
  it("should return a response conforming to the published schema", async () => {
    const response = await fetch("http://localhost:3001/stock/item-42");
    const body = await response.json();

    expect(response.status).toBe(200);

    // Asserts structure, types, required fields, and additionalProperties
    // rules exactly as the OpenAPI schema declares them.
    const result = validator.validate(body, "StockResponse");
    expect(result.errors).toEqual([]);
  });
});

Connection to CD Pipeline

Contract tests run after unit tests in the pipeline:

1. Local development: run before committing. Deterministic scope keeps them fast enough to run locally without slowing the development loop.
2. PR verification: CI executes the full suite; failures block merge.
3. Trunk verification: the same tests run on the merged HEAD to catch conflicts.

They should always halt the CD pipeline on failure.

3.6.3 - End-to-End Tests

Definition

Verification of a complete end-to-end user or business transaction through the entire deployed application stack, matching the perspective and experience of an actual user or external consumer.

Scope & Boundaries

Encompasses the entire system topology—from the frontend UI or external API gateway through all internal microservices, asynchronous workers, live queues, databases, and necessary third-party sandbox integrations.

Core Characteristics

Highest real-world confidence, highest execution cost, slowest run time, and highest vulnerability to environment or network-induced flakiness.

Good Practices

• Restrict to critical revenue/operational paths: Focus E2E coverage strictly on non-negotiable user journeys (e.g., user registration, primary checkout, key ingest pipelines).
• Automate environment provisioning: Deploy ephemeral, on-demand preview environments to run E2E suites and tear them down immediately upon completion.
• Implement resilient element selection: Select UI elements using accessibility roles or stable data attributes (e.g., data-testid) rather than fragile CSS classes or absolute XPath selectors.

Anti-Patterns

• Using E2E tests for regression safety nets: Relying on E2E suites to catch regressions that could have been detected upstream in unit, component, or contract stages (the “inverted testing pyramid”).
• Arbitrary thread sleeps: Adding fixed pauses (e.g., sleep(5)) to wait for asynchronous events rather than using explicit, condition-driven polling.
• Accepting flaky tests: Rerunning failing E2E tests until they turn green rather than quarantining and fixing the underlying timing or state issues immediately.

Weaknesses & Challenges

• High Flakiness and Low Signal-to-Noise Ratio: Non-deterministic failures are common. Network blips, browser rendering lag, race conditions in asynchronous frontend frameworks, and transient third-party service outages often cause false-negative test failures that erode developer trust.
• Poor Root-Cause Localization: When an E2E test fails with a generic error (e.g., TimeoutError: Element #confirmation-banner not found), finding the source of the issue requires combing through client logs, gateway routes, backend microservice traces, and database state to determine what actually broke.
• Environment Maintenance & Resource Cost: E2E suites typically demand fully integrated staging or preview environments. Keeping these environments populated with realistic test data, configured with active credentials, and synchronized across dozens of microservices is notoriously resource-intensive.
• Prohibitive Execution Times: Running full browser automation or multi-service distributed flows can take anywhere from tens of minutes to several hours. This latency breaks continuous delivery flow, encouraging teams to defer testing to late-stage, batch-processed pipelines rather than getting instant feedback on change.
• Tight Coupling to Volatile UI/API Layouts: Small cosmetic changes (like modifying class names, reordering markup, or tweaking a multi-step user flow) often break brittle E2E tests even though the underlying business capability remains completely functional.

Examples

import { test, expect } from '@playwright/test';

test('user can complete entire purchase flow', async ({ page }) => {
  // Navigates real UI against a fully deployed environment
  await page.goto('https://checkout.staging.example.com');
  await page.fill('#username', 'test_user');
  await page.fill('#password', 'SecurePass123!');
  await page.click('button[type="submit"]');

  // Add item to cart and initiate purchase
  await page.click('button[data-item="product-42"]');
  await page.click('#cart-checkout');
  await page.fill('#card-element', '4242424242424242');
  await page.click('#submit-payment');

  // Confirms response propagated across API, async billing, and UI rendering
  await expect(page.locator('.order-confirmation')).toHaveText(/Order #\d+ Confirmed/);
});

When to Use / Avoid

Use them for:

• Happy-path validation of critical business flows that cannot be verified any other way (e.g., a payment flow that depends on a real payment provider).
• Entangled domain workflows that span multiple deployables and cannot be isolated within a single component test.

They are the most expensive test type to write, run, and maintain. Use them sparingly.

Avoid for:

• Edge cases, error handling, or input validation. Those scenarios belong in unit or component tests.

Connection to CD Pipeline

E2E tests should only run in the pipeline as part of the longer running acceptance tests if they can be made dependable and deterministic. Otherwise, they should be run on a schedule and not act as a delivery decision.

3.6.4 - Integration Tests

Definition

Verification that two or more distinct architectural subsystems or external dependencies interact correctly across their transport layer and data boundaries.

Scope & Boundaries

Broader than a component test because it explicitly validates communication with real external systems (such as a database, message queue, cache, or filesystem), but narrower than a full end-to-end test because it targets a specific integration boundary rather than complete multi-service user workflows.

Core Characteristics

Detects driver/dialect mismatches, schema serialization issues, connection pooling misconfigurations, and ORM/SQL query errors that mock-heavy tests overlook.

Good Practices

• Use disposable, ephemeral infrastructure: Spin up real databases and queues using container tooling (e.g., Testcontainers) rather than using shared, persistent static environments.
• Verify transport-level error handling: Intentionally test connection timeouts, pool exhaustion, network blips, and transaction rollbacks.
• Run tests against clean boundary state: Truncate tables, flush caches, and clear queues between test runs to guarantee deterministic execution.

Anti-Patterns

• Using shared remote environments: Pointing integration test suites to shared dev/staging databases, causing data collisions and race conditions between concurrent CI jobs.
• Testing business permutations: Testing dozens of conditional logic branches through real databases instead of pushing that logic down to fast unit tests or leveraging component tests.
• Ignoring production parity: Testing against an SQLite in-memory database locally when production runs Postgres, masking dialect, constraint, and indexing differences.

Weaknesses & Challenges

• Infrastructure Orchestration Overhead: Requires managing real databases, message brokers, and caches inside the test execution context. Maintaining container definitions (e.g., Docker/Testcontainers) and keeping schema migrations up to date adds operational burden to developers.
• Test Isolation and State Contamination: When tests write real rows to a database or publish messages to an active broker, dirty state from one test can bleed into another. Cleaning, truncating, or rolling back transactions between runs adds latency and complexity.
• Slow Pipeline Feedback Cycles: Because integration tests involve real I/O, network socket handshakes, and disk writes, they are orders of magnitude slower than in-memory unit tests. Over-relying on them severely bloats commit-stage feedback loops.
• Local vs. Production Discrepancies: Test-specific database configurations, lightweight containerized replicas, or local mocks often mask subtle production issues—such as database clustering behavior, regional latency, connection pool limits, or privilege boundaries.

Examples

import pytest
from sqlalchemy import create_engine
from myapp.storage import OrderRepository, Order

# Runs against an actual ephemeral Postgres container, not an in-memory mock
def test_order_repository_persists_and_retrieves_roundtrip(real_pg_session):
    repo = OrderRepository(session=real_pg_session)
    new_order = Order(order_id=101, customer_id="cust_abc", total=49.99)

    repo.save(new_order)
    retrieved = repo.find_by_id(101)

    assert retrieved is not None
    assert retrieved.customer_id == "cust_abc"
    assert retrieved.total == 49.99

Connection to CD Pipeline

Integration tests should only run in the pipeline as part of the longer running acceptance tests if they can be made dependable and deterministic. Otherwise, they should be run on a schedule and not act as a delivery decision.

3.6.5 - Static Analysis

Definition

Static analysis (also called static testing) evaluates non-running code against rules for known good practices, inspecting source, configuration, and dependency manifests to catch errors, complexity, and security issues before the code ever runs.

Scope & Boundaries

Analysis runs against source code, configuration files, and dependency manifests at rest - no application starts and no test doubles are needed. Scope is the entire codebase, not a single unit, component, or transaction.

Characteristics

Seconds-scale execution (the fastest test category), fully deterministic, and codebase-wide scope, with no external dependencies except the network calls a dependency scanner makes to a vulnerability database.

Good Practices

• Run it everywhere feedback is possible: IDE plugins, pre-commit hooks, and CI each catch issues before the next stage makes them more expensive to fix.
• Customize the ruleset: default rules are a starting point; add rules for patterns that keep coming up in code review.
• Enforce it as a gate: treat lint, type, and security findings as build-breaking, the same as a failing test.

Anti-Patterns

• Disabling rules instead of fixing code: suppressing linter warnings or ignoring security findings erodes the value of static analysis over time.
• Skipping ruleset customization: default rules are a starting point, not a ceiling, for patterns specific to the codebase.
• Running static analysis only in CI: by the time CI reports a formatting error, the developer has context-switched; IDE and pre-commit feedback catch it sooner.
• Ignoring dependency vulnerabilities: known CVEs in dependencies are a direct attack vector and should break the build.
• Treating static analysis as optional: if developers can bypass the checks, they will.

When to Run It

• In the IDE: real-time feedback as developers type, via editor plugins and language server integrations.
• On save: format-on-save and lint-on-save catch issues immediately.
• Pre-commit: hooks prevent problematic code from entering version control.
• In CI: the full suite of static checks runs on every PR and on the trunk after merge, verifying that earlier local checks were not bypassed.

Static analysis is always applicable. Every project, regardless of language or platform, benefits from linting, formatting, and dependency scanning.

Examples

Linting

A .eslintrc.json configuration enforcing test quality rules:

{
  "rules": {
    "no-disabled-tests": "warn",
    "require-assertions": "error",
    "no-commented-out-tests": "error",
    "valid-assertions": "error",
    "no-unused-vars": "error",
    "no-console": "warn"
  }
}

Type Checking

Statically typed languages catch type mismatches at compile time, eliminating entire classes of runtime errors. Java, for example, rejects incompatible argument types before the code runs:

public static double calculateTotal(double price, int quantity) {
    return price * quantity;
}

// Compiler error: incompatible types: String cannot be converted to double
calculateTotal("19.99", 3);

Dependency Scanning

Dependency scanning tools scan for known vulnerabilities:

$ npm audit
found 2 vulnerabilities (1 moderate, 1 high)
  moderate: Prototype Pollution in lodash < 4.17.21
  high:     Remote Code Execution in log4j < 2.17.1

Types of Static Analysis

Accessibility Linting

Accessibility linting catches deterministic WCAG violations the same way a security scanner catches known vulnerability patterns. Automated checks cover structural issues (missing alt text, invalid ARIA attributes, insufficient contrast ratios, broken heading hierarchy) while manual review covers subjective aspects like whether alt text is actually meaningful.

Linting is the first of three tiers. For how it fits with component-test DOM scans and manual audits across the pipeline - and the caveat that automated checks catch only a fraction of WCAG criteria - see Accessibility testing.

An accessibility checker configuration running WCAG 2.1 AA checks against rendered pages:

{
  "defaults": {
    "standard": "WCAG2AA",
    "timeout": 10000,
    "wait": 1000
  },
  "urls": [
    "http://localhost:1313/docs/",
    "http://localhost:1313/docs/foundations/testing-fundamentals/test-architecture/"
  ]
}

An accessibility scanner test asserting that a rendered component has no violations:

// accessibility scanner setup (e.g. import scanner and extend assertions)

it("should have no accessibility violations", async () => {
  const { container } = render(<LoginForm />);
  const results = await accessibilityScanner(container);
  expect(results).toHaveNoViolations();
});

Connection to CD Pipeline

Static analysis is the first gate in the CD pipeline, providing the fastest feedback:

1. IDE / local development: plugins run in real time as code is written.
2. Pre-commit: hooks run linters, formatters, and accessibility checks on changed components, blocking commits that violate rules.
3. PR verification: CI runs the full static analysis suite (linting, type checking, security scanning, dependency auditing, accessibility linting) and blocks merge on failure.
4. Trunk verification: the same checks re-run on the merged HEAD to catch anything missed.
5. Scheduled scans: dependency and security scanners run on a schedule to catch newly disclosed vulnerabilities in existing dependencies.

Because it requires no running code and no external dependencies, static analysis is the cheapest, fastest gate. A mature CD pipeline treats its failures like any other test failure: they break the build.

3.6.6 - Smoke Tests

Definition

A lightweight, automated verification run executed immediately after code is deployed to a live target environment (staging, pre-production, or production) to verify that the environment is healthy, operational, and capable of handling traffic before fully shifting user load.

Scope & Boundaries

Strictly limited to high-level system vitality. It checks that critical infrastructure components (processes, routing, database connectivity, secret access, core endpoints) are alive and reachable. It explicitly avoids deep workflow testing, exhaustive edge-case permutations, or long-running operational flows.

Core Characteristics

Fast (seconds to 2–3 minutes max), strictly non-destructive/read-only, high criticality, and directly tied to deployment orchestration (triggers immediate automated rollback or stops traffic migration if it fails).

Examples

import requests
import sys

BASE_URL = "https://checkout.internal.net"

def run_smoke():
    # 1. Deep health check (probes DB connection, cache, and queue availability)
    health = requests.get(f"{BASE_URL}/healthz/ready", timeout=5)
    assert health.status_code == 200, f"Health check failed: {health.text}"
    assert health.json().get("database") == "UP"

    # 2. Key static config / version check
    version = requests.get(f"{BASE_URL}/version", timeout=3)
    assert version.status_code == 200
    assert version.json().get("commit_sha") == sys.argv[1]

    # 3. Non-destructive read query against a core endpoint
    catalog = requests.get(f"{BASE_URL}/api/v1/products/featured", timeout=5)
    assert catalog.status_code == 200
    assert len(catalog.json()) > 0

if __name__ == "__main__":
    run_smoke()

A Java sociable unit test exercising real domain logic through its public interface. The collaborators (the pricing policy and the order model) are real objects, not mocks, and the test asserts on the observable outcome - the computed total - rather than on which methods were called:

@Test
public void appliesBulkDiscountWhenQuantityReachesThreshold() {
    // Arrange: real collaborators, no test doubles - this is pure in-process logic
    PricingPolicy pricing = new PricingPolicy(
        bulkThreshold(10), bulkDiscountRate(0.15));
    Order order = new Order(new LineItem("widget", money("20.00"), quantity(12)));

    // Act
    Money total = pricing.totalFor(order);

    // Assert: the observable result, not the sequence of internal calls
    // 12 * 20.00 = 240.00, less 15% = 204.00
    assertEquals(money("204.00"), total);
}

@Test
public void chargesFullPriceBelowTheThreshold() {
    PricingPolicy pricing = new PricingPolicy(
        bulkThreshold(10), bulkDiscountRate(0.15));
    Order order = new Order(new LineItem("widget", money("20.00"), quantity(9)));

    assertEquals(money("180.00"), pricing.totalFor(order));
}

Good Practices

• Design for idempotency and read-only behavior: Keep smoke tests non-destructive so they can run safely against live production environments without corrupting customer data, charging credit cards, or sending false operational emails.
• Target deep health endpoints: Use application endpoints that actively verify connectivity to backend resources (database reads, Redis caches, downstream dependency reachability) rather than shallow /ping endpoints that only return 200 OK from the web server process.
• Automate immediate rollback gates: Tie smoke test outcomes directly into your CD pipeline (e.g., progressive delivery, canary, or blue/green deployments). If the smoke suite fails, the deployment halts and rolls back automatically without human intervention.
• Verify deployment identity: Assert that the deployed system is serving the exact build artifact, tag, or Git commit hash intended for the deployment to catch caching or orchestration misconfigurations. Anti-Patterns
• Mutating real production data: Creating synthetic test users, modifying real records, or generating phantom financial transactions without rigorous isolation or synthetic-data isolation strategies.
• Bloating into a full regression suite: Packing dozens of detailed integration checks into the smoke run, slowing pipeline execution from seconds into tens of minutes and defeating the purpose of a fast sanity gate.
• Ignoring downstream read timeouts: Setting long or infinite request timeouts that cause the deployment pipeline to hang indefinitely when an infrastructure route or firewall rule is misconfigured.
• Running exclusively in staging: Skipping smoke tests in production under the false assumption that passing staging guarantees environment-specific configs, network policies, and IAM roles are correctly wired in production. Weaknesses & Challenges
• Coarse-Grained Blind Spots: Smoke tests only verify that the front door is open and core pipes are connected; they cannot catch subtle business regressions, boundary calculation errors, or minor UI rendering glitches.
• Risk of Production State Side Effects: If write checks are included, synthetic data can leak into operational reports, analytics dashboards, or customer views, requiring custom clean-up routines that are prone to failure.
• Permissions and Security Boundaries: Probing internal services and operational health endpoints in locked-down production environments often requires elevated network routes or secure service tokens that must be strictly audited and maintained.
• Handling Transient Startup Latency: Newly launched containers, warm-up caches, or JIT compilation can cause false-positive smoke failures immediately following a rollout if proper readiness probes and retry loops are not configured.

Connection to CD Pipeline

Smoke tests are run after every deploy to validate the deploy. They are also used to trigger auto-rollback in the pipeline if they don’t pass.

3.6.7 - Unit Tests

Definition

Verification of the smallest testable piece of code—typically a single function, method, or class—in complete isolation from the rest of the application, network, file system, or external services. Test doubles are used where needed.

Scope & Boundaries

Execution runs entirely in-memory. All external dependencies (databases, APIs, message brokers, system clocks) are replaced with test doubles (stubs, mocks, or fakes).

Characteristics

Millisecond execution speeds, highly deterministic (zero flakiness), and pinpoint failure localization.

Good Practices:

• Test public behavior, not implementation details: Assert on return values and visible side effects rather than internal private state or execution paths.
• Strict isolation: Keep all tests in-memory; mock or stub out network, disk I/O, database, and system time to ensure sub-millisecond execution.
• Single assertion concept: Each test should verify one specific behavior or edge case to maintain pinpoint failure localization.

Anti-Patterns

• Over-mocking: Mocking domain entities, data transfer objects, or language primitives instead of purely external/infrastructure boundaries.
• Testing private methods: Forcing visibility or coupling tests to internal helper methods, which causes refactoring resistance without increasing behavioral confidence.
• Inter-test dependencies: Letting execution order matter or sharing mutable global state between test cases.

Solitary vs. sociable unit tests

A solitary unit test replaces all collaborators with test doubles. A sociable unit test allows real in-process collaborators while still replacing any external I/O. Both styles are unit tests as long as no real external dependency is involved.

When to Run Them

• During development: run the relevant subset of unit tests continuously while writing code. TDD (Red-Green-Refactor) is the most effective workflow.
• On every commit: use pre-commit hooks or watch-mode test runners so broken tests never reach the remote repository.
• In CI: execute the full unit test suite on every pull request and on the trunk after merge to verify nothing was missed locally.

Unit tests are the right choice when the behavior under test can be exercised without network access, file system access, or database connections. If you need any of those, you likely need a component test or an end-to-end test instead.

Examples

// castArray.test.js
describe("castArray", () => {
  it("should wrap non-array items in an array", () => {
    expect(castArray(1)).toEqual([1]);
    expect(castArray("a")).toEqual(["a"]);
    expect(castArray({ a: 1 })).toEqual([{ a: 1 }]);
  });

  it("should return array values by reference", () => {
    const array = [1];
    expect(castArray(array)).toBe(array);
  });

  it("should return an empty array when no arguments are given", () => {
    expect(castArray()).toEqual([]);
  });
});

A Java sociable unit test exercising real domain logic through its public interface. The collaborators (the pricing policy and the order model) are real objects, not mocks, and the test asserts on the observable outcome - the computed total - rather than on which methods were called:

@Test
public void appliesBulkDiscountWhenQuantityReachesThreshold() {
    // Arrange: real collaborators, no test doubles - this is pure in-process logic
    PricingPolicy pricing = new PricingPolicy(
        bulkThreshold(10), bulkDiscountRate(0.15));
    Order order = new Order(new LineItem("widget", money("20.00"), quantity(12)));

    // Act
    Money total = pricing.totalFor(order);

    // Assert: the observable result, not the sequence of internal calls
    // 12 * 20.00 = 240.00, less 15% = 204.00
    assertEquals(money("204.00"), total);
}

@Test
public void chargesFullPriceBelowTheThreshold() {
    PricingPolicy pricing = new PricingPolicy(
        bulkThreshold(10), bulkDiscountRate(0.15));
    Order order = new Order(new LineItem("widget", money("20.00"), quantity(9)));

    assertEquals(money("180.00"), pricing.totalFor(order));
}

Connection to CD Pipeline

Unit tests run in the earliest stages of the CD pipeline and provide the fastest feedback loop:

1. Local development: watch mode reruns tests on every save.
2. Pre-commit: hooks run the suite before code reaches version control.
3. PR verification: CI runs the full suite and blocks merge on failure.
4. Integrated change verification: CI reruns tests on the merged HEAD to catch integration issues.

They should always halt the CD pipeline on failure.

3.7 - Patterns

Each page in this subsection covers one component pattern. The structure is the same on every page so you can scan-compare:

1. What needs covered - the layers of testing the pattern typically benefits from.
2. Positive test cases - common success behaviors worth testing.
3. Negative test cases - common failure modes that produce production incidents.
4. Test double validation - how the doubles in pipeline tests stay honest.
5. Pipeline placement - where each test type tends to run.
6. Example - a short code sample illustrating one of the harder cases for that pattern.

These are recommended starting points, not exhaustive lists or required gates. Real components have details these pages don’t capture; ignore items that don’t apply, and add items the pattern doesn’t mention but your component clearly needs. The goal is to prompt the conversation, not to constrain it.

API provider, API consumer, scheduled job, and user interface are covered in depth. Event consumer, event producer, CLI/library, and stateful service are deliberately briefer sketches: the same six principles apply, the same checklist still prompts useful questions, and the test double validation model is the same. Use the briefer sketches as a starting point and expand the depth in your own runbooks for the patterns your services actually use.

The patterns

• API provider - a backend service exposing an HTTP/gRPC/GraphQL API and owning its own data.
• API consumer - the above, plus outbound calls to other services. The most failure-prone pattern.
• Scheduled job - a service triggered on a cron, queue, or external scheduler.
• User interface - a UI that renders data and accepts user interaction.
• Event consumer - a service that consumes messages from a broker.
• Event producer - a service that produces messages to a broker.
• CLI tool or library - a binary or package consumed by other developers.
• Stateful service - a service that maintains long-lived in-memory state.

3.7.1 - API Provider

A backend service that exposes an HTTP/gRPC/GraphQL API and owns its own data. No outbound calls to other services in your control.

What needs covered

Positive test cases

Common cases to consider, not an exhaustive list. Drop items that don’t apply and add ones the pattern doesn’t mention but your component needs.

• Documented endpoints: return the expected shape and status for valid input.
• Auth: succeeds for valid credentials and tokens.
• Pagination, filtering, sorting: all return the documented results.
• Idempotency: idempotent operations are idempotent; non-idempotent operations create exactly one record.
• Success-path side effects: events emitted and audit log entries happen on the success path.

Negative test cases

Common cases to consider, not an exhaustive list. Drop items that don’t apply and add ones the pattern doesn’t mention but your component needs.

• Malformed body: bad JSON, missing required fields, wrong types, extra fields handled per the documented policy (reject vs. ignore).
• Out-of-range values: negatives where positives are expected, oversize strings, unicode edge cases.
• Auth failures: missing token, expired token, valid token with insufficient scope, valid token for a different tenant.
• Authorization boundaries: user A cannot read or modify user B’s resources.
• Resource not found: referenced IDs don’t exist, return 404 not 500.
• Concurrency: two writes to the same resource at once, optimistic-lock conflict handled with the documented status code.
• Persistence failure: DB unavailable, deadlock, constraint violation. The error envelope is correct and no partial state is committed.
• Rate limiting and request size limits: both enforce as documented.
• Idempotency under retry: same idempotency key within the window returns the original result, not a duplicate write.

Test double validation

Doubles in this pattern are mostly around persistence. Two layers keep them honest:

1. Adapter integration tests run against a real instance of your production database engine (the same major version, same extensions). If component tests use an in-memory SQLite shim while production runs Postgres, the shim is the lie. The adapter integration test exercises every query and migration against a Postgres testcontainer in CI.
2. Provider-side contract tests verify the API still satisfies every published consumer expectation. See Consumer and Provider Perspectives. Provider verification is where you discover that a “harmless” field rename broke a consumer before that consumer deploys.

Pipeline placement

• Unit + sociable unit tests: pre-commit and CI Stage 1.
• Adapter integration tests against testcontainers: CI Stage 1 if fast, Stage 2 otherwise.
• Component tests: CI Stage 1.
• Provider-side contract verification: CD Stage 1 (Contract and Boundary Validation).

Example: component test

A flow-oriented component test for an order-placement endpoint. The full app is assembled with an in-memory order repository and an in-memory event bus. The test drives the assembled component through its HTTP handlers and asserts on observable outcomes (status, persisted state, emitted event):

• Java
• C#
• JavaScript

@SpringBootTest
@AutoConfigureMockMvc
class OrderPlacementTest {

  @Autowired MockMvc mvc;
  @Autowired InMemoryOrderRepo orderRepo;
  @Autowired InMemoryEventBus events;

  @Test
  void places_order_with_valid_payment_creates_order_and_emits_OrderPlaced() throws Exception {
    var body = """
      { "items": [{"sku": "A1", "qty": 2}], "paymentToken": "pm_ok" }
      """;

    var result = mvc.perform(post("/orders")
        .header("Authorization", "Bearer tok_valid")
        .contentType(APPLICATION_JSON)
        .content(body))
      .andExpect(status().isCreated())
      .andReturn();

    var orderId = JsonPath.<String>read(result.getResponse().getContentAsString(), "$.id");
    assertThat(orderRepo.findById(orderId)).isPresent();
    assertThat(events.published()).anyMatch(e ->
        e.type().equals("OrderPlaced") && e.orderId().equals(orderId));
  }
}

public class OrderPlacementTests : IClassFixture<WebApplicationFactory<Program>>
{
    private readonly HttpClient client;
    private readonly InMemoryOrderRepo orderRepo = new();
    private readonly InMemoryEventBus events = new();

    public OrderPlacementTests(WebApplicationFactory<Program> factory)
    {
        client = factory.WithWebHostBuilder(b => b.ConfigureServices(s =>
        {
            s.AddSingleton<IOrderRepo>(orderRepo);
            s.AddSingleton<IEventBus>(events);
        })).CreateClient();
    }

    [Fact]
    public async Task Places_order_with_valid_payment_creates_order_and_emits_OrderPlaced()
    {
        client.DefaultRequestHeaders.Authorization = new("Bearer", "tok_valid");
        var body = new { items = new[] { new { sku = "A1", qty = 2 } }, paymentToken = "pm_ok" };

        var response = await client.PostAsJsonAsync("/orders", body);

        response.StatusCode.Should().Be(HttpStatusCode.Created);
        var created = await response.Content.ReadFromJsonAsync<OrderCreated>();
        orderRepo.FindById(created!.Id).Should().NotBeNull();
        events.Published.Should().Contain(e =>
            e.Type == "OrderPlaced" && e.OrderId == created.Id);
    }
}

import request from "supertest";
import { buildApp } from "./app.js";
import { InMemoryOrderRepo } from "./test/in-memory-order-repo.js";
import { InMemoryEventBus } from "./test/in-memory-event-bus.js";

test("places order with valid payment creates order and emits OrderPlaced", async () => {
  const orderRepo = new InMemoryOrderRepo();
  const events = new InMemoryEventBus();
  const app = buildApp({ orderRepo, events });

  const res = await request(app)
    .post("/orders")
    .set("Authorization", "Bearer tok_valid")
    .send({ items: [{ sku: "A1", qty: 2 }], paymentToken: "pm_ok" });

  expect(res.status).toBe(201);
  expect(orderRepo.findById(res.body.id)).toBeDefined();
  expect(events.published).toContainEqual(
    expect.objectContaining({ type: "OrderPlaced", orderId: res.body.id })
  );
});

The test asserts on what a real caller can observe, not on private methods or call sequences inside the controller.

3.7.2 - API Consumer

Same as API provider, plus outbound HTTP/gRPC calls to services the team does not own (or does own but deploys independently). This is the most failure-prone pattern in distributed systems and gets the most testing attention.

What needs covered

Everything from the API provider pattern, plus:

Positive test cases

Common cases to consider, not an exhaustive list. Drop items that don’t apply and add ones the pattern doesn’t mention but your component needs.

• Outbound call: constructs the right URL, headers, body, auth, and timeout.
• Success response: parsed correctly, including optional fields and unknown fields per Postel’s Law.
• Multi-call composition: multiple downstream calls in sequence or parallel produce the documented composite response.
• Caching: returns the cached value within TTL and refreshes after.
• Trace context: propagates downstream.

Negative test cases

Common cases to consider, not an exhaustive list. The bulk of the negative testing happens here, and it’s where most production incidents originate. Drive each failure mode through a client double that simulates it.

• Timeout (downstream exceeds configured deadline): the deadline enforces; the upstream caller gets the documented response (e.g., 504); no partial state is committed. Use a client double that delays past the deadline.
• Connection refused: retry policy executes the documented count and backoff; falls over to fallback or returns an error. Use a client double that rejects the connection.
• 5xx responses (500, 502, 503): retry only on retryable codes. Use a client double that returns 5xx.
• 4xx responses (400, 401, 403, 404, 409, 422, 429): each maps to documented behavior; 4xx generally not retried; 429 respects Retry-After. Use a client double that returns each code.
• Slow response within timeout: performance-budget assertions hold if the service has SLO commitments. Use a client double that delays within the deadline.
• Malformed response body: the response is rejected, not silently coerced. Use a client double that returns a truncated or wrong-type body.
• Schema drift (extra or missing fields): extra fields tolerated; missing required fields detected with a clear error. Use a client double that returns a drifted body.
• Wrong status code (200 with error body, 500 with success body): the client trusts the status code, not the body. Use a client double that returns mismatched status and body.
• Circuit open: the circuit opens under sustained failure; fast-fails subsequent calls; recovers on a half-open probe. Use a client double that sustains failures.
• Partial multi-call failure: compensation, rollback, or documented partial-success behavior. First client double succeeds, second fails.

Test double validation

This is where the “doubles need tests” rule lives or dies. Four layers:

1. Consumer-side contract tests run in the pipeline on every commit using doubles. They pin the request the consumer sends and the response shape the consumer depends on. Contract artifacts are published to a broker. Fast, deterministic, blocks the build.
2. Adapter integration tests exercise the outbound HTTP client against the real dependency in a controlled state - typically a testcontainer running an in-house service the team owns. They verify the adapter code correctly speaks the protocol: serialization, deserialization, header handling, timeout behavior, error mapping. The test asserts the adapter’s correctness, not the dependency’s behavior: if the test asks for a user, it validates that the response parses into a valid User, not which user was returned. For third-party dependencies the team can’t run in a controlled state, run these tests out-of-band on a schedule. WireMock loaded with provider-supplied fixtures is a useful complement but functions more like a contract test against recorded shapes than an integration test against the live protocol.
3. Provider-side contract verification runs in the provider’s pipeline. The provider executes every consumer’s published contract against the real provider implementation. Breaking changes are caught at the source before the provider deploys.
4. Post-deploy integration check runs periodically against the real downstream in a non-production environment. Same fixtures used in contract tests. Catches drift in fields the contract didn’t pin, version skew, environment differences. Failures trigger review, not a build break. See Out-of-Pipeline Verification.

For third-party APIs you do not control, there is no provider verification step. The post-deploy check against the live (or sandbox) API is the only mechanism keeping doubles honest. Run it more often than for in-house dependencies. Daily at minimum.

The anti-pattern to avoid: stubbing the third-party SDK directly. Always wrap third-party clients in a thin adapter the team owns, then double the adapter. This is called out explicitly as Mocking what you don’t own and is the single most common source of “but it worked in tests” incidents.

Pipeline placement

Same as the API provider pattern, plus:

• Consumer-side contract tests: pre-commit and CI Stage 1.
• Adapter integration tests for the outbound HTTP client against an in-house dependency the team controls (a testcontainer running the team’s own service in a known state): CI Stage 1 or Stage 2.
• Adapter integration tests against a third-party API or a service owned by another team: out-of-band on a schedule, never in-band. The risk of a flaky external service blocking deploys outweighs any in-band coverage benefit, and adapter tests with WireMock fixtures already cover the team’s adapter code.
• Resilience component tests with fault injection: CI Stage 1.
• Post-deploy integration checks against real downstreams: out of pipeline, on a schedule.

Example: fault injection at the client double

A negative-path test for downstream timeout. The payment client double simulates a slow response, the test asserts the deadline enforces and the upstream caller gets the documented error envelope:

• Java
• C#
• JavaScript

@SpringBootTest
@AutoConfigureMockMvc
class PaymentTimeoutTest {

  @Autowired MockMvc mvc;
  @Autowired InMemoryOrderRepo orderRepo;
  @MockBean PaymentsGateway payments;

  @Test
  void returns_504_when_payment_service_exceeds_deadline() throws Exception {
    when(payments.charge(any())).thenAnswer(inv -> {
      Thread.sleep(50);
      throw new UpstreamTimeoutException("payments");
    });

    var body = """
      { "items": [{"sku": "A1", "qty": 1}], "paymentToken": "pm_ok" }
      """;

    mvc.perform(post("/orders")
        .header("Authorization", "Bearer tok_valid")
        .contentType(APPLICATION_JSON)
        .content(body))
      .andExpect(status().isGatewayTimeout())
      .andExpect(jsonPath("$.error.code").value("UPSTREAM_TIMEOUT"));

    assertThat(orderRepo.all()).isEmpty();
  }
}

public class PaymentTimeoutTests : IClassFixture<WebApplicationFactory<Program>>
{
    private readonly HttpClient client;
    private readonly InMemoryOrderRepo orderRepo = new();
    private readonly Mock<IPaymentsGateway> payments = new();

    public PaymentTimeoutTests(WebApplicationFactory<Program> factory)
    {
        payments.Setup(p => p.ChargeAsync(It.IsAny<ChargeRequest>()))
            .Returns(async () =>
            {
                await Task.Delay(50);
                throw new UpstreamTimeoutException("payments");
            });

        client = factory.WithWebHostBuilder(b => b.ConfigureServices(s =>
        {
            s.AddSingleton<IOrderRepo>(orderRepo);
            s.AddSingleton(payments.Object);
        })).CreateClient();
    }

    [Fact]
    public async Task Returns_504_when_payment_service_exceeds_deadline()
    {
        client.DefaultRequestHeaders.Authorization = new("Bearer", "tok_valid");
        var body = new { items = new[] { new { sku = "A1", qty = 1 } }, paymentToken = "pm_ok" };

        var response = await client.PostAsJsonAsync("/orders", body);

        response.StatusCode.Should().Be(HttpStatusCode.GatewayTimeout);
        var error = await response.Content.ReadFromJsonAsync<ErrorEnvelope>();
        error!.Error.Code.Should().Be("UPSTREAM_TIMEOUT");
        orderRepo.All().Should().BeEmpty();
    }
}

test("returns 504 when payment service exceeds deadline", async () => {
  const slowPayments = {
    charge: () => new Promise((_, reject) => {
      setTimeout(() => reject(new TimeoutError("payments")), 50);
    })
  };
  const orderRepo = new InMemoryOrderRepo();
  const app = buildApp({ orderRepo, payments: slowPayments, deadlineMs: 30 });

  const res = await request(app)
    .post("/orders")
    .set("Authorization", "Bearer tok_valid")
    .send({ items: [{ sku: "A1", qty: 1 }], paymentToken: "pm_ok" });

  expect(res.status).toBe(504);
  expect(res.body.error.code).toBe("UPSTREAM_TIMEOUT");
  expect(orderRepo.all()).toHaveLength(0);
});

The test verifies three things at once: the documented status code, the structured error body the API contract promises, and that no partial state was committed.

3.7.3 - Scheduled Job

A job that runs on a cron, queue, or external scheduler. Reads from data sources, writes reports or updates state. Often has no inbound API surface. The entrypoint is the scheduler.

This pattern has two test design challenges that the API provider and API consumer patterns don’t have: time and data volume.

What needs covered

Process startup matters more here than for an API service, because scheduled jobs typically have non-trivial startup behavior (config loading, secret resolution, lock acquisition) that a component test with the SUT in-memory can bypass. The right shape is many component tests for behavior, plus one or two tests that invoke the actual deployed binary the scheduler will invoke.

Positive test cases

Common cases to consider, not an exhaustive list. Drop items that don’t apply and add ones the pattern doesn’t mention but your component needs.

• End-to-end run: with representative input, produces the expected output (report file, database update, message published).
• Idempotency: running the job twice for the same logical period produces the same result, not duplicates.
• Checkpointing: a job that processes a stream resumes from the last checkpoint, not from scratch.
• Time windows: “yesterday’s data” computes correctly for various reference times, especially around DST, month boundaries, and year boundaries.
• Empty input: zero records produces a valid empty report, not an error.
• Output format: the report or message conforms to the documented schema.

Negative test cases

Common cases to consider, not an exhaustive list. Drop items that don’t apply and add ones the pattern doesn’t mention but your component needs.

• Source unavailable: DB down, source API returning 5xx. Verify the job fails cleanly with a documented exit code/status, doesn’t write partial output, and is safely re-runnable.
• Sink unavailable: destination DB or message broker rejects writes. Verify no source state changes (e.g., “marked as processed”) happen if the sink fails.
• Partial-write failure: half the batch writes successfully, then the connection drops. Verify the next run reprocesses the failed half without duplicating the successful half. This is where idempotency keys, transactional outboxes, or compensating reads earn their keep.
• Slow job: job exceeds its expected runtime. Verify it surfaces as alertable, doesn’t silently overlap with the next scheduled run, and that the lock prevents concurrent execution.
• Malformed source data: null where non-null was expected, wrong type, encoding issues. Verify the bad record is logged with enough context to investigate, and the job decides per its policy: skip, dead-letter, or fail the whole run. The choice is design; the test pins it.
• Time-zone bugs: the job runs at 02:30 UTC for a “daily” report. What does it do on the day clocks shift? Test it. Use the injected clock so the test deterministically simulates the boundary.
• Concurrent run: the previous run hadn’t finished when the next was triggered. Verify the lock prevents overlap or, if overlap is acceptable, that the work is partitioned correctly.
• Crash mid-run: kill -9 in the middle of processing. Verify on restart the job resumes from a consistent state.
• Schema drift on source: a new field appears or a field changes type. Verify per the contract policy.

Test double validation

Three classes of doubles need validation, each through a different mechanism:

1. The injected clock. Every in-band test that depends on “now” uses an injected clock. Validate it with one out-of-band check that runs against the real system clock, exercises a known time-window calculation, and confirms the production wiring of the clock dependency is correct. This catches the “tests use UTC, prod uses container local time” class of bug.
2. Source and sink gateways. Same model as the API consumer pattern. Adapter integration tests in the pipeline exercise each gateway against a real source/sink container or WireMock. Contract tests pin the shape. Post-deploy integration checks confirm the doubles still match the real systems on a schedule.
3. The scheduler trigger. The doubled trigger in component tests must match what the real scheduler invokes. Verify with a post-deploy integration check that runs the real scheduler against a deployed instance in a non-prod environment and confirms the entrypoint is found, the cron expression fires at the expected times, environment variables and secrets resolve, and the concurrency policy holds. This is the test that catches “passed in CI, didn’t run in prod because the cron expression had a typo.”

Pipeline placement

• Unit and component tests: CI Stage 1.
• Adapter integration tests for the source and sink adapters: CI Stage 1 or Stage 2.
• Contract tests for each source and sink: CI Stage 1.
• Component tests of the deployed binary (small set): CI Stage 1 or Stage 2.
• Real-clock and real-scheduler integration check: out of pipeline, scheduled, against a non-prod environment.
• Post-deploy: a synthetic invocation of the job in production that verifies it ran, processed records, and met its SLO.

Example: time-window logic with an injected clock

A test that pins the daily-report window calculation around a DST boundary. The clock is injected so the test deterministically simulates the moment of interest. source and sink are field-level fakes set up in the test class with seeded data for 2026-03-08 and 2026-03-09.

• Java
• C#
• JavaScript

@Test
void daily_report_run_after_DST_spring_forward_uses_correct_window() {
  Clock fixedClock = Clock.fixed(
      Instant.parse("2026-03-09T07:30:00Z"),
      ZoneOffset.UTC);
  ReportJob job = new ReportJob(fixedClock, source, sink);

  job.run();

  Report emitted = sink.lastReport();
  assertThat(emitted.windowStart())
      .isEqualTo(Instant.parse("2026-03-08T05:00:00Z"));
  assertThat(emitted.windowEnd())
      .isEqualTo(Instant.parse("2026-03-09T05:00:00Z"));
  assertThat(emitted.recordsProcessed())
      .isEqualTo(source.recordsForDay("2026-03-08"));
}