Example App Testing Strategy

This guide documents how the example app E2E tests (examples/opennext-cloudflare-do/tests/*.spec.ts, examples/research-assistant-cloudflare-do/tests/*.spec.ts) are designed and what discipline contributors must follow when modifying them.

Read this before adding, modifying, or "fixing" any example test.

The principle: mock as little as possible

The example tests exercise the SDK end-to-end against a near-production stack:

• Real wrangler dev with workerd (the production Cloudflare runtime, not a test double)
• Real Cloudflare Durable Object emulation
• Real D1 SQLite storage
• Real Workers AI / Vectorize bindings (when used)
• Real next.js production build served by OpenNext
• Real Playwright browser (Chromium) driving the UI
• Real Server-Sent Event streams over HTTP
• Real network refresh / reload semantics (page.reload())

The only thing mocked is the OpenAI HTTP API, and only because real LLM responses are nondeterministic in ways that have nothing to do with what the SDK is doing. Everything else runs as it would in production.

This is intentional. Examples are the place where bugs that only manifest under the full production runtime get caught. A bug in stream resumption that only shows up when workerd serves SSE through a specific code path will not be caught by mocking workerd.

What "highly realistic mocks" means here

The OpenAI mock is not a hand-crafted set of "happy path" responses. It is a record/replay HTTP proxy:

1. Record mode forwards every request to api.openai.com verbatim and persists the actual response body to a fixture file.
2. Replay mode serves the recorded fixture for the same logical request.

Fixtures are real OpenAI Responses API SSE streams, captured from a real model (gpt-4o, gpt-4o-mini, gpt-5-mini, etc.) responding to a real prompt with real tool definitions. They include all the quirks, retry headers, partial JSON streaming, and failure modes that the real API exhibits.

This matters because the SDK's job is to handle whatever a real LLM returns. If we hand-crafted "clean" fixtures, the tests would only prove that the SDK works against fictional LLM behavior. Real LLMs sometimes emit:

• Tool calls with empty argument JSON before the full payload streams
• Multiple parallel tool calls in a single response
• Reasoning tokens before the actual response
• Error mid-stream that the SDK has to recover from
• Different status codes for retries

All of those need to be in the fixtures because all of those happen in production. Hand-crafted fixtures would lie to us.

Hard rules — no shortcuts allowed

These are not suggestions. Code review will reject changes that violate them.

1. Fixtures MUST come from real OpenAI

Every JSON file in test-utils/openai-fixtures/ must have been produced by OPENAI_MOCK_MODE=record against the real api.openai.com. Do not write fixture JSON by hand. Do not modify fixture bodies. Do not synthesize SSE streams.

If you change a test prompt or modify the agent's system prompt / tool definitions, the existing fixture is stale — re-record it. See "Re-recording" below.

2. The mock MUST stay structural, not selective

The proxy hashes requests by their semantic content (model + messages + tools), with non-deterministic fields (call_id, timestamps, epoch ms, "Today's date is X") normalized away. It does not match by prompt substring, "first user message contains X, return Y." That kind of selective matching encodes "what we wish the LLM would do" rather than "what it actually does," which is the shortcut this whole architecture exists to prevent.

If you find yourself wanting to add a "if request matches pattern, return canned response" branch to the mock, stop. That is the line between a record/replay proxy and a stub. We're not crossing it.

3. Tests MUST assert SDK invariants, not LLM behavior

The SDK's invariants are:

• Snapshot consistency: re-fetching the same session returns the same data
• Resume idempotence: page refresh during/after streaming does not duplicate or lose messages
• Stream completion: every started stream eventually closes with status: ended
• Tool result threading: tool calls and results pair correctly across refresh boundaries

These are deterministic for a given LLM response stream. They hold regardless of whether the LLM made 1 tool call or 12.

What the tests must not assert:

• "LLM made exactly N tool calls" — that's testing the model
• "LLM said the word 'time' in its response" — that's testing the model
• "LLM returned a JSON-shaped argument" — that's testing the model

If a test prompt was "Call get_current_time exactly once" and the recorded fixture shows the model called it 6 times anyway, the correct test assertion is "consistency of the count across refreshes," not "the count is 1." The fixture captures what the LLM actually did; the test verifies the SDK handled it correctly.

4. Test prompts MUST be natural language, not strict directives

Old-style strict prompts:

// ❌ Don't write tests like this. Real LLMs ignore these directives ~13%
// of the time, and the test reduces to "is the model's mood good today."
'Call get_current_time EXACTLY ONCE to tell me the time. EXACTLY 1 tool call, no more.';

Natural prompts:

// ✅ The LLM responds predictably-ish, and the recorded fixture locks
// in whatever it actually did. Subsequent runs replay that recording
// deterministically.
'What time is it? Just tell me once.';

The "Just tell me once" / "briefly" / "in one sentence" hints are fine — they steer the model toward the kind of response real users would write. Strict imperatives ("EXACTLY", "MUST", "DO NOT") are not, because they're testing the model's compliance behavior rather than the SDK.

5. NEVER add test.skip to dodge a real failure

test.skip is only acceptable when a test exercises a path that the mock infrastructure provably cannot make deterministic, and the skip is paired with:

• A clear comment explaining the specific reason (cite the symptom, not "flaky")
• A TODO that names what would need to change to re-enable it (e.g., "needs hand-crafted fixtures + timing-invariant rewrite")
• A linked issue or note in test-utils/README.md

If the test is failing because the assertion is wrong, fix the assertion. If it's failing because the SDK has a real bug, fix the SDK. Do not skip to make CI green.

6. The mock MUST cover ALL OpenAI calls a test makes

If a single test triggers 30 OpenAI requests and you only have 29 fixtures, the missing one will hit the mock's miss path and dump a diagnostic file (_miss-<hash>-<timestamp>.json). That file is ignored by git on purpose — it's a signal that recording was incomplete, not something to commit and hope nobody runs that path.

Re-record until the test runs end-to-end with zero misses in replay mode.

Test architecture diagram

┌─────────────────────────────────────────────────────────┐
│                 Playwright Browser                       │
│           (real Chromium, real refresh)                  │
└──────────────────────┬──────────────────────────────────┘
                       │ HTTP
                       ▼
┌─────────────────────────────────────────────────────────┐
│              wrangler dev → workerd                      │
│   (real Cloudflare Worker runtime, real DOs, real D1)   │
│                                                          │
│  Reads OPENAI_BASE_URL=http://localhost:3030/v1         │
│  from .dev.vars                                          │
└──────────────────────┬──────────────────────────────────┘
                       │ HTTP (POST /v1/responses)
                       ▼
┌─────────────────────────────────────────────────────────┐
│       openai-mock-server.ts (port 3030)                  │
│                                                          │
│  RECORD mode:  forwards to api.openai.com,              │
│                  saves response to fixture               │
│  REPLAY mode:  matches by semantic hash, serves         │
│                  fixture verbatim                        │
└──────────────────────┬──────────────────────────────────┘
                       │ (record mode only)
                       ▼
              api.openai.com
              (the actual LLM)

Everything below "Playwright Browser" runs identically in CI and locally, including the full SSE streaming chain. The only thing that differs is whether the mock is in record (developer machine, populating fixtures) or replay (CI, deterministic).

Re-recording

When test prompts, agent system prompts, or tool definitions change, the recorded fixtures become stale (the semantic hash will miss). Re-record:

# 1. Get a real OpenAI API key. The data-feeds-platform repo has it
#    in Infisical:
export OPENAI_API_KEY=$(cd ~/code/helix/data-feeds/data-feeds-platform && \
  infisical secrets get OPENAI_API_KEY --env=dev --plain)

# 2. Wipe stale fixtures
cd examples/opennext-cloudflare-do
rm -rf test-utils/openai-fixtures/*.json

# 3. Configure worker to point at mock + use real key for forwarding
printf 'OPENAI_API_KEY=%s\nOPENAI_BASE_URL=http://localhost:3030/v1\n' \
  "$OPENAI_API_KEY" > .dev.vars

# 4. Boot mock in record mode
OPENAI_MOCK_MODE=record \
  node --experimental-strip-types --no-warnings \
       test-utils/openai-mock-server.ts &

# 5. Boot the worker
npm run build:worker
npx wrangler dev --local &
until curl -s http://localhost:8787 >/dev/null; do sleep 1; done

# 6. Run the tests — fixtures populate as tests hit endpoints
npm run test:e2e

# 7. Verify the recording is clean. Inspect a fixture:
ls test-utils/openai-fixtures/ | head -5
cat test-utils/openai-fixtures/<hash>.json | jq '.request.body | fromjson | .input'

# 8. Run again in REPLAY mode to confirm zero misses
pkill -f "wrangler dev"; pkill -f openai-mock-server
OPENAI_MOCK_MODE=replay node --experimental-strip-types --no-warnings \
  test-utils/openai-mock-server.ts &
npx wrangler dev --local &
until curl -s http://localhost:8787 >/dev/null; do sleep 1; done
npm run test:e2e   # must pass with zero `_miss-*.json` files left behind

# 9. If clean, commit fixtures + any test changes
git add test-utils/openai-fixtures/ tests/
git commit -m "test(opennext): re-record fixtures for <reason>"

If step 8 leaves any _miss-*.json files, the recording is incomplete. Common causes:

• Test took a different code path the second time (real LLM may have returned a different response between runs, leading to different follow-up requests)
• Hash normalization missed a non-deterministic field — inspect the miss file and the closest-matching recorded fixture, find the field that differs, and either extend normalizeForHashing in openai-mock-server.ts or accept the test's nondeterminism as a test-design issue (rare; almost always there's a normalizable field at fault)

Do not "fix" misses by re-running record mode and accumulating both the original and the new fixtures — that hides the underlying non-determinism. Wipe openai-fixtures/ and re-record cleanly.

Adding a new test

1. Write the test against a natural-language prompt
2. Make the test assert SDK invariants only (consistency, no duplication, snapshot stability)
3. Run in record mode locally — fixtures populate
4. Run in replay mode locally — confirm no misses, deterministic pass
5. Commit fixtures + test together

Do not commit a test without committing its fixtures. CI runs in replay mode; an unrecorded test will hit the mock's miss path and fail.

CI

CI runs the example tests in OPENAI_MOCK_MODE=replay against checked-in fixtures. The CI script in .gitlab-ci.yml spawns openai-mock-server.ts alongside wrangler dev before running playwright.

A miss in CI means a fixture is missing or stale, and the test fails with a clear mock_fixture_missing error pointing at the hash. The fix is always: pull the branch locally, re-record, push the fixtures. There is no production-equivalent fallback to real OpenAI in CI.

Anti-patterns to call out in review

Anti-pattern	Why it's wrong	Correct alternative
expect(toolCount).toBe(1)	Tests LLM compliance, not SDK	Assert consistency: expect(snap2).toEqual(snap1)
'Call get_current_time EXACTLY ONCE'	Strict prompt that LLMs ignore	Natural prompt: 'What time is it?'
Hand-written fixture JSON	Doesn't match real LLM behavior	Re-record from real OpenAI
// match: prompt contains "time" → return canned response in mock	Selective matching = hidden stub	Use semantic hash + recorded fixture
test.skip without comment + TODO	Hides real failures	Either fix it or document specifically why it's deferred
expect(text).toContain('hello') (text from LLM)	Tests LLM output	Test SDK behavior; if you need text presence, assert the SDK didn't drop it via length > 0
Bumping per-test setTimeout to 5+ minutes	Hides slow paths	Profile the slow path; if it's legitimate, document it; if it's not, fix it
Adding retries to mask flakiness	Hides nondeterminism	Find and fix the source of nondeterminism

Current coverage status

CC57 — 100% of example E2E tests now follow the discipline. Every spec file in both examples/opennext-cloudflare-do/tests/ and examples/research-assistant-cloudflare-do/tests/ has been converted to natural prompts + SDK-invariant assertions, and both examples route OpenAI calls through the VCR-style mock at test-utils/openai-mock-server.ts. CI runs the full suite in OPENAI_MOCK_MODE=replay against committed fixtures and is independent of OPENAI_API_KEY for all checked-in tests. No testMatch restrictions remain.

opennext-cloudflare-do — 7 spec files, all converted:

• api-consistency.spec.ts — 5 tests (all enabled; 2 previously skipped mid-stream-refresh tests were rewritten to be timing- invariant and re-enabled in CC57).
• brief-flow.spec.ts — 7 tests, marker-based assertions replaced with content-preservation invariants (text length doesn't shrink across refresh; tool counts stable; both deterministic agent tools surface).
• comprehensive-refresh.spec.ts — 9 tests, "CRITICAL INSTRUCTIONS" prompts and expect(text).toContain('1, 2, 3, 4, 5') numeric- sequence checks replaced with snapshot-stability + part-order invariants.
• follow-up-message.spec.ts — 1 test, "EXACTLY ONCE" prompts replaced with natural prompts; the load-bearing assertion (no tool-input-delta console errors) preserved.
• interleaved-content.spec.ts — 4 tests, "CRITICAL INSTRUCTIONS" prompt and >=2 tools + 2-of-3-phrase-presence assertions replaced with conditional invariants ("if the LLM produced N tools, all are completed; text length doesn't shrink across refresh").
• reasoning-display.spec.ts — 2 tests, was already mostly natural; strengthened the post-refresh assertion to require text length is non-decreasing.
• stream-resume-consistency.spec.ts — 4 tests, "EXACTLY ONCE" / "EXACTLY 2" prompts and .toBe(1) tool-count assertions replaced with consistency invariants (toolCountsAfter ≤ toolCountsBefore + 1 mid-stream; equal post-completion).

research-assistant-cloudflare-do — 2 spec files, both converted:

• research-assistant.spec.ts — 10 tests + 1 previously skipped stream test re-enabled (mock makes it deterministic). Routes through the mock now; CI no longer depends on OPENAI_API_KEY.
• sub-agent-delegation.spec.ts — 8 tests, INSTRUCTIONS: / You must prompts replaced with natural prompts ("Use the summarizer to give me one short summary"). LLM-compliance asserts (subagentStarts.length > 0) replaced with conditional SDK invariants ("IF the LLM fired the sub-agent, THEN the SDK pairing / isolation / propagation contracts hold"). Pairing invariant (starts.length === ends.length) and terminal-event invariant (subagent_end < end) hold unconditionally.

Re-recording the full set

When prompts, agent definitions, or tool definitions change in either example, re-record fixtures using the per-example procedure documented above. Both examples now have their own test-utils/openai-fixtures/ directory; they are independent — re-recording one does not affect the other.

To convert another spec file:

1. Read this whole document
2. Open the spec file and audit every expect(...).toBe(N) involving tool counts, message counts, or LLM text. Convert to invariant assertions (toBeGreaterThanOrEqual, toEqual(snap1), etc.)
3. Audit every prompt for "EXACTLY", "MUST", "DO NOT", strict compliance directives. Convert to natural language with mild hints ("just tell me once").
4. Re-record fixtures for the new prompts (procedure above)
5. Verify replay passes with zero misses
6. Update playwright.config.ts's testMatch to include the file
7. Update this doc's "Current coverage status" section
8. Commit fixtures + spec changes + config change in one commit

Why this matters

These tests are the closest thing we have to "does the SDK work in production?" The whole point is that they catch bugs that only show up in the full production stack. Every shortcut taken (a hand-crafted fixture, a stubbed code path, an "EXACTLY" prompt that masks a real bug under "the LLM was non-compliant") reduces the value of the tests until they're verifying nothing.

The mock infrastructure exists specifically so we can have deterministic, fast tests that still exercise every byte of real production code except the LLM HTTP call. Don't spend that deterministic budget on shortcuts.