Testing and AI development quality

Unit tests can quickly check boundary values in a tax-calculation function, but incorrectly serialized monetary units in JSON or a migration missing a column emerge at integration boundaries. Browser E2E checks the full path from the user clicking a button to seeing the final result, but narrowing the cause of failure is costly. Instead of duplicating the same defect checks across all layers, distinguish the least costly detection layer from the critical journeys that must be protected.

When designing testing layers, examine where recent incidents escaped detection rather than asking what percentage should be unit tests. If types were correct but authorization was missing, contract and abuse tests are needed. If the API was correct but submission through the keyboard failed, browser tests are needed. Update the defect classification regularly so the suite follows actual risk.

An expiration function that reads the current time directly returns different results depending on when the test runs. Passing the clock in as a dependency lets you repeat just before expiration, the exact boundary, and after expiration with the same values. Random IDs, environment variables, and global caches must be controlled the same way to separate code failures from environmental ones.

Mocks are useful for blocking external payment calls, but checking only how many times an internal helper was called just duplicates the implementation. First verify user-observable contracts such as inputs, returned business results, and stored state, and supplement the adapter itself with contract tests close to the real protocol. If a test keeps passing after a mutation is introduced, an important assertion is missing regardless of the coverage number.

If a consumer expects `status: "paid"` but the provider changes the enum to `completed`, both JSON schemas can be individually valid yet fail when used together. Contract tests must compare actual consumer expectations and provider responses by version. Error statuses, retry headers, pagination, and authorization are as much part of the contract as normal responses.

Database changes in particular involve a time dimension. You may need to add a nullable column first, confirm that old and new code can both read and write, and enforce not-null only after the backfill finishes. Migration tests must cover not just an empty database but existing data of representative size, reruns after interruption, and rollback conditions before they can justify a production deployment.

A Playwright test that locates a `button` by CSS class can break during a design refactor while missing the disappearance of a user-recognizable label. Locate it by accessible role and name, for example `getByRole("button", { name: "Submit order" })`, then check that the order number and status appear after clicking. This brings the test closer to the user contract.

Browser tests should wait for business signals. A 3-second sleep wastes time in fast environments and still fails in slow ones, whereas an assertion linked to a success message or network response finishes as soon as the condition is met. Create independent accounts and orders for each test and clean them up afterward so earlier state does not contaminate parallel runs or retries.

When AI fixes a production error, first turn the failure the user observed into an automated test and confirm that the test fails on the existing revision. If an agent changes both the code and the test expectations at once, it can turn a wrong result into the new correct answer, so compare the diff against the issue's acceptance criterion or an independent oracle.

A release gate must link not only successful command names but also the tested source revision, dependency lock, build-artifact digest, and result files. Hiding intermittent failures through retries or arbitrarily excluding affected tests makes a green status meaningless. After deployment, use the same representative journeys and operational metrics to check for regressions, and automatically stop or roll back if thresholds are exceeded.

Assume a teaching shipping policy grants free shipping when the discounted item total is at least KRW 30,000. An original total of KRW 32,000 with a KRW 3,000 discount must be compared as KRW 29,000. An implementation that uses the original total and a copied test can pass together. Keep the shipping amount as a separate policy value to avoid confusing the free-shipping condition with fee changes.

A reviewer marks the after-discount condition in the table and independently specifies that shipping is charged. The developer links the table identifier to the test name. If the implementer also changes the policy, review that requirement change separately from the behavioral fix. Record the table approval date to distinguish later policy-driven expectation changes from regressions.

A snapshot of the whole screen can fail when date wording changes. If the purpose is checking shipping policy, assert the amount and reason directly to isolate the issue. Retain a separate snapshot when wording or layout is itself a requirement. Repeated failures caused by unrelated screen changes can lead reviewers to ignore genuine amount errors.

Finish by checking that a deliberately changed comparison fails the boundary tests. Testing KRW 30,000 and KRW 29,999 distinguishes an inclusive threshold from a strict one. Preserve the failure evidence to show that the tests monitor policy rather than merely execute code. Checking both sides of the threshold also rejects a fake implementation that always returns one value.

Assume two teaching account-settings tests change the same user’s display name. Each waits for saving, but in parallel the other test overwrites the final value. Increasing button wait time does not change shared data ownership. A shared account has no conflict in a solo run, so local success does not prove parallel safety.

Comparing request bodies and post-response reads shows two names written to the same user identifier. Switch to separate accounts while preserving parallelism and response delay. Correct save feedback and the correct name after refresh provide evidence that the race has been removed. Recording the owner account for each request also detects reading another test’s saved result as success.

If an external service actually went down, account isolation will not fix the failure. Test internal UI behavior using a controlled error response and check external connectivity under a separate contract. This prevents an outage and incorrect application error handling from sharing an indistinguishable failure. The external double’s error code and body must match the contract the app handles to remain diagnostically useful.

Finally rerun the original parallel conditions and compare the first failure with the corrected execution. Record both success without retries and the disappearance of data conflicts. More repeated runs are not a substitute for evidence that the cause was removed. Verify cleanup of isolated data so leftover state cannot become the cause again in the next run.

In a teaching book search, hold the response for the first query, sea, and return the later query, mountain, first. Release the sea response after the mountain list appears. If the input and results still refer to mountain, the stale success was ignored. Use different headings and items for the two results to detect false passes where only the input changes.

A test that controls only successful responses is incomplete. Failing the old sea request could still remove the current mountain result. Run stale-success and stale-failure cases separately to verify that the latest selection survives either status. If error feedback is not bound to the latest request, an incorrect warning may remain even when the list survives.

Include accessibility observations in the same flow. The task is not complete if the result announcement reads an old count or loading never ends, even when the visible list is correct. Check current results, completed loading, and assistive-technology announcements together. The goal is to verify that visual and assistive-technology states describe the same request.

Record the order in which requests were held and released and the expected UI state at each point. Connect controlled responses to UI conditions instead of using long arbitrary waits. Preserve the same response order after fixing the defect to verify regression protection. Releasing a held response after navigation additionally checks the view’s lifetime boundary.