B E T A | Tech & Digital Culture

By Nora Ishikawa Consider a payments team that shipped a mobile release after a four-week beta. The exit checklist had eleven items. Ten were marked green. The eleventh, “crash-free sessions ≥ 99.5%,” was marked green as well, because the dashboard showed 99.6%. The release went out to 100% of users. Continue Reading

The Day We Realized Our Integration Tests Only Passed Because They Mocked the Bug March 14, 2024. A payments team at a mid-stage fintech ships a schema migration. Every unit test passes. Every contract test passes. Every CI gate goes green at 2:47 PM. By 9:12 PM, 2,300 user records Continue Reading

The Problem With Runbooks Written by People Who Never Ran Them March 14, 2024. A payments-processing team at a mid-stage fintech gets a SEV-1 page at 2:47 AM. Primary PostgreSQL is unresponsive after a failed connection-pool expansion. The on-call engineer opens the incident response runbook for the payments service — Continue Reading

At 03:14 on a Tuesday, a payment service in a mid-sized European fintech began returning HTTP 200 with an empty body. The checkout flow interpreted this as success. Balances were not updated. For eleven minutes, customers saw “order confirmed” while no money moved. The correctness suite had passed in CI Continue Reading

QA engineers own the question no one else wants to ask: “How do we know this works?” They are the operational truth-tellers of software delivery, the people who treat a green dashboard as a claim to be verified rather than a status to be celebrated. In a field that rewards Continue Reading

You’re staring at a production incident with no stack trace, no metrics, and a log file that stopped being useful three releases ago. The system wasn’t designed to be observed. It was designed to ship. That distinction matters, because observability isn’t a feature you can bolt on after the fact. Continue Reading

At 2:14 a.m. on a Tuesday, a payment-processing service started returning HTTP 500s for roughly one in every 900 requests. The release that had gone out six hours earlier had passed every staging test. Load tests were green. Contract tests were green. The staging environment had been running the exact Continue Reading

Last October, a team I was advising found a regression test that had been passing for eighteen months while the feature it guarded was silently broken. The feature handled billing reconciliation—matching usage events against invoice line items. The test asserted that reconciliation output matched expected totals for a fixed dataset. Continue Reading

Why Staging Never Mirrors Production: The Structural Gaps No One Fixes By Nora Ishikawa At 2:47 a.m. on a Tuesday, the payment gateway for a mid-sized e-commerce platform started rejecting valid transactions. The code wasn’t broken. The assumption was. The release had sailed through staging—every integration test, every smoke test, Continue Reading

We keep building staging environments as if they were miniature productions. Then we act surprised when they lie to us. The truth is, staging never mirrors production—not because we’re bad at our jobs, but because the forces that shape a live system can’t be replicated in a sandbox. This article Continue Reading