Why your pipeline pages you at 3am (and how big tech stopped ours)

Eleven years of on-call across five big-tech data orgs taught me that pipeline incidents cluster into three patterns: silent upstream schema changes, retry storms that turn one failure into a cascade, and “temporary” manual backfills that become load-bearing.

This is the first post in a weekly series on production data engineering. Each week I’ll take one failure mode or design decision and show the concrete fix — with diagrams and, where it helps, interactive examples you can poke at in the browser.

The pattern to steal

The single highest-leverage practice we ran at scale: contracts at the boundary. Every producer publishes a schema; every consumer validates against it in CI, before deploy, not in production at 3am. Tools change — the discipline doesn’t.

More next week. If you have a pipeline that’s paging you today, get in touch.