August 2026 · Datasource Pulse

How to Detect When a Scraper’s Output Schema Changes

The answer: fingerprint the fields of a known-good run, then compare a sample of every new run against that fingerprint and alert at two severities. A field that was present in nearly every record and is now missing or null in most of them is a break; a field that still exists but is increasingly empty is a warning. You need the fingerprint because a schema change never throws an error: the scraper exits green, the JSON still parses, and the first symptom anyone sees is a downstream column full of nulls, days later.

Why a renamed field is the quietest failure in the stack

When a maintainer renames jobTitle to job_title, nothing breaks loudly. Run status: succeeded. Item count: normal. Your pipeline reads the old path, gets undefined, and writes it somewhere as an empty value, which most schemas happily accept. Uptime monitors see 200s. Run-status monitors see green. The only thing that changed is that a slice of your data went hollow, and the only way to see that is to look at the shape of the output itself.

The manual route: a fingerprint and a diff

The core loop fits in a shell script. Take the latest output, list which fields are filled at what rate, and diff that against the same list from a run you trust:

Run that against a saved known-good sample and against each new run, compare the two lists, and you have a working schema-drift detector. The logic is genuinely simple. What makes homemade versions page people at 3 a.m. is three traps that are not simple, and we hit all three building ours:

Trap one: sampling from the top. Many scrapers emit items in clusters: a places scraper can return every open business first and every closed one last, with different fields filled in each. Fingerprint the first ten items and you learn the shape of the first cluster, not the dataset. Draw the sample from spread positions instead: first, middle, last.

Trap two: alerting on sparse fields. Some fields are legitimately empty on a third of records or more. Check ten items against a field that is 70% filled on a good day and simple binomial variance will hand you a false alarm on a meaningful fraction of perfectly healthy runs, which is how a monitor teaches its owner to ignore it. Sparse fields should be checked for presence only, and emptiness thresholds belong only on fields that are nearly always filled.

Trap three: comparing across different inputs. The same scraper pointed at a different target legitimately returns a different shape. A baseline is only valid for one configuration; when the input changes, the old fingerprint must be thrown away and relearned, not diffed against.

The automated route: the same math as a service

Datasource Pulse runs this exact comparison on every check. It samples up to ten items, drawn from spread positions rather than the top of the file, and validates them against a curated field fingerprint or, for generic scrapers, a baseline learned from your own recent runs. Two severities come back under the schema_drift failure class: a required field missing in most of the sample is a break, a usually-filled field going hollow past its threshold is a warning. Alerts fire on state transitions only, so a drifted schema produces one alert and one recovery notice, not a page per run.

Two honest scope notes. First, if there is no curated fingerprint and no established baseline yet, the check skips the schema verdict rather than guessing; the baseline needs a few successful runs before it is trusted, and the other alarms cover you in the meantime. Second, this watches the shape of the data, not its truth: a field that exists and is filled with the wrong value is a correctness problem no fingerprint can catch. Shape drift is simply the earliest signal that something upstream changed, which is why it is the one worth automating.

Which route should you take?

For one scraper with a stable query, the jq script plus a saved known-good sample is an honest afternoon of work, provided you respect the three traps. The automated route earns its keep the same way it does for API keys: more scrapers means more fingerprints to maintain, more sparse-field judgment calls, and a recovery-notice problem the script does not solve. At $0.02 per check with a daily schedule, a watched scraper costs about $0.61 a month. If the scraper is a Store actor you did not build, the same check also watches failed runs, zero-yield runs, and maintainer staleness; that lane has its own guide.

Whichever route you take, trip the alarm once on purpose: rename a field in a copy of your known-good sample, run the comparison, and watch the drift verdict land where you expect it. An alert you have never seen fire is a guess, not a monitor.