Almost every homegrown data check starts the same way. Something breaks, someone investigates, and the fix is a rule:

-- alerts.sql, added after the March incident
SELECT COUNT(*) FROM orders WHERE created_at::date = CURRENT_DATE - 1
HAVING COUNT(*) < 1000;

It is a reasonable response to a real incident. It also has a shelf life of about six weeks, and nobody puts that in the commit message.

How a fixed threshold rots

The number 1000 was correct on the day it was written, because someone looked at a week of data and picked a floor comfortably below it. Then reality moves underneath it:

  • It is wrong on weekends. Most business tables have a weekly rhythm. If Saturday is a third of Tuesday, a threshold set from a Tuesday fires every weekend, and a threshold set from a Saturday never fires at all.
  • It is wrong after growth. Six months later the table does 40k rows a day. A floor of 1000 now means “alert me only if the pipeline is completely dead”. A 70% drop — the one that actually costs you — passes silently.
  • It is wrong after a backfill. One reload of two years of history and the day’s count is 400x normal. Nothing in a lower bound catches it, and if someone added an upper bound too, it screams for a week afterwards while the averages settle.
  • Nobody maintains it. This is the real failure. Threshold files are written under incident pressure and never revisited, because revisiting them means re-deriving the right number for every table by hand. Within a quarter, half of them are muted in someone’s inbox filter.

The pattern underneath all of these: a fixed threshold encodes a snapshot of what normal looked like on one day, and then normal keeps moving.

What a learned baseline captures instead

The alternative is to stop hardcoding the number and derive it from the table’s own history, recomputed every time you check. A useful baseline models four things:

  1. Level — the central value for this table, on this kind of day. Not one number for the table; one number per context.
  2. Weekly seasonality — the shape of the week. Monday backfills, weekend dips, the Friday spike from a batch job. Comparing today to the last 30 days indiscriminately smears all of that together; comparing today to the last several same weekdays does not.
  3. Trend — the table is growing, or shrinking, and normal drifts with it. A rolling window follows that for free. A constant cannot.
  4. Variance — how much this table normally bounces around. A table that does 10,000 rows every single day and one that does 10,000 give or take 6,000 need completely different alerting bands, and the mean alone tells you nothing about which you have.

That last point is the one most people skip. “How far from normal is abnormal” is not answerable in absolute units; it is only answerable relative to the table’s own noise.

How far from normal is abnormal

The standard move is to express today’s value as a number of standard deviations from the baseline mean — a z-score:

z = (today - mean_of_comparable_days) / stddev_of_comparable_days

A z of 1 is an ordinary day. A z of 2 happens roughly one day in twenty by chance alone, so on a table checked daily it is a weak signal at best. A z of 4 or 5 on a stable table is a genuine event. This is the same logic behind the out-of-range check on any monitoring tool: every prior order value sat within one standard deviation; today’s maximum is five out.

Three practical caveats:

  • Comparable days, not all days. The mean and stddev must come from days that resemble today. Same weekday is the cheapest good proxy; add “excluding known holidays” if your business has them.
  • Robust statistics beat clean ones. A single backfill in the trailing window inflates the standard deviation enormously and blinds the check for weeks afterwards. Using a median and a median absolute deviation, or trimming the top and bottom of the window, is more work and far more stable.
  • Relative change matters as well as absolute. On a small table, a jump from 3 rows to 9 is a huge z-score and usually nothing. Sensible checks require both a statistical deviation and a minimum practical magnitude before they say anything.

Computing a same-weekday baseline by hand

This is not complicated maths, and it is worth writing once so you can see exactly what a learned baseline is doing. In Postgres:

WITH daily AS (
  SELECT
    created_at::date            AS day,
    EXTRACT(DOW FROM created_at) AS dow,
    COUNT(*)                     AS rows_loaded
  FROM orders
  WHERE created_at >= CURRENT_DATE - INTERVAL '90 days'
  GROUP BY 1, 2
),
today AS (
  SELECT * FROM daily WHERE day = CURRENT_DATE - 1
),
baseline AS (
  SELECT
    AVG(rows_loaded)    AS mean_rows,
    STDDEV(rows_loaded) AS sd_rows,
    COUNT(*)            AS sample_days
  FROM daily
  WHERE dow = (SELECT dow FROM today)
    AND day < (SELECT day FROM today)
)
SELECT
  t.rows_loaded,
  ROUND(b.mean_rows)  AS expected_rows,
  b.sample_days,
  ROUND((t.rows_loaded - b.mean_rows) / NULLIF(b.sd_rows, 0), 2) AS z_score
FROM today t CROSS JOIN baseline b;

Ninety days gives about twelve comparable weekdays — enough for a usable mean and a shaky standard deviation. The NULLIF guard matters: a table that loads the same count every week has zero deviation, and dividing by it is how a monitoring query starts returning nulls instead of alerts.

Now maintain that per table, per column, for null rates, formats, distributions and uniqueness as well as row counts. That is the real argument for not doing it by hand: not that the statistics are hard, but that there are hundreds of instances of them and they all need refreshing nightly.

Where learned baselines fail too

A baseline is not magic, and it fails in ways a fixed threshold does not. Both of these are worth knowing before you trust one:

Slow drift becomes the new normal. If a column’s null rate climbs by one point a week, no single day is anomalous, and after three months a 40% null rate is exactly what the baseline expects. Rolling windows are, by construction, blind to change slower than the window. The mitigation is to compare against a long horizon as well as a short one — today versus 30 days and today versus the same period last quarter — and to alert on the trend itself, not only the daily deviation.

Cold start on a new table. With four days of history there is no seasonality, no trend, and a standard deviation computed from almost nothing. The honest behaviour is to say so: report what the table looks like, hold off on confident anomaly claims until there are enough comparable days, and treat the first weeks as observation rather than judgement.

Real changes look like incidents. A marketing campaign, a new customer onboarding, a deliberate migration — all large deviations, all entirely fine. A baseline cannot know your roadmap.

That last one is why feedback matters more than the statistics do. The system needs a way to be told a deviation was expected, and that signal has to fold back into what “normal” means rather than silencing one email. Mark as expected does that: it accepts the new shape as part of the baseline going forward. Mute says something different — “I know, stop telling me” — and you need both. Conflating them is how you end up with a monitoring system that is quiet because it has been beaten into silence.

How Sentry handles it

Sentry does not ask you to set thresholds, because setting thresholds is the part that does not survive contact with a real table.

  • Baselines are learned from each table’s own history, automatically. Connecting and picking tables is the whole configuration; there is no rules file.
  • Each table’s rhythms are respected — weekend dips, weekly load schedules — rather than one flat rule applied everywhere.
  • All six checks work from the same baseline logic: freshness, volume, out-of-range, null-rate, format and duplicates.
  • Every finding shows the baseline it was judged against, alongside the queries run and sample offending values, so you can check the reasoning rather than take it on faith.
  • Mark as expected folds a pattern into the baseline; mute and snooze silence a check without teaching it. Two different actions for two different situations.
  • A per-table sensitivity dial on the higher plans, for tables needing a tighter or looser band than the default — see pricing for which.

Fixed thresholds fail quietly, which is the worst way for a monitoring system to fail. A baseline that recomputes itself every night at least fails loudly enough to correct.

Pick a plan and get started — read-only credentials, first digest tomorrow morning.