The message is always some version of the same words: “revenue looks low — is this right?”

You have maybe twenty minutes before the person asking forms their own theory. The instinct is to open the dashboard’s SQL and start reading. That is the wrong first move: the query is usually fine, and reading it costs half your twenty minutes.

Work outward from cheapest to most expensive instead — cache, arrival, amount, shape, meaning, upstream. Each step is one query and about ninety seconds, and most incidents stop by step three.

Steps 1-3: is the number even the number?

1. Is the dashboard cache stale? Every BI tool caches, usually more aggressively than its settings page suggests. Force a refresh, check the “last updated” timestamp on the tile rather than the page, and run the underlying query directly against the warehouse.

A “yes” looks like: the raw query returns a different, sane number.

What to tell them: “The tile was serving a cached result from last night. The live number is X. Refreshed.”

2. Did the data arrive at all for the period? The single most common cause of a low number is that the period is not finished loading — or never started.

select max(created_at) as latest_row,
       now() - max(created_at) as lag
from analytics.orders;

A “yes” looks like: lag materially larger than the table’s normal gap — orders usually lands by 04:00 and the latest row is from 19:00 yesterday. Judge against the table’s own rhythm, not a fixed threshold: a weekly load is not broken on a Tuesday.

What to tell them: “The load hasn’t completed for today. Nothing is lost; the number will be right once it does. I’ll confirm when it lands.”

3. Did the right amount arrive? Partial loads are nastier than missing ones: the dashboard renders a plausible number instead of a zero. Compare against the same weekday over recent weeks, not yesterday — Mondays and Sundays rarely resemble each other.

select date_trunc('day', created_at)::date as day, count(*) as rows
from analytics.orders
where created_at >= current_date - interval '35 days'
  and extract(dow from created_at) = extract(dow from current_date)
group by 1 order by 1 desc;

A “yes” looks like: today at 20% or 300% of the same-weekday band. Partial loads and double loads both surface here.

What to tell them: “About a fifth of the usual volume arrived today. That’s a load problem, not a business one. Investigating the pipeline.”

Steps 4-5: is the data the right shape?

4. Are key columns suddenly null, or in a new format? An upstream API that starts sending null for amount, or a value arriving as "1,234.00" instead of 1234.00, silently drops rows out of a sum or a where clause. Volume looks fine; the total does not.

select count(*) as total,
       count(*) filter (where amount is null) as null_amount,
       count(*) filter (where status is null) as null_status
from analytics.orders
where created_at >= current_date - interval '1 day';

Run it again for a week ago and compare the ratios.

A “yes” looks like: a null share that has moved by an order of magnitude, or a text column that used to parse cleanly and now does not.

What to tell them: “Rows are arriving, but the amount is empty on 40% of them, so they fall out of the total. The count is right; the sum isn’t. This is upstream.”

5. Did a join fan out? The mirror image: the number is too high, or the count is right while the sum is inflated. One duplicated row in a dimension table multiplies every fact row joined to it.

select count(*) as rows, count(distinct order_id) as distinct_orders
from analytics.orders o
join analytics.customers c on c.customer_id = o.customer_id
where o.created_at >= current_date - interval '1 day';

A “yes” looks like: rows exceeding distinct_orders. Find the culprit with group by customer_id having count(*) > 1 on the dimension.

What to tell them: “A customer record loaded twice, duplicating every order attached to it. The real figure is X, not Y. Fixing the dimension load.”

Steps 6-7: did the meaning change?

These come last because both require reading code or talking to people.

6. Did a definition change? The data is fine and the number is different because somebody changed what the number means. The usual suspects, in rough order of frequency:

  • A filter. Someone added and status != 'test', or removed one, or narrowed a window from 30 days to 28.
  • A currency. A multi-currency source where amounts stopped being converted, or started being.
  • A timezone. A date_trunc moved from UTC to local, shifting hours of orders into the previous day. Looks like a permanent single-digit drop starting on one date.
  • A status mapping. completed became complete, or a new status appeared that the dashboard’s in (...) list does not include.

The query: plot the daily total over 60 days, find the date the level shifted, then look at what shipped that day.

select date_trunc('day', created_at)::date as day, sum(amount) as revenue
from analytics.orders
where created_at >= current_date - interval '60 days'
group by 1 order by 1;

A “yes” looks like: a clean step change on one date rather than a gradual drift. Steps mean code; drifts mean business.

What to tell them: “The definition changed on the 14th — test orders are now excluded. The old figure was overstated by about 4%. Nothing is broken, but history before that date isn’t comparable.”

7. Did the source system change? Last, because it is slowest to confirm and needs someone outside the data team: a new field, a deprecated endpoint, a changed enum, a new billing provider. Ask the owning team directly — “did anything ship against orders on Tuesday?”

What to tell them: “The payments provider changed how refunds are represented. We’re reflecting reality; the definition needs updating to match.”

What to write back

Reply once, early, in this shape.

Looking at it now. The number is [low / high / different] because [one clause].

Is it real? [Yes, the data is correct and the business changed / No, this is a data issue / Not yet known — I’ll confirm by TIME.]

Should you act on it? [Safe to use / Don’t use this figure until I confirm / Use the corrected figure: X.]

Next update: [time].

The three questions a stakeholder actually has are “is it real”, “should I act”, and “when will I know”. Answer those and you will not be asked again for an hour.

Turn the incident into a permanent check

The debugging above is fine. The failure is that it started with a human noticing. Every step here is a rule that could have run at 05:00:

  • Load did not run → a freshness rule on that table’s load column.
  • Partial load → a volume rule compared against the same weekday, not a fixed floor.
  • Column went null → a null-rate rule on the two or three columns the number depends on.
  • Format shifted → a format rule on the parsed columns.
  • Join fanned out → a uniqueness rule on the dimension’s key.

Write them the same day, while you still remember the shape of the failure. The test is simple: next time this class of failure happens, you send the message rather than receive it.

How Sentry handles it

Writing those rules by hand for every table, and maintaining them as the tables change, is a job nobody on a small data team has time for. That is Sentry’s job.

  • Five of the six checks it runs nightly are the five above — freshness, volume, null-rate, format and duplicates — plus out-of-range for values landing far outside the learned distribution.
  • No thresholds to write. Sentry profiles each table’s history to learn its normal behaviour, weekend dips and weekly load schedules included, so “same weekday, recent weeks” is the default rather than something you configure.
  • Findings arrive with evidence — the queries run, sample offending values, the baseline the day was judged against — so steps 1-5 are done before you open the email.
  • It diagnoses, it never writes. Read-only credentials you create; the decision stays with the person who understands the pipeline.

The goal is not to skip the checklist. It is to be holding the answer before anyone thinks to ask.

Pick a plan and get started — read-only credentials, first digest tomorrow morning.