The failure mode of data monitoring is not missing an incident. It is firing so often that nobody reads it any more.

You can watch it happen in a channel history. Week one, every alert gets a reply. Week two, the replies thin out. Week three, someone adds a filter rule “just until we tune it”. By week four the monitoring is still running, still green in the vendor dashboard, still on the invoice — and functionally switched off. That is worse than no monitoring at all, because a team with no monitoring knows it is exposed. A team with muted monitoring believes it is covered.

The fix is not better alerts in the abstract. It is a number you commit to and design backwards from.

Noise is not the same as a false positive

These get conflated constantly, and the confusion makes the problem unfixable, because the two have opposite remedies.

A false positive is wrong. The tool said orders stopped loading; orders loaded fine. The claim does not survive contact with the evidence. False positives destroy trust fast and the answer is to fix the detection — better baselines, seasonality awareness, a rethink of the threshold.

Noise is true but unimportant. orders really did load 4% fewer rows than the seven-day mean. The null share on a column nobody queries really did move from 2% to 4%. Nothing about the claim is wrong. It just did not warrant interrupting a human. The answer here is not better detection; it is better triage, prioritisation and routing.

Teams that treat noise as a false-positive problem spend months tuning thresholds and end up with a tool that misses real incidents. Teams that treat false positives as a noise problem mute genuine breakage. Sort every complaint into one of the two buckets before you touch a setting.

Pick the number first

Here is a starting budget for a team of one to five people with no dedicated data-quality function:

Five to ten findings per morning, of which no more than three demand action today.

That is roughly a fifteen-minute triage slot before the day starts, and it is deliberately small. Under ten items, a human reads all of them properly. Above about twenty, people scan for red and skip the rest — so anything not marked critical is now invisible, whatever your severity scale claims.

Two corollaries worth stating explicitly:

  • The budget is per morning, not per table. Adding tables does not entitle you to more alerts. If you double what you monitor, the signal per table has to get more selective, or you have quietly doubled your way out of the budget.
  • Zero is not the target. A surface that has been silent for six weeks is not necessarily healthy; it may be broken, misconfigured, or watching the wrong columns. One or two informational findings a day is a heartbeat. If you see nothing for a month, break something on purpose in a staging table and check it gets caught.

Write the number down somewhere the team can see it. A budget nobody stated is a budget nobody can breach.

The levers that keep you under it

In rough order of how much they buy you:

Severity that reflects table priority, not just anomaly size. A 30% volume drop in orders and a 30% volume drop in marketing_utm_staging are the same statistical event and completely different business events. Tier your tables — critical, normal, low — before you tune a single threshold. Priority is the multiplier that turns a statistic into a severity. Most teams find that fewer than a fifth of their monitored tables are genuinely critical, and the honesty of that exercise cuts noise more than anything else on this list.

One digest instead of a stream. Batching is the largest lever after prioritisation, and the one vendors resist because a real-time push feels more valuable. It is not. Data quality issues are almost never actionable in the minute they are detected; last night’s load is already loaded. A stream of pushes fragments attention across the day and trains you to dismiss without reading. One email, arriving after the scan finishes, ordered by severity, read once with coffee, is both calmer and more thorough.

Mute and snooze, with an expiry. Permanent mutes are how monitoring dies quietly. Every silence should carry an end date: this check, on this table, until Friday, because we know the backfill is running. When it expires, it comes back and you make the decision again with fresh information. If you find yourself re-muting the same thing for the fourth month running, that is not a mute — that is a signal that the check is wrong for that table, or the table should not be monitored.

“Expected” feedback that teaches rather than silences. There is a meaningful difference between “stop telling me this” and “this is normal, learn it”. Black Friday volume, the monthly billing spike, the weekend dip in a B2B pipeline: these should be folded into the baseline so the tool stops flagging them and correctly flags the year the spike does not arrive. A mute would have hidden that too.

Sensitivity per table. Different tables deserve different paranoia. The finance table feeding revenue reporting should flag on a wobble; the event log that ingests whatever the mobile SDK sends should not. A single global setting forces a choice between missing things on the important tables and drowning in the noisy ones — you will pick the second, then mute your way back.

Prune what you monitor. The cheapest finding to triage is the one that never fires because you stopped watching a table nobody has queried in a year. Check your warehouse’s query logs and be ruthless. In Postgres, a rough starting point:

SELECT relname AS table_name,
       seq_scan + idx_scan AS total_reads,
       n_live_tup AS approx_rows
FROM pg_stat_user_tables
WHERE schemaname = 'analytics'
ORDER BY total_reads ASC
LIMIT 25;

Anything near the top with almost no reads is a candidate for removal. Most warehouses expose an equivalent — Snowflake’s ACCESS_HISTORY, BigQuery’s INFORMATION_SCHEMA.JOBS. A monitored table that feeds nothing is pure noise generation with a maintenance bill attached.

The weekly mute review

Ten minutes, same slot every week. Pull up everything currently silenced and everything marked expected in the last seven days, and for each one ask:

  1. Is it still true? The backfill finished a fortnight ago. Unmute it.
  2. Has it been muted more than twice? Recurring mutes are a design signal, not a triage action. Either the table’s sensitivity is wrong, its priority tier is wrong, or the pattern is genuinely normal and should be marked expected instead.
  3. Did anything expire and immediately come back? That is a real, unresolved issue that has been sitting behind a snooze. Give it a ticket or accept it properly.
  4. What did we mark expected, and was that right? This is the one that erodes silently. Marking a 40% null rate “expected” because the fix is hard is how a broken column becomes permanent.
  5. Are we inside the budget? Count the findings from the last five mornings. If the average is above the number, something on the lever list needs adjusting this week, not next quarter.

The point of the ritual is that mutes decay. Nobody revisits them unprompted, and a monitoring setup is only as trustworthy as the list of things it has been told to ignore.

How Sentry handles it

Sentry is built on the assumption that a small team can absorb one email a morning and not much more.

  • One digest, not a stream. A single email after the nightly scan: what was scanned, what is healthy, and findings ordered by severity. No pushes during the day, no separate channel to check.
  • Severity is weighted by table priority. You assign each table a tier — Critical, Normal or Low — when you pick it, and that tier shapes the severity of everything found on it.
  • Mute and snooze are time-boxed. Silence one check on one table for a chosen period. It comes back when the period ends.
  • “Mark as expected” teaches the baseline rather than hiding the check, so a recurring pattern stops firing but its absence still can.
  • A per-table sensitivity dial — paranoid to relaxed — is available on the larger plans; see pricing for which.
  • Monitoring is opt-in per table, so pruning is a one-click decision rather than a config migration.

None of that removes the need for the weekly review. It just means reviewing ten mutes rather than four hundred.

Pick a plan and get started — read-only credentials, first digest tomorrow morning.