← All alert recipes Alert recipes

How to get alerted when an app or integration stops working

The worst failures are the quiet ones, where everything looks fine and nothing has synced since Tuesday.

Alert on the absence of expected activity rather than on error messages, because integrations usually fail silently. A sync that stops returns no error to anybody watching; the data simply stops changing, and the first visible symptom is a customer reporting something your systems believe is correct.

Why this one is worth an interruption

This is the same structural problem as the missing shift handover, in a technical costume. You cannot alert on an event that did not happen, so you have to alert on a clock.

The delay before discovery is what makes it expensive. A tracking sync that stopped on Tuesday and is noticed on Friday means three days of customers being told wrong information by a system everybody trusted.

The threshold

Time since last successful activity, per integration. There is no error to wait for.

ConditionSet it toWhy that number
Triggerno successful sync within its normal interval plus a marginAbsence is the signal
Per integrationeach has its own normal frequencyOne threshold across all of them misfires
Secondarysuccess rate falling below normalPartial failure is more common than total
Baselineestablish normal frequency firstOtherwise the threshold is a guess
Quiet hoursnoneOvernight failures are the longest-lived

Who should receive it

Whoever can actually reconnect it, which is often an agency or contractor rather than anyone in the store.

  • Whoever administers apps and integrations.
  • Your developer or agency, if that is who maintains it, with an agreed response expectation.
  • Whoever depends on the data, so they know to stop trusting it until it is fixed.

What to do when it fires

  • Check whether the connection needs reauthorising. Expired tokens are the most common cause and the easiest fix.
  • Check the provider's status page before investigating your own configuration.
  • Tell whoever relies on that data that it is stale, immediately. That is the part most often skipped.
  • When it is restored, check what happened during the gap rather than assuming it caught up.

When not to set this alert

Do not set this on an integration whose data nobody acts on. If a stale sync would not change any decision, the alert is noise.

Do not set it before establishing each integration's normal frequency. A shared threshold across integrations with different cadences produces constant false alarms.

Common questions

Why do integrations fail silently?

Because a stopped sync produces no error for anybody to see. Expired authorisation tokens, changed permissions and provider-side changes all present as data simply not arriving.

How do I alert on something that produces no error?

Alert on time since the last successful activity rather than on failures. Set the threshold per integration, because each has its own normal frequency.

What is the most common cause?

Expired or revoked authorisation. It is also the easiest to fix, which is why reauthorising is worth checking before investigating anything else.

Related