Failure alerts people don't mute: at most one email per app every 12 hours

Logs shows everything an app does, but only to someone who is looking at it. Failure emails reach the people who aren't looking. When an app's production fails, the team hears about it within a minute. After that, it hears at most once every 12 hours.

What counts as a failure

A failure is a production event with a 5xx status, or a 4xx one that ended failed, timed out, canceled or partial. In practice:

• an action that threw, or didn't answer in time
• a deploy or build that failed
• a configuration release Shopify refused
• a storefront or extension route that answered 5xx
• a model call or an email that was refused

A background job counts once, when it gives up, and not on every attempt. A job that fails twice and then succeeds is never reported.

Some things are left out on purpose. Development's failures belong to whoever is developing, so they're never sent. Record writes a caller got wrong (rejected) and 4xx answers your app chose to give are the app working as written. Including them would bury the real failures.

Who gets the email

The team's owners and admins, and the members whose access reaches the app. Viewers don't get it. The email's footer says why each person received it.

The email comes from Inference Bureau, so there's nothing to set up. It doesn't go through an app's Resend integration, which is for the email your app sends to its own users.

One email, then a 12-hour hold

An alert that fires on every failure gets muted by the end of its first bad afternoon, and after that it tells nobody anything. So the cadence is fixed:

1. The first failure after a quiet spell is sent within a minute.
2. For the next 12 hours, further failures are held. Nothing is sent.
3. When the 12 hours are up, one email carries everything that failed since the last one.
4. If nothing else failed in those 12 hours, nothing is sent, and the next failure goes out at once.

So a team gets at most two emails a day per app, however badly production is going. Every failure still appears in exactly one email. The hold only delays the news, and nothing gets dropped.

We picked 12 hours because two emails a day is a pace people still read. The first email tells you something is wrong while you can still act. The second tells you whether it stopped.

What the email says, and where it links

The subject names the app and the count, such as "portal: 14 failures in production". The body gives the time window, then one line for each kind of failure, most frequent first:

• what kind it was (Action, Job, Deploy, Storefront request, Model call and so on) and its name
• how often it happened and when it last did
• its error message
• a link to its newest event in Logs, with what it was given and what it said

The first eight kinds are listed and the rest are counted ("and 6 more in Logs"). The Open Logs button opens production's Logs filtered to status >= 400 over the window. That's the last 24 hours, or 7 days for a longer window. Logs has the full picture, so the email doesn't try to repeat it.

Turning them off

The emails are on for every app. To turn them off for one, set alerts in its settings.ib.ts:

import type { AppSettings } from "@inferencebureau/apps-sdk";

export const settings: AppSettings = {
  type: "ib/settings/v1",
  platform: "shopify",
  alerts: { failures: false },
};

Production's settings are the ones from its last deploy, so the change takes effect once you deploy it.