Why we rebuilt Sola's alert routing engine from scratch.

The problem with our first alert router

When we launched Sola's first version, alert routing was simple: if a check fails, send a notification. It worked fine for individual developers monitoring a handful of endpoints. But as our customers grew into larger engineering teams with complex on-call rotations, the cracks started to show.

Teams were getting paged for the wrong incidents. Escalation paths were hardcoded and brittle. There was no way to route different alert types to different channels based on severity or service ownership.

So we rebuilt it from scratch.

What the new engine supports

The new alert routing engine is built around three concepts: rules, channels, and escalation policies. Here's an example routing rule configuration:

{
  "rule": "payment-critical",
  "conditions": {
    "endpoint_group": "payments",
    "regions_failing": { "gte": 3 },
    "time_of_day": "any"
  },
  "channels": ["pagerduty-payments", "slack-incidents"],
  "escalation_policy": "payments-oncall"
}

{
  "rule": "payment-critical",
  "conditions": {
    "endpoint_group": "payments",
    "regions_failing": { "gte": 3 },
    "time_of_day": "any"
  },
  "channels": ["pagerduty-payments", "slack-incidents"],
  "escalation_policy": "payments-oncall"
}

{
  "rule": "payment-critical",
  "conditions": {
    "endpoint_group": "payments",
    "regions_failing": { "gte": 3 },
    "time_of_day": "any"
  },
  "channels": ["pagerduty-payments", "slack-incidents"],
  "escalation_policy": "payments-oncall"
}

Rules define when an alert should be sent. You can combine conditions: endpoint group, failure region count, HTTP status code, response time threshold, or time of day. Rules are evaluated in priority order.

Channels are where alerts go — Slack, PagerDuty, email, SMS, webhooks, or any combination. Each channel can have its own message template.

Escalation policies define what happens if nobody acknowledges an alert within a set time window. You can chain multiple escalation steps: first page the primary on-call, then the secondary, then the entire team.

Here's a more complete example showing a multi-step escalation policy:

{
  "policy": "payments-oncall",
  "steps": [
    { "notify": "primary-oncall", "wait_minutes": 5 },
    { "notify": "secondary-oncall", "wait_minutes": 10 },
    { "notify": "engineering-team", "wait_minutes": 15 }
  ],
  "repeat": false
}

{
  "policy": "payments-oncall",
  "steps": [
    { "notify": "primary-oncall", "wait_minutes": 5 },
    { "notify": "secondary-oncall", "wait_minutes": 10 },
    { "notify": "engineering-team", "wait_minutes": 15 }
  ],
  "repeat": false
}

{
  "policy": "payments-oncall",
  "steps": [
    { "notify": "primary-oncall", "wait_minutes": 5 },
    { "notify": "secondary-oncall", "wait_minutes": 10 },
    { "notify": "engineering-team", "wait_minutes": 15 }
  ],
  "repeat": false
}

The result

Since rolling out the new routing engine, customers have reported a 40% reduction in unnecessary pages and significantly faster incident acknowledgment times. The right alert goes to the right person, every time.