Alerting
Alerting (Pro) notifies your team when run behavior crosses defined conditions.
Alerting components
Section titled “Alerting components”The alerting system has three parts:
destinations: where notifications are sentrules: what conditions trigger alertsevents: historical delivery and trigger log
Supported destinations
Section titled “Supported destinations”Three delivery channels are available:
- Webhook: generic HTTP callback
- Email: one or more addresses
- Slack: posts to a Slack channel via incoming webhook
Trigger types
Section titled “Trigger types”Seven trigger types are available:
| Trigger | Description |
|---|---|
run_failed | Consecutive run failures exceed threshold |
latency_regression | Latency percentile exceeds threshold |
suite_disabled | A suite is auto-disabled |
endpoint_disabled | An endpoint is auto-disabled |
low_faucet | Faucet wallet balance is low |
api_key_expiry | API key is nearing expiration |
http_status_spike | HTTP status code spike detected (e.g. 429, 5xx) |
Rule scope
Section titled “Rule scope”Rules can be scoped to:
- tenant-wide
- specific chain
- specific suite
This lets you combine broad safety nets with high-signal critical-suite rules.
Rules support cooldown windows to prevent alert floods.
Recovery notifications
Section titled “Recovery notifications”Rules can send recovery signals when condition clears.
Use recovery notifications to close incident loops automatically.
Operational guidance
Section titled “Operational guidance”- page only on sustained failures, not single transient errors
- keep cooldown longer than expected retry turbulence
- route low-priority drift (
warn) to async channels - review alert events weekly and tune rules
Example destination payload strategy
Section titled “Example destination payload strategy”- Slack: concise summary, run link, suite name, chain name
- Webhook: include structured fields for downstream routing
- Email: include human-readable summary for on-call handoff
Example escalation design
Section titled “Example escalation design”warnverdict drift -> Slack channel only- repeated suite failure in 10m -> urgent webhook or email
- recovery signal -> close incident thread automatically
Alert noise controls
Section titled “Alert noise controls”- deduplicate on suite and condition key
- require a minimum count in a time window
- disable low-value rules during maintenance windows