Skip to content
Documentation
Operate
Alerting

Alerting

An alert puts a threshold on one signal; the api-server evaluates it on a schedule and fires a webhook or email when it breaches for a sustained window, and again when it recovers. Notification config never enters your cluster. A rule’s source decides where the sample comes from.

# Golden-signal alert: 5xx rate above 5% for 5 minutes.
kubrain deploy alert set myapp \
  --signal error_rate --op '>' --threshold 0.05 --for 5m \
  --notify https://hooks.slack.com/services/XXX

kubrain deploy alert list myapp             # each rule's live state + last value
kubrain deploy alert delete myapp error_rate

Three sources

metric (default) — golden signals

Thresholds a signal from get_metrics: error_rate (fraction), p95_ms, or req_rate. --op is > (default) or <.

kubrain deploy alert set myapp --component api \
  --signal p95_ms --op '>' --threshold 500 --notify oncall@example.com

pod — pod health

Watches the deployment’s pods with a live cluster check (no metrics backend needed). Fires when pods are failed, not ready (CrashLoopBackOff, ImagePullBackOff, …), or stuck pending past a 5-minute grace. The single signal pods_unhealthy defaults to > 0, so only --notify is required:

kubrain deploy alert set myapp --source pod --for 5m \
  --notify https://hooks.slack.com/services/XXX

The firing notification names the offending pods and why.

probe — uptime & TLS

A black-box HTTP(S) check the api-server makes from outside the cluster to one of your app’s public ingress hosts — the true end-to-end “is my site up.” Two signals:

  • http_up (default) — GET https://<host>/; fires when it stops answering 2xx/3xx.
  • cert_days — days until the host’s TLS certificate expires; cert_days < 14 by default warns two weeks ahead.
kubrain deploy alert set myapp --source probe --host app.example.com --for 2m \
  --notify oncall@example.com
kubrain deploy alert set myapp --source probe --host app.example.com \
  --signal cert_days --threshold 21 --notify oncall@example.com
The probe host is not arbitrary — it must be a host the deployment actually serves through its own Ingress. Kubrain checks ownership at set time and refuses to probe anything else.

How it fires

A debounced state machine: ok → pending → firing, where --for is how long a breach must persist before firing (omit to fire on the first breaching sample). A recovery moves it back to ok. Steady-state firing notifies once (on start) and once on resolve — not every tick.

Rules are idempotent per (component, signal) — setting the same signal again replaces the rule and resets its state. list_alerts shows each rule’s state (ok/pending/firing) and last sampled value.

Notifications

--notify takes a webhook URL (Slack, Discord, PagerDuty, any incoming webhook) or an email address — the channel is inferred from the shape. Webhooks POST a stable JSON payload; email needs the platform SMTP relay configured.

Gotchas

  • req_rate < N won’t fire on total silence — a no-traffic window reads as no-data, treated as “not breaching.” Fine for error_rate, p95_ms, and req_rate >.
  • source=log (log-keyword alerts) is accepted but not wired yet.

Playbook

  1. kubrain deploy metrics myapp → confirm it’s serving; note the baseline.
  2. kubrain deploy alert set myapp --signal error_rate --op '>' --threshold 0.05 --for 5m --notify …
  3. kubrain deploy alert list myapp → verify it’s stored in state ok.

See also

  • Metrics — read the numbers you’re thresholding.
MetricsBlueprints