Alert Responder

​

What is an Alert Responder?

Alert responders listen for alerts and automatically start investigations pulling in context across your systems to produce a likely root cause and recommended next actions.

• It starts investigating the moment an alert arrives.
• It uses telemetry data, code, deployments, relevant runbooks, and past incidents and postmortems
• It analyzes trends and extracts insights from all historical investigations

​

Supported Alert Sources

TierZero can respond to alerts from:

• Slack
• Opsgenie
• PagerDuty
• FireHydrant
• Rootly
• Datadog (coming soon)
• New Relic (coming soon)

​

Use Cases

​

Accelerate Incident Response

• Challenge: Engineers lose time context‑switching, gathering telemetry, and triaging severity.
• How it works: Slack and webhook alerts trigger an immediate investigation using your runbook prompt; the details page shows a concise summary, tags (category, services), and a link to the full investigation.
• Outcome: Faster triage with pre‑populated likely causes and relevant evidence before responders join.

​

Reduce Alert Fatigue

• Challenge: High‑volume channels and duplicate alerts obscure what needs action.
• How it works: Configure alert responders to focus on relevant alerts, use debounce to prevent frequent repeated investigations during noisy spikes, and use alert insights to extract significant alerts and reduce noise.
• Outcome: Higher signal‑to‑noise, fewer duplicate investigations, and clearer prioritization.

​

Scale team knowledge

• Challenge: Investigation know‑how lives in tribal knowledge and inconsistent practices.
• How it works: Codify investigation steps in “Configure Runbook”, then iterate in Prompt Studio using real alert inputs. Every run produces a summarized, searchable history.
• Outcome: Consistent investigations across rotations and faster onboarding for new responders.

​

How to Set Up Alert Responders

​

Prerequisites

• Ensure you have the appropriate integration configured (Slack, Opsgenie, PagerDuty, FireHydrant, or Rootly).
• For webhook-based sources (Opsgenie, PagerDuty, FireHydrant, Rootly), set up webhook subscriptions in Integrations settings first.

​

Create the Alert Responder

• Go to Alert Responders and click “Create Alert Responder”.
• Choose a source: Slack, Opsgenie, PagerDuty, FireHydrant, or Rootly (only active integrations are shown).

​

Configure Detection

• Slack:
  • Channel: Slack Channel ID to monitor.
  • Sender ID (Optional): Slack bot, app, or user ID.
  • Text Matches: Text to match against, for example, alert name or monitor ID
• Webhooks (Opsgenie, PagerDuty, FireHydrant, Rootly):
  • Webhook Subscription: One or more webhooks in the specific integration
  • Text Matches: Text to match against, for example, alert name or monitor ID

​

Configure a Runbook (Optional, Recommended)

• Once the alert responder is created, click “Configure Runbook” on the alert responder page.
• Provide instructions that TierZero should follow in ‘Investigation Prompt’.
• Use Prompt Studio to test and refine with real alert inputs.

​

Review results

• See a 30‑day trigger chart, plus a timeline of investigations with summaries, tags, and links to deep‑dive.

​

Best Practices

1. Configure Alert Runbooks

• Custom runbooks will instruct the agent to focus on what matters and produce consistent results.
• Begin with alerts that already have documented steps or predictable diagnostics.

2. Leverage Prompt Studio to Iterate on Runbooks

• Test prompts against real alert inputs, compare versions, and refine quickly.
• Save improved prompts back to the responder once satisfied.

​

Example Runbooks

Alert Responder Runbooks