# Access Control
Source: https://docs.tierzero.ai/administration/access-control
Role-based access control (RBAC) for TierZero organizations
## Overview
TierZero uses role-based access control (RBAC) to manage what users can access and modify within your organization. Each user is assigned one of three roles that determine their permissions across different features and capabilities.
## Roles
TierZero supports three roles, each with progressively increasing permissions:
Can chat with AI and view alert and digest results. Ideal for users who need read-only access to AI-generated insights.
Can chat with AI and manage all operational resources like knowledge, alert agents, and digests.
Full administrative control including integrations, users & roles, usage & billing, and other advanced settings.
## Permissions Matrix
The table below shows which capabilities each role has access to across different domains.
| Domain | Admin | Member | Chat Only |
| ------------------------- | :----------: | :----------: | :----------: |
| **Chat** | Read & Write | Read & Write | Read & Write |
| **Alert Agents** | Read & Write | Read & Write | Read Only |
| **Support Agents** | Read & Write | Read & Write | Read Only |
| **Digest Agents** | Read & Write | Read & Write | Read Only |
| **Prompt Studio** | Read & Write | Read & Write | ✕ |
| **Instructions** | Read & Write | Read & Write | ✕ |
| **Skills** | Read & Write | Read & Write | Read Only |
| **Integrations** | Read & Write | Read Only | ✕ |
| **Users & Roles** | Read & Write | ✕ | ✕ |
| **Usage & Billing** | Read & Write | ✕ | ✕ |
| **Organization API Keys** | Read & Write | ✕ | ✕ |
**Read** permissions allow users to view information but not modify it.
**Write** permissions allow users to create, modify, and delete resources within that domain.
## Managing User Roles
Organization admins can manage user roles through the account settings:
1. Click on your **profile picture** in the top right corner
2. Navigate to **Users** under the **Organization** section
3. Find the user you want to update
4. Click on their role dropdown to change it
5. Changes take effect immediately
# Teams
Source: https://docs.tierzero.ai/administration/teams
Organize your organization into teams with independent configurations
## What are Teams?
Teams are organizational units in TierZero that let you segment alert agents, digest agents, and configurations by team. Each team can have its own instructions, alert configurations, and onboarding workflow.
## Why Use Teams?
* **Scoped investigations**: TierZero applies team-specific context when investigating alerts for that team
* **Independent alert management**: Each team manages its own alert agents and digest agents
* **Per-team instructions**: Customize TierZero's behavior for each team's unique services and workflows
* **Onboarding tracking**: Track setup progress independently per team
## Creating a Team
1. Navigate to **Settings → Teams**
2. Click **Create Team**
3. Enter a **Name** and optional **Description**
4. Click **Create**
After creation, you'll be guided through the team onboarding process to connect integrations and configure settings.
## Managing Teams
### Editing a Team
1. Go to **Settings → Teams**
2. Click on the team you want to edit
3. Update the name or description
4. Click **Save**
### Deleting a Team
Teams can only be deleted if they have no connected alert agents or digest agents.
1. Go to **Settings → Teams**
2. Click the delete button next to the team
3. Confirm the deletion
Deleting a team also removes its instructions and onboarding task history. This action cannot be undone.
### Switching Teams
Users belong to one active team at a time. To switch teams:
1. Use the team selector in the navigation
2. Select the team you want to view
3. The dashboard and agent lists will update to show that team's data
## Team Instructions
Each team can have its own set of instructions that supplement the organization-wide instructions, giving the agent team-specific context for more relevant responses.
See [Instructions](/knowledge/instructions) for guidance on how organization-wide and team-specific instructions should be structured.
## Best Practices
**1. Align Teams with Ownership**
* Create teams that mirror your engineering organization's ownership boundaries
* Each team should own a clear set of services
**2. Start with One Team**
* Begin with a single team to learn the workflow
* Expand to additional teams once you're comfortable with the setup
**3. Assign Team-Specific Instructions Early**
* Don't wait for incidents to write team instructions
* Proactively document services, escalation paths, and common issues
# Alert Agent
Source: https://docs.tierzero.ai/features/alert-agent
Automated alert investigation that surfaces root cause with evidence
## What is an Alert Agent?
Every paging alert should matter. Most don't. TierZero Alert Agent investigates every alert by pulling in telemetry data, code, deployments, relevant runbooks, and past incidents to produce a likely root cause and recommended next actions. Noisy alerts get flagged, related alerts get grouped, and known issues get rediscovered.
## Key Capabilities
### Auto-Investigates Every Alert
When an alert fires, TierZero pulls logs, traces, metrics, recent deploys, and past incidents to build a complete picture. By the time an engineer sees it, the investigation is already done.
* **Cross-stack correlation**: Connects signals across your observability tools, code repos, and deployment pipelines automatically.
* **Known-issue matching**: Checks memory for similar past alerts and applies known fixes without human intervention.
* **Full context on escalation**: When a human is needed, they get the investigation summary, not a raw alert.
### Trend Analysis
TierZero tracks alert frequency, timing, and co-occurrence across your stack. It surfaces trends that humans miss, like an alert that fires 3x more often after Thursday deploys, or two services that always fail together.
* **Noisy alert detection**: Identifies alerts that fire frequently but never lead to action, so you can tune or suppress them.
* **Correlated failure patterns**: Discovers which alerts tend to fire together, revealing shared root causes across services.
### Severity Classification
Determines blast radius and severity based on historical patterns, affected services, and downstream impact. Prioritizes alerts by SLO impact and error budget burn rate.
### Smart Escalation
Integrates with your IDP and escalates with the full investigation context already attached. Service dependency mapping calculates blast radius across your infrastructure so the right team is paged first.
### Alert Grouping
Related alerts become one thread, not ten. Your channel stays clean while the AI handles grouping behind the scenes.
## Supported Alert Sources
TierZero can respond to alerts from:
* Slack
* OpsGenie
* PagerDuty
* FireHydrant
* Rootly
* Datadog
* New Relic
## Use Cases
### Accelerate Incident Response
* **Challenge**: Engineers lose time context-switching, gathering telemetry, and triaging severity.
* **How it works**: Slack and webhook alerts trigger an immediate investigation using your investigation runbook; the details page shows a concise summary, tags (category, services), and a link to the full investigation.
* **Outcome**: Faster triage with pre-populated likely causes and relevant evidence before responders join.
### Reduce Alert Fatigue
* **Challenge**: High-volume channels and duplicate alerts obscure what needs action.
* **How it works**: Configure alert agents to focus on relevant alerts, use debounce to prevent frequent repeated investigations during noisy spikes, and use alert insights to extract significant alerts and reduce noise.
* **Outcome**: Higher signal-to-noise, fewer duplicate investigations, and clearer prioritization.
### Scale Team Knowledge
* **Challenge**: Investigation know-how lives in tribal knowledge and inconsistent practices.
* **How it works**: Codify investigation steps in your Investigation Runbook, then iterate in [Prompt Studio](/features/prompt-studio) using real alert inputs. Every run produces a summarized, searchable history.
* **Outcome**: Consistent investigations across rotations and faster onboarding for new responders.
## How to Create an Alert Agent
### Step 1: Alert Source
Connect your alert source to automatically respond to alerts.
* Go to Alert Agents and click "Create Alert Agent"
* Choose a source: Slack, OpsGenie, PagerDuty, FireHydrant, or Rootly
* **Slack**: Select the channel to monitor
* **Webhooks**: Select one or more webhook subscriptions
### Step 2: Filtering & Notifications
Define which alert events to respond to and where to send notifications.
* **Text Matches**: Specify text patterns to match against (e.g., alert name or monitor ID)
* **Sender ID** (Slack only): Optionally filter by Slack bot, app, or user ID
* **Notifications**: Configure where to send investigation results
### Step 3: Advanced Instructions (Optional)
Customize how your agent investigates alerts. These settings are optional but can improve investigation quality.
* **Investigation Runbook**: Provide custom instructions that TierZero should follow when investigating alerts. This helps the agent focus on what matters most for your specific alerts.
* **Impact & Severity Analysis Runbook**: Define how TierZero should assess the impact and severity of alerts for better prioritization.
### Step 4: Review
Review your configuration and create the alert agent.
* Set a name for your alert agent
* Review all settings
* Click "Create Agent"
## Best Practices
**1. Configure Investigation Runbooks**
* Custom runbooks instruct the agent to focus on what matters and produce consistent results
* Begin with alerts that already have documented steps or predictable diagnostics
**2. Configure Impact & Severity Analysis Runbook**
* Keep these prompts shorter than investigation runbooks for quick triage
* Focus on identifying blast radius: which users, orgs, or services are affected
* Define severity criteria specific to your business (e.g., revenue impact, customer-facing vs internal)
* Helps responders prioritize when multiple alerts fire simultaneously
**3. Leverage Prompt Studio to Iterate on Runbooks**
* Test prompts against real alert inputs in [Prompt Studio](/features/prompt-studio), compare versions, and refine quickly
* Save improved prompts back to the agent once satisfied
**4. Use Text Matches for Precision**
* Filter alerts to specific monitors, services, or alert types
* Reduce noise by only investigating relevant alerts
## Example Runbooks
* [Investigation Runbooks](/prompt-library/investigation-runbooks)
* [Impact & Severity Runbooks](/prompt-library/impact-severity-runbooks)
# Capabilities
Source: https://docs.tierzero.ai/features/capabilities
Workflow automations that TierZero can trigger on your behalf
## What are Capabilities?
Capabilities are workflow automations that TierZero can execute during investigations or conversations. Connect your CI/CD pipelines and let TierZero trigger deployments, rollbacks, and other workflows, with configurable approval gates so you stay in control.
## Supported Types
* **GitHub Actions Workflows**: Trigger any workflow in your connected repositories
* **Buildkite Pipelines**: Trigger any pipeline in your connected Buildkite organization
## Approval Styles
| Style | Behavior |
| ------------- | --------------------------------------------------------- |
| **Automatic** | Executes immediately when TierZero determines it's needed |
| **Manual** | Requires human approval before execution |
## How to Create a Capability
### Step 1: Start Creation
Navigate to **Capabilities** in the sidebar and click "Create Capability".
### Step 2: Select Type
Choose the capability type: GitHub Actions or Buildkite.
### Step 3: Configure Target
Select the repository and workflow (GitHub Actions) or pipeline (Buildkite) that this capability will trigger.
### Step 4: Define Parameters Schema
Define a JSON schema for the inputs this capability accepts. Add descriptions for each parameter so TierZero knows when and how to use them.
For example, a rollback capability might accept parameters like `service_name`, `target_version`, and `environment`.
### Step 5: Set Approval Style
Choose between Manual and Automatic. Start with **Manual** until you've built confidence in the capability's behavior, then switch to Automatic.
### Step 6: Activate
Review the configuration and activate the capability.
## How TierZero Uses Capabilities
During investigations or chat, TierZero can suggest and execute capabilities when relevant. For example:
* **Triggering a rollback pipeline** after identifying a bad deploy as root cause
* **Running a diagnostic workflow** to gather additional data during an investigation
* **Restarting a service** after confirming a fix has been applied
For Manual capabilities, TierZero presents the action and parameters for your approval before executing. For Automatic capabilities, execution happens immediately with a notification.
## Monitoring Executions
From the Capabilities page, you can track:
* **Execution history**: Full log of every time the capability was triggered
* **Approval rate (30-day)**: Percentage of Manual executions that were approved
* **Status** of each run (success, failure, pending approval)
* **Parameters passed** for each execution
## Best Practices
**1. Start with Manual Approval**
* New capabilities should require human approval until you're confident in when and how TierZero uses them
**2. Define Clear Parameter Schemas**
* Include descriptions for every parameter, as TierZero uses these to determine when a capability is relevant and what values to pass
**3. Monitor the 30-Day Approval Rate**
* A high approval rate for Manual capabilities signals readiness to switch to Automatic
* A low rate suggests the trigger conditions or parameters need refinement
**4. Use Capabilities for Common Remediation Workflows**
* Rollbacks, service restarts, feature flag toggles, and diagnostic data collection are good candidates
# CI/CD Agent
Source: https://docs.tierzero.ai/features/cicd-agent
Automated CI failure diagnosis, flaky test detection, and pipeline health tracking
## What is the CI/CD Agent?
CI failures shouldn't break your momentum. TierZero CI/CD Agent diagnoses build failures, detects flaky tests, and tracks CI health metrics, so your team ships instead of debugging pipelines.
## Key Capabilities
### CI Failure Diagnosis
When a build fails, TierZero reads the logs, identifies the root cause, and tells you what broke and why. No more scrolling through 500 lines of build output to find the one error that matters.
* **Intelligent log parsing**: Parses build logs across CI providers to pinpoint the exact failure, filtering out noise from thousands of output lines.
* **Error attribution**: Correlates failures with recent code changes, environment diffs, and dependency updates to identify the root cause.
* **Fix suggestions**: Recommends specific fixes based on the failure pattern, from missing env vars to version conflicts.
### Flaky Test Detection
Flaky tests are never "just flaky." TierZero collects evidence across hundreds of historical failures to identify the pattern.
* **Statistical detection**: Analyzes pass/fail patterns across runs to identify flaky tests with high confidence, not just "did it fail once."
* **Quarantine**: Isolate flaky tests and page issue owner so that the build train keeps chugging.
* **Production-informed test validation**: Prioritizes tests by production failure correlation to focus CI resources on tests that catch real production issues.
### CI Health Tracking
Track the metrics that matter: PR Merge to Live time, build success rates, and pipeline throughput.
* **PR Merge to Live tracking**: Create new baselines or integrate with your existing dashboards. Measure well so you can improve.
* **Build success trends**: Tracks success rates over time so regressions don't become the norm.
### Automated Fix PRs
When CI Agent diagnoses a failure, it can propose fix PRs automatically.
* Notifies the PR owner with the diagnosis
* Opens a fix PR ready for review and merge
* Approval required before merge (human-in-the-loop)
## Use Cases
### Reduce CI Debugging Time
* **Challenge**: Engineers spend 20+ hours per week debugging pipeline failures.
* **How it works**: CI/CD Agent parses build logs, correlates with code changes, and surfaces the root cause with a fix suggestion.
* **Outcome**: 20+ hours saved per week across the team.
### Faster PR Merge to Live
* **Challenge**: Flaky tests and intermittent failures slow down the merge pipeline.
* **How it works**: Flaky tests get detected and quarantined. CI health metrics surface bottlenecks.
* **Outcome**: 27% faster PR Merge to Live on average.
### Focus CI Resources on What Matters
* **Challenge**: CI runs thousands of tests, but not all are equally valuable.
* **How it works**: Production-informed test validation prioritizes tests that correlate with real production failures.
* **Outcome**: CI resources focused on high-value tests, fewer false positives.
## How to Set Up
1. Navigate to **CI/CD** in the sidebar
2. Connect your code integrations ([GitHub](/integrations/github), [GitLab](/integrations/gitlab), or [Bitbucket](/integrations/bitbucket))
3. Connect your CI/CD platform ([Buildkite](/integrations/buildkite) or GitHub Actions)
4. Configure which pipelines and repositories to monitor
5. Set up notification channels for failure diagnoses
## Best Practices
**1. Start with Highest-Traffic Pipelines**
* Focus on the pipelines that affect the most engineers first
* Maximize impact and build confidence in the agent's diagnoses
**2. Review Flaky Test Quarantine Decisions**
* Periodically review quarantined tests to ensure they get fixed, not forgotten
* Track quarantine age and escalate long-standing flaky tests
**3. Use CI Health Metrics to Drive Improvement**
* Set team goals around PR Merge to Live time and build success rates
* Use trends to justify investment in CI infrastructure improvements
# Context Engine
Source: https://docs.tierzero.ai/features/context-engine
Inspectable AI knowledge base that captures tribal knowledge from incidents, Slack, and code
## What is the Context Engine?
Not your grandma's RAG. TierZero Context Engine is a hybrid knowledge graph that captures investigation patterns from your operational history. Tribal knowledge gets captured automatically from Slack, incidents, and code changes. Structured, searchable, and editable, it is an AI agent you can actually inspect.
## Key Capabilities
### Ingestion
Multi-source capture from incidents, Slack threads, code reviews, and post-mortems, with no forms or tagging needed.
* Every memory is a **structured record** with type, source, confidence score, tags, linked services, and full version history
* **Reasoning traces** capture investigation paths: branches taken, ruled out, root cause found
* **Service dependency mapping** across your infrastructure
* **Team language mapping**: learns your internal names, abbreviations, and slang
### Retrieval
Three retrieval strategies run in parallel with reciprocal rank fusion for best results.
* **Hybrid search**: Vector embeddings, keyword search, and summary indexes combined
* **Graph traversal**: Results expand through linked incidents, services, teams, and runbooks
* **Trajectory replay**: Past investigation paths that match the current failure pattern are recalled and adapted to the current context
### Adaptation
This is not RAG and not fine-tuning. TierZero extracts investigation patterns from your operational history and replays them in-context.
1. **Capture investigation playbooks**: Every investigation records the full reasoning path: queries run, hypotheses tested, dead ends ruled out, root cause confirmed.
2. **Track what actually helped**: Memory attribution links each outcome to the specific memories that influenced it.
3. **Incorporate engineer feedback**: Corrections, reactions, and direct edits take effect immediately.
4. **Replay and adapt patterns**: When a new incident matches a past failure pattern, the agent replays the successful investigation trajectory adapted to the current context.
### Long-Term Tribal Knowledge
* **Searchable**: Ranked by relevance, recency, and confidence
* **Editable**: Every record can be inspected, updated, or deleted
* **Survives team turnover**: Knowledge persists even as people leave
* **Version history & audit trail**: Full history of all changes
## Trust & Transparency
* **Evidence chains**: Every finding cites specific sources: Datadog logs, GitHub commits, past incidents
* **Full version history**: Complete audit trail of all organizational memory changes
* **SOC 2 Type II certified**
## Performance vs RAG
97.3% vs 71.1%
89.0% vs 40.2%
93.3% vs 58.8%
Evaluated on 12,847 real operational queries.
## How to Use
### Snippets
Navigate to **Context Engine → Snippets** to browse, search, add, and edit organizational memories.
* **Search**: Query your knowledge base with natural language
* **Filter by source**: Dashboard, Document, UI, Slack, Discord, Incident Channel, Alert Thread, Chrome Extension
* **Filter by type**: Fact, Best Practice, Query, Resource
* **Filter by modifier**: See memories you've created or edited
* **Add memories**: Manually add knowledge that TierZero should remember
* **Edit or delete**: Correct inaccurate memories immediately
### Resources
Navigate to **Context Engine → Resources** to browse integrated catalogs.
Resources are structured metadata from your connected integrations:
* **Datadog**: Dashboards, notebooks, logs/spans/events configuration
* **New Relic**: Dashboards, applications, logs/spans configuration
* **Grafana Cloud**: Dashboards
* **GCP Cloud Monitoring**: Dashboards
* **Notion**: Ingested pages
* **Confluence**: Ingested documents
## Best Practices
**1. Review and Curate Memories Regularly**
* Periodically browse recent memories to ensure accuracy
* Correct or delete any inaccurate memories immediately, as they affect future investigations
**2. Encourage Team Feedback**
* Thumbs up/down reactions on TierZero responses improve memory quality
* Direct corrections take effect immediately
**3. Import Key Dashboards and Documents**
* Ingest your most important Datadog dashboards, Notion pages, and Confluence documents
* These give TierZero richer context for investigations
**4. Use the Edit Capability**
* When TierZero gets something wrong, edit the memory directly rather than hoping it learns
* Every edit has an immediate effect and is tracked in version history
# Copilot
Source: https://docs.tierzero.ai/features/copilot
AI debugging partner that queries your production systems in real time
## What is Copilot?
Your AI debugging partner. Ask TierZero questions about your infrastructure and get answers grounded in your actual production systems, not just documentation. Available via the web UI and Slack DMs.
## Key Capabilities
* **Query logs, metrics, traces, and spans** across all connected observability platforms (Datadog, New Relic, CloudWatch, Grafana Cloud, etc.)
* **Read and search code** across connected repositories
* **Access documentation** from Confluence and Notion
* **Inspect CI/CD pipelines** and workflow executions
* **Reference past incidents** and organizational knowledge from the [Context Engine](/features/context-engine)
## How to Use
### Web UI
Navigate to **Chat** in the sidebar. Type your question and TierZero investigates across your connected systems. Conversations are saved and accessible from the Chat page.
### Slack DM
Send a direct message to the TierZero bot in Slack. Conversations are threaded, so reply in the thread for follow-up questions.
### Feedback
Use thumbs up/down reactions on TierZero's responses to help improve answer quality. Feedback is incorporated into the [Context Engine](/features/context-engine) immediately.
### Viewing Past Conversations
All conversations are saved. Access them from the **Chat** page in the web UI to review past investigations or continue where you left off.
## Example Questions
**Debugging**
* "What errors are occurring in the payment service in the last hour?"
* "Show me the logs for the checkout-api with status:error from the past 30 minutes"
* "Why is the auth service returning 503s?"
**Deployment & Code**
* "What was deployed to production yesterday?"
* "Show me the recent commits to the api-gateway repo"
* "What changed in the last deploy of the billing service?"
**Performance**
* "Why is latency increasing on the API gateway?"
* "What's the p99 latency for the search service this week vs last week?"
* "Are there any SLOs at risk of breaching?"
**Documentation & Knowledge**
* "What does the runbook say about database failover?"
* "How do we roll back a deployment?"
* "What's the on-call rotation for the platform team?"
**Observability**
* "Show me the Datadog dashboard for service X"
* "What monitors are currently alerting?"
* "What are the top error messages in New Relic for the last 24 hours?"
## Best Practices
**1. Be Specific About Time Ranges and Services**
* "Errors in payment-service in the last hour" is better than "what errors are happening?"
* Specificity helps TierZero query the right data sources efficiently
**2. Reference Specific Metrics or Logs When Possible**
* If you know the metric name or log attribute, include it
* This reduces ambiguity and speeds up the investigation
**3. Use Follow-Up Questions**
* Ask follow-up questions to drill deeper into findings
* TierZero maintains conversation context within a thread
# Digest Agent
Source: https://docs.tierzero.ai/features/digest-agent
Proactive discovery and operational reports that surface hidden issues, trends, and cost insights
## What is a Digest Agent?
Also known as **Proactive Discovery**, TierZero Digest Agent actively scans for reliability risks, performance degradation, and creeping observability costs that no alert would catch. It generates comprehensive operational reports and delivers them on your schedule.
## Key Capabilities
### Continuous Monitoring
Surfaces slow degradation patterns that slip past threshold-based alerts and go unnoticed until something breaks.
* **Slow degradation detection**: Catch latency creep and memory leaks before they trigger alerts.
* **Cross-service correlation**: Individual metrics look fine. Together, they tell a different story.
* **Historical trend analysis**: Compare against baselines from weeks ago, not just hours.
* **SLO burn rate monitoring**: Catch error budget depletion before it breaches your SLO thresholds.
### Anomaly Detection
Detects unusual spend spikes, latency creep, and rising error rates before they compound into outages. Each anomaly comes with context and a suggested next step.
* **Cost anomalies**: Catch unexpected spend increases before they hit your cloud bill.
* **Performance regression**: Surface latency trends and throughput drops, along with the commit that caused them.
* **Error rate analysis**: Track error patterns and correlate with deployments and infrastructure changes.
### Pre-Deploy Risk Detection
Correlates recent deploys with performance regression. Identifies which commit introduced latency, error rate spikes, or resource anomalies before they become incidents.
* **Deploy-correlated regression detection**: Automatically links performance degradation to specific deployments and commits.
* **Pre-merge risk scoring**: Surface high-risk changes based on historical deployment failure patterns.
### Cost Intelligence
Proactive Discovery scans for cost anomalies, billing spikes, and over-provisioned resources across your cloud infrastructure.
* **Billing anomaly alerts**: Detect unexpected cost spikes across cloud providers before they hit your monthly bill.
* **Resource utilization analysis**: Identify over-provisioned and under-utilized resources across your infrastructure.
* **Auto-scaling policy drift**: Detect when scaling policies diverge from actual usage patterns.
* **Capacity trend monitoring**: Surface capacity trends before they become incidents.
### Scheduled Delivery
Runs on your schedule via cron and posts results directly to Slack, keeping the whole team informed without manual effort.
## Use Cases
### Detect Emerging Risks Earlier
* **Challenge:** Slow drift and multi-week trends (e.g., rising 95th percentile latency) are easy to miss.
* **How it works:** Digest Agents compare recent periods against baselines and flag material degradation.
* **Outcome:** Early warnings with clear evidence so teams can act before issues escalate.
### Operational Hygiene
* **Challenge:** Recurring issues and noisy alerts persist because insights are fragmented and hygiene work is deprioritized.
* **How it works:** Digest Agent sections track repeating patterns (flaky tests, noisy alerts, SLO misses), highlight cleanup actions, and link to investigations or tickets. Posting to Slack keeps hygiene tasks visible.
* **Outcome:** Continuous improvement loop that reduces toil and alert noise while improving reliability over time.
### Cost Management
* **Challenge:** Infrastructure costs grow silently until someone notices a surprising bill.
* **How it works:** Digest Agents track spend across cloud providers, identify cost spikes, and surface optimization opportunities (idle resources, oversized instances, unused storage).
* **Outcome:** Proactive cost control with clear visibility into where money is going.
### Stakeholder Alignment
* **Challenge:** Leaders and partner teams need consistent, comparable updates; ad-hoc reporting creates misalignment and delays decisions.
* **How it works:** Scheduled Digest Agents present week-over-week trends (e.g., error rates, latency, cost), summarize incidents and mitigations, and outline next-week priorities. Deliver to stakeholder Slack channels for broad visibility.
* **Outcome:** Shared source of truth that speeds decision-making and reduces status meetings.
## How to Create a Digest Agent
### Step 1: Define Task
Specify instructions for your digest agent to generate a report or perform a task.
* Go to Digest Agents and click "Create Digest Agent"
* Add one or more content sections with:
* **Section Name**: A label for this part of the report (e.g., "Weekly Metrics Summary")
* **Instructions**: What TierZero should analyze and report on
* Note: Sections are independent and do not share context with each other
### Step 2: Schedule
Set when and where to send your digest report.
* **Cron Expression**: Define the schedule (e.g., `0 9 * * 1` for 9am every Monday)
* **Slack Channels**: Select channels to post results to
* **Discord Webhooks**: Optionally send to Discord
* **Notification Format**: Choose between summary or full report
### Step 3: Review
Review your configuration and create the digest agent.
* Set a name for your digest agent
* Review all sections and schedule settings
* Click "Create Agent"
## Running Digest Agents
### Manual Runs
* From the digest agent page, click "Run Now" to generate a report immediately
* Choose whether to send Slack notifications for that run
### Scheduled Runs
* Runs automatically based on your cron schedule
* Posts to configured Slack channels
### Reviewing Results
* See your digest agent runs listed with timestamps and status
* Open a run to review the compiled report
## Best Practices
**1. Group Context-Dependent Instructions**
* Sections are independent and do not share context
* If multiple instructions rely on the same telemetry data or intermediate results, keep them together in a single section for higher efficiency
**2. Be Specific About Outputs and Steps**
* Define the desired report format and the exact "how" (e.g., data sources to use, metrics to compare, thresholds/SLAs)
* Specific instructions produce consistent, comparable, and high quality reports across runs
**3. Leverage Prompt Studio**
* Use [Prompt Studio](/features/prompt-studio) to test section prompts against recent data
* Compare versions side-by-side, tighten wording for clarity
* Save improvements back to the digest agent
## Example Prompts
[Digest Prompts](/prompt-library/digest-prompts)
# Incident Agent
Source: https://docs.tierzero.ai/features/incident-agent
Automated incident investigation that surfaces root cause and generates post-mortems
## What is the Incident Agent?
Hours of digging done in minutes. TierZero Incident Agent joins incidents as your right-hand: gathering context, surfacing what's relevant, and helping you figure out how to stop the bleeding and why it happened.
## How It Works
When an incident is raised, TierZero joins and starts gathering context across logs, metrics, traces, code, deploys, and past incidents. Tag @TierZero to delegate new investigation theories or ask for updates.
TierZero synthesizes signals across your stack, including code changes, logs, traces, metrics, deploys, past incidents, and runbooks, then surfaces high-signal clues to the channel.
Auto-generated post-mortem, action items, and Jira tickets. Reduces the painful "recovery to resolution" cycle from days to hours.
## Key Capabilities
### Real-Time Catch-Up
Keep stakeholders in the loop. When your CTO, customer success, or another engineer joins an incident channel mid-flight, they don't need to ask "what's going on?" and no one has to stop debugging to explain.
* **Live dashboard**: Full context, timeline, investigation findings, and charts from your observability tools.
* **Ask TierZero directly**: Tag it anytime for the latest status or to ask specific questions.
* **Ephemeral Slack message**: Private summary sent the moment someone joins the incident.
### Post-Mortem Generation
Post-mortems drafted before the retro starts. TierZero generates a first draft from the signals it collected during the incident.
* **True incident timeline**: Grounded in telemetry data collected during the incident.
* **Customer and service impact assessment**: Scope and severity documented automatically.
* **Report drafted based on your template**, or standard 5-whys format.
* **Action items with suggested ownership**: Clear next steps assigned to the right people.
* **SLO impact assessment**: Which SLOs were breached, error budget consumed.
### Autonomous Debugging
TierZero goes beyond finding the root cause to generate fix PRs. It correlates errors with specific code changes, identifies the offending commit, and opens a pull request with the fix.
* **Code-level root cause attribution**: Pinpoints the exact commit and code path responsible for the failure.
* **Automated fix PR generation**: Opens pull requests with proposed fixes, ready for human review.
* **CI/CD failure diagnosis**: Intelligent log parsing to identify build and deployment failures.
### Automated Remediation
TierZero executes remediation with one-click approval. Every action logged with full audit trail.
* **Rollback to last healthy deploy**: One-click rollback with automatic health validation after deployment.
* **Service restart with health validation**: Restart degraded services and verify recovery before marking resolved.
* **Feature flag toggle**: Disable problematic features instantly to stop the bleeding.
* **Approval workflows**: Human-in-the-loop approval for destructive actions with full audit trail.
## Supported Incident Sources
TierZero integrates with your incident management platform to sync context:
* [PagerDuty](/integrations/pagerduty)
* [FireHydrant](/integrations/firehydrant)
* [Rootly](/integrations/rootly)
* [incident.io](/integrations/incident-io)
* Datadog Incidents
## Use Cases
### Accelerate Incident Resolution
* **Challenge**: Engineers spend hours gathering context from multiple tools during incidents.
* **How it works**: TierZero auto-joins the incident channel, pulls in telemetry from all connected integrations, and surfaces root cause candidates with evidence.
* **Outcome**: 2-minute time to first clue. 40%+ MTTR reduction.
### Consistent Post-Mortems
* **Challenge**: Post-mortems get deprioritized, delayed, and sometimes never finished after an incident.
* **How it works**: TierZero generates a first draft from the signals it collected (timeline, impact, RCA, five-whys, and action items) so the retro starts with substance, not a blank page.
* **Outcome**: Every incident gets a post-mortem. Teams focus on discussing fixes, not reconstructing events.
### Track Incident Metrics
* **Challenge**: MTTA and MTTR are hard to measure consistently without manual data entry.
* **How it works**: TierZero tracks configurable milestones (time to acknowledge, time to mitigate, time to resolve) automatically based on incident channel activity and external system events.
* **Outcome**: Accurate incident metrics without manual effort.
## How to Set Up
1. Navigate to **Settings → Incidents**
2. Configure incident channel name patterns (e.g., `inc-*`, `incident-*`)
3. Select your external incident management system (FireHydrant, Rootly, incident.io, or PagerDuty)
4. Configure metric definitions and milestones
5. Enable the Incident Agent
## Viewing Incidents
The Incidents page provides:
* **Overview dashboard**: MTTA/MTTR charts and trends over time
* **Incident list**: All tracked incidents with status, severity, and key timestamps
* **Incident detail view** with tabs:
* **Executive Summary**: High-level overview of the incident
* **Timeline**: Chronological sequence of events
* **Root Cause Analysis**: Findings with confidence levels
* **Five-Whys**: Structured root cause analysis
* **Follow-Ups**: Action items and their status
* **Communications**: Auto-generated stakeholder updates
* **Custom Post-Mortem**: Report based on your template
## Best Practices
**1. Use Consistent Incident Channel Naming**
* Configure name patterns that match your team's convention (e.g., `inc-*`, `sev1-*`)
* Consistent naming ensures TierZero automatically joins every incident
**2. Connect an External Incident Management System**
* Richer context from FireHydrant, Rootly, or incident.io improves TierZero's investigation quality
* External system data is synced in real-time during incidents
**3. Configure Relevant Milestones**
* Define milestones that match your MTTR goals
* Track what matters to your team's incident response process
**4. Review and Refine Auto-Generated Post-Mortems**
* Use the generated post-mortem as a starting point, not the final word
* Add team-specific context and discuss action items in your retro
# Prompt Studio
Source: https://docs.tierzero.ai/features/prompt-studio
Interactive environment for testing and iterating on agent prompts
## What is Prompt Studio?
Prompt Studio is an interactive environment for testing and iterating on the prompts that power your Alert Agents and Digest Agents. Test prompts against real data, compare versions, and save improvements back to your agents without recreating them.
## Key Capabilities
* **Test investigation runbooks** against real alert payloads to see how TierZero would investigate
* **Test digest section prompts** against real data to preview report output
* **Compare prompt versions** side-by-side to evaluate improvements
* **Save refined prompts** back to agents without recreating them
## How to Use
### Step 1: Navigate to Prompt Studio
Go to **Prompt Studio** in the sidebar.
### Step 2: Select Prompt Type
Choose what you want to test:
* **Alert Investigation Runbook**: The instructions TierZero follows when investigating an alert
* **Impact & Severity Analysis**: How TierZero assesses alert impact and severity
* **Digest Section**: The prompt for a specific section of a digest report
### Step 3: Provide Test Input
Enter a real alert payload or relevant context. Using real data from alerts that produced unsatisfactory results is the best way to improve prompt quality.
### Step 4: Run and Review
Run the test and review the output. TierZero executes the prompt against your test input using the same tools and data sources it would use in production.
### Step 5: Iterate
Make changes to the prompt and re-run to compare. The side-by-side view shows how different prompt versions produce different results.
### Step 6: Save
When satisfied with the output, save the improved prompt back to the agent. The update takes effect immediately for future runs.
## Best Practices
**1. Start with Real Alerts**
* Use an actual alert that produced unsatisfactory results as your test input
* This grounds your iteration in a real problem
**2. Make Incremental Changes**
* Change one thing at a time and compare outputs
* Large rewrites make it hard to tell what improved the results
**3. Test with Diverse Alert Types**
* A prompt that works well for one alert type might fail on another
* Test across different categories to ensure consistency
**4. Save Versions That Work**
* Once a prompt version produces consistently good results across diverse inputs, save it back to the agent
## Related
* [Investigation Runbooks](/prompt-library/investigation-runbooks)
* [Impact & Severity Runbooks](/prompt-library/impact-severity-runbooks)
* [Digest Prompts](/prompt-library/digest-prompts)
# Support Agent
Source: https://docs.tierzero.ai/features/support-agent
AI-powered internal support that investigates live systems to answer engineering questions
## What is a Support Agent?
Get unblocked instantly. Stay in flow forever. TierZero Support Agent responds to queries in your #ask-eng and #ask-infra channels. It goes beyond searching docs to investigate live systems, cross-reference telemetry, and store tribal knowledge from past answers. Tribal knowledge scales without burning out the people who hold it.
## How It Works
Engineers ask questions directly in Slack. "How do I roll back service X?" "What's the runbook for database failover?" No context-switching, no ticket filing, no waiting.
TierZero searches Notion, Confluence, runbooks, code repos, past incidents, and Slack history. It goes beyond docs by cross-referencing live telemetry, deployment state, and code to give answers that are actually current.
Track deflection rate, CSAT, and question categories out of the box. See which topics need better documentation, which answers get thumbs-down, and fine-tune the agent's behavior without writing code.
## Key Capabilities
### Investigates Live Systems
Not just doc retrieval. Support Agent queries your actual infrastructure:
* Build logs and CI/CD pipelines
* Observability platforms (Datadog, New Relic, etc.)
* Recent code commits and deployments
* Service configurations and dependencies
### Per-Channel SOPs
Maximum customizability per team. Configure different behaviors for different channels:
* \#ask-eng can focus on code and deployment questions
* \#ask-infra can prioritize infrastructure and system health
* Each channel can have its own instructions and knowledge sources
### Insights Dashboard
See what's working and tune what isn't:
* Question analytics and trending topics
* Deflection rate tracking
* Knowledge gap detection, highlighting topics where the agent struggles
* CSAT scores from team feedback
### Standard Operating Procedures
Execute common ops tasks directly from Slack with guardrails:
* Restart pods, clear caches, scale deployments
* Junior engineers handle routine operations safely
* Senior engineers stop being a human API
### Gets Smarter Over Time
Learns from feedback:
* Thumbs up/down on responses improves future answers
* Common questions get faster, more accurate responses
* New tribal knowledge is captured automatically
## Use Cases
### Reduce Interruptions for Senior Engineers
* **Challenge**: Senior engineers spend hours answering the same questions repeatedly.
* **How it works**: Support Agent monitors your ask channels and responds with accurate, context-rich answers by investigating live systems.
* **Outcome**: Senior engineers stay focused on building while the team still gets expert-level answers. 85% of questions answered without escalation.
### Faster Onboarding
* **Challenge**: New team members don't know where to find information or who to ask.
* **How it works**: New engineers ask questions in Slack and get immediate, detailed answers with links to relevant code, docs, and systems.
* **Outcome**: Faster ramp-up time and reduced dependency on specific team members. Median response time under 30 seconds.
### Preserve Tribal Knowledge
* **Challenge**: Critical knowledge lives in people's heads and is lost when they leave.
* **How it works**: Every question and answer becomes part of TierZero's knowledge base, building institutional memory over time.
* **Outcome**: Knowledge persists even as team composition changes.
## How to Create a Support Agent
### Step 1: Select Slack Channel
Choose the Slack channel where your support agent will respond to messages.
* Go to Support Agents and click "Create Support Agent"
* Select a public Slack channel to monitor
* The channel must be public and TierZero must be able to join it
### Step 2: Review Instructions & Categories
Review and customize the auto-generated instructions and categories for your support agent.
* TierZero analyzes the channel history to understand the types of questions asked
* **Instructions**: Review and edit the auto-generated instructions that guide how the agent responds
* **Categories**: Review the auto-detected question categories for your channel
* Wait for processing to complete before continuing
### Step 3: Review
Review your configuration and create the support agent.
* Set a name for your support agent
* Review all settings
* Click "Create Agent"
## After Creation
### Monitoring Responses
* View all conversations in the Support Agent details page
* See response quality metrics and feedback
* Track deflection rate and CSAT scores
### Refining Instructions
* Edit instructions based on response quality
* Add or modify categories as new question types emerge
* Use feedback to continuously improve
## Best Practices
**1. Start with High-Volume Channels**
* Begin with channels that get the most questions
* This maximizes impact and provides more data for improvement
**2. Review Auto-Generated Instructions**
* The auto-generated instructions are a starting point
* Customize them based on your team's specific needs and terminology
**3. Encourage Feedback**
* Ask team members to react to responses (thumbs up/down)
* Use feedback to continuously improve answer quality
**4. Monitor Early Responses**
* Pay close attention to the first few days of responses
* Make adjustments quickly based on any issues you see
## Example Instructions
[Support Agent Instructions](/prompt-library/support-agent-instructions)
# Introduction
Source: https://docs.tierzero.ai/getting-started/introduction
Deploy and scale AI agents for incidents, alerts, internal support, and other engineering operations.
![TierZero Dashboard]()
## Core Products
In TierZero, you can spawn different types of AI agents to help you with your day-to-day operational challenges.
When an incident fires, TierZero joins the channel and starts investigating — scanning logs, traces, metrics, and deploys across your entire stack. It surfaces root cause with evidence, recommends a fix, and executes with your approval.
TierZero picks up every alert from PagerDuty, Datadog, Sentry, and Slack and investigates automatically. Known issues get resolved. Related alerts get grouped. No more manual triage.
TierZero generates comprehensive operational reports that surface hidden issues, trends, and cost insights. Get weekly summaries of incidents, alert patterns, system health, and infrastructure spend delivered to your team.
TierZero responds to queries in your #ask-eng and #ask-infra channels. It doesn't just search docs — it investigates live systems: checking build logs, querying Datadog, reading recent commits.
TierZero monitors your CI/CD pipelines end-to-end — detecting flaky tests, diagnosing build failures, and identifying deployment bottlenecks. It auto-triages failures, bisects breaking changes, and keeps your pipelines green.
Ask TierZero anything about your infrastructure directly from Slack or the web app. It connects to your observability stack, codebase, and runbooks to give context-rich answers — no tab-switching required.
# AWS
Source: https://docs.tierzero.ai/integrations/aws
Connect your AWS account to give TierZero access to cloud infrastructure
## Overview
Connect your AWS account to give TierZero read and (optional) write access to your cloud infrastructure. TierZero uses a cross-account IAM role to query CloudWatch metrics and logs, inspect resource configurations, and correlate infrastructure state with incidents without storing long-lived credentials.
## Prerequisites
* Administrative access to your AWS account
* Permission to create IAM roles and attach policies
## Setup Instructions
### Step 1: Navigate to Integration Settings
1. Log into your TierZero dashboard
2. Go to **Settings → Integrations**
3. Click **Connect** next to AWS
4. Click **View Instructions** to see a setup guide with values specific to your organization (Account ID, External ID)
### Step 2: Create an IAM Role
Create a new IAM role in your AWS account that TierZero can assume via cross-account access.
1. In the AWS Console, go to **IAM → Roles → Create role**
2. Select **Custom trust policy** and paste the trust policy below (replacing the placeholder values with the ones from the TierZero setup wizard):
```json theme={null}
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::851725519002:role/tierzero-managed"
},
"Action": "sts:AssumeRole",
"Condition": {
"StringEquals": {
"sts:ExternalId": ""
}
}
}
]
}
```
### Step 3: Attach Permissions
TierZero needs read-only access across your AWS services. Because AWS limits each role to **10 managed policy attachments**, we recommend using the [Terraform setup](#terraform-setup) below for full coverage.
If you prefer to attach managed policies manually, attach the following core set (up to the 10-policy limit):
| Policy | Purpose |
| ------------------------------ | --------------------------------- |
| `CloudWatchReadOnlyAccess` | Logs, metrics, and alarms |
| `AmazonEC2ReadOnlyAccess` | EC2, VPC, and networking metadata |
| `AmazonS3ReadOnlyAccess` | S3 bucket metadata |
| `AmazonRDSReadOnlyAccess` | RDS instance metadata |
| `AWSLambda_ReadOnlyAccess` | Lambda function metadata |
| `IAMReadOnlyAccess` | IAM role and policy metadata |
| `AWSBillingReadOnlyAccess` | Billing and cost data |
| `ElasticLoadBalancingReadOnly` | Load balancer metadata |
| `AutoScalingReadOnlyAccess` | Auto Scaling group metadata |
| `AmazonEKSMCPReadOnlyAccess` | EKS cluster and Kubernetes access |
The managed policy approach covers the most common services but is limited to 10 policies. For comprehensive coverage including DynamoDB, ElastiCache, EBS/EFS, Route53, CloudFront, SQS, and more, use the [Terraform setup](#terraform-setup) which consolidates all permissions into inline policies.
4. Name the role (e.g., `TierZeroAccess`) and click **Create role**
### Step 4: Provide the Role ARN
1. Copy the IAM Role ARN from your AWS console
2. Paste it into the TierZero setup wizard
3. Click **Connect**
## Terraform Setup
If you manage infrastructure as code, use the following Terraform configuration to create the cross-account IAM role with comprehensive read-only access. This approach uses inline policies to consolidate permissions from 18+ AWS managed policies, avoiding the 10-policy attachment limit.
Replace the `external_id` variable value with the External ID shown in your TierZero setup wizard.
```hcl theme={null}
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = ">= 5.0"
}
}
}
variable "role_name" {
description = "Name for the IAM role"
type = string
default = "TierZeroAccess"
}
variable "external_id" {
description = "External ID from TierZero setup wizard"
type = string
}
# IAM Role with Trust Policy
resource "aws_iam_role" "tierzero" {
name = var.role_name
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Effect = "Allow"
Principal = {
AWS = "arn:aws:iam::851725519002:role/tierzero-managed"
}
Action = "sts:AssumeRole"
Condition = {
StringEquals = {
"sts:ExternalId" = var.external_id
}
}
}
]
})
tags = {
ManagedBy = "Terraform"
Purpose = "TierZero cross-account access"
}
}
# EKS MCP Read-Only (managed policy — no inline equivalent)
resource "aws_iam_role_policy_attachment" "eks_mcp_readonly" {
role = aws_iam_role.tierzero.name
policy_arn = "arn:aws:iam::aws:policy/AmazonEKSMCPReadOnlyAccess"
}
# Compute & Networking Read-Only
# Covers: EC2, Lambda, ECS, ELB, Auto Scaling, VPC, CloudFormation, Step Functions
resource "aws_iam_role_policy" "compute_networking_readonly" {
name = "compute-networking-readonly"
role = aws_iam_role.tierzero.id
policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Sid = "EC2ReadOnly"
Effect = "Allow"
Action = [
"ec2:Describe*",
"ec2:Get*",
"ec2:List*",
"ec2:Search*",
]
Resource = "*"
},
{
Sid = "LambdaReadOnly"
Effect = "Allow"
Action = [
"lambda:Get*",
"lambda:List*",
]
Resource = "*"
},
{
Sid = "ECSReadOnly"
Effect = "Allow"
Action = [
"ecs:Describe*",
"ecs:List*",
]
Resource = "*"
},
{
Sid = "ELBReadOnly"
Effect = "Allow"
Action = [
"elasticloadbalancing:Describe*",
"elasticloadbalancing:Get*",
]
Resource = "*"
},
{
Sid = "AutoScalingReadOnly"
Effect = "Allow"
Action = [
"autoscaling:Describe*",
"application-autoscaling:DescribeScalableTargets",
"application-autoscaling:DescribeScalingActivities",
"application-autoscaling:DescribeScalingPolicies",
]
Resource = "*"
},
{
Sid = "CloudFormationReadOnly"
Effect = "Allow"
Action = [
"cloudformation:DescribeStacks",
"cloudformation:ListStacks",
"cloudformation:ListStackResources",
]
Resource = "*"
},
]
})
}
# Storage & Database Read-Only
# Covers: S3, RDS, DynamoDB, DAX, EBS, EFS, ElastiCache, Data Pipeline, Kinesis
resource "aws_iam_role_policy" "storage_database_readonly" {
name = "storage-database-readonly"
role = aws_iam_role.tierzero.id
policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Sid = "S3MetadataOnly"
Effect = "Allow"
Action = [
"s3:ListAllMyBuckets",
"s3:ListBucket",
"s3:ListBucketVersions",
"s3:ListBucketMultipartUploads",
"s3:ListMultipartUploadParts",
"s3:Describe*",
"s3:GetBucket*",
"s3:GetAccessPoint*",
"s3:GetMultiRegionAccessPoint*",
"s3:GetStorageLens*",
"s3:GetEncryptionConfiguration",
"s3:GetLifecycleConfiguration",
"s3:GetReplicationConfiguration",
"s3:GetAccelerateConfiguration",
"s3:GetAnalyticsConfiguration",
"s3:GetIntelligentTieringConfiguration",
"s3:GetInventoryConfiguration",
"s3:GetMetricsConfiguration",
"s3:GetObjectTagging",
"s3:GetObjectVersionTagging",
"s3:GetObjectAcl",
"s3:GetObjectVersionAcl",
"s3:GetObjectRetention",
"s3:GetObjectLegalHold",
"s3:GetObjectAttributes",
"s3:GetObjectVersionAttributes",
]
Resource = "*"
},
{
Sid = "RDSReadOnly"
Effect = "Allow"
Action = [
"rds:Describe*",
"rds:ListTagsForResource",
]
Resource = "*"
},
{
Sid = "DynamoDBMetadataOnly"
Effect = "Allow"
Action = [
"dynamodb:Describe*",
"dynamodb:List*",
"dynamodb:GetAbacStatus",
"dynamodb:GetResourcePolicy",
]
Resource = "*"
},
{
Sid = "EBSReadOnly"
Effect = "Allow"
Action = [
"ec2:DescribeVolumes",
"ec2:DescribeVolumeStatus",
"ec2:DescribeVolumeAttribute",
"ec2:DescribeSnapshots",
"ec2:DescribeSnapshotAttribute",
"ec2:DescribeSnapshotTierStatus",
"ec2:ListSnapshotsInRecycleBin",
"ebs:ListSnapshotBlocks",
"ebs:ListChangedBlocks",
]
Resource = "*"
},
{
Sid = "EFSReadOnly"
Effect = "Allow"
Action = [
"elasticfilesystem:Describe*",
"elasticfilesystem:ListTagsForResource",
]
Resource = "*"
},
{
Sid = "ElastiCacheReadOnly"
Effect = "Allow"
Action = [
"elasticache:Describe*",
"elasticache:List*",
]
Resource = "*"
},
{
Sid = "KinesisReadOnly"
Effect = "Allow"
Action = [
"kinesis:ListStreams",
"kinesis:DescribeStream",
"kinesis:DescribeStreamSummary",
]
Resource = "*"
},
{
Sid = "DynamoDBContributorInsights"
Effect = "Allow"
Action = [
"cloudwatch:GetInsightRuleReport",
]
Resource = "arn:aws:cloudwatch:*:*:insight-rule/DynamoDBContributorInsights*"
},
]
})
}
# Monitoring & Observability Read-Only
# Covers: CloudWatch, Logs, Synthetics, RUM, Application Signals,
# Observability Admin, CloudTrail
resource "aws_iam_role_policy" "monitoring_observability_readonly" {
name = "monitoring-observability-readonly"
role = aws_iam_role.tierzero.id
policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Sid = "CloudWatchReadOnly"
Effect = "Allow"
Action = [
"cloudwatch:BatchGet*",
"cloudwatch:Describe*",
"cloudwatch:GenerateQuery",
"cloudwatch:Get*",
"cloudwatch:List*",
]
Resource = "*"
},
{
Sid = "LogsReadOnly"
Effect = "Allow"
Action = [
"logs:Describe*",
"logs:Get*",
"logs:List*",
"logs:StartQuery",
"logs:StopQuery",
"logs:TestMetricFilter",
"logs:FilterLogEvents",
"logs:StartLiveTail",
"logs:StopLiveTail",
]
Resource = "*"
},
{
Sid = "ApplicationSignalsReadOnly"
Effect = "Allow"
Action = [
"application-signals:BatchGet*",
"application-signals:Get*",
"application-signals:List*",
]
Resource = "*"
},
{
Sid = "SyntheticsReadOnly"
Effect = "Allow"
Action = [
"synthetics:Describe*",
"synthetics:Get*",
"synthetics:List*",
]
Resource = "*"
},
{
Sid = "RUMReadOnly"
Effect = "Allow"
Action = [
"rum:BatchGet*",
"rum:Get*",
"rum:List*",
]
Resource = "*"
},
{
Sid = "ObservabilityAdminReadOnly"
Effect = "Allow"
Action = [
"observabilityadmin:Get*",
"observabilityadmin:List*",
"observabilityadmin:Test*",
"observabilityadmin:Validate*",
]
Resource = "*"
},
{
Sid = "CloudTrailReadOnly"
Effect = "Allow"
Action = [
"cloudtrail:ListChannels",
]
Resource = "*"
},
{
Sid = "CloudTrailAppSignalsScoped"
Effect = "Allow"
Action = [
"cloudtrail:GetChannel",
]
Resource = "arn:aws:cloudtrail:*:*:channel/aws-service-channel/application-signals/*"
},
{
Sid = "ServiceQuotasScoped"
Effect = "Allow"
Action = [
"servicequotas:GetServiceQuota",
]
Resource = [
"arn:aws:servicequotas:*:*:s3/*",
"arn:aws:servicequotas:*:*:dynamodb/*",
"arn:aws:servicequotas:*:*:kinesis/*",
"arn:aws:servicequotas:*:*:sns/*",
"arn:aws:servicequotas:*:*:bedrock/*",
"arn:aws:servicequotas:*:*:lambda/*",
"arn:aws:servicequotas:*:*:fargate/*",
"arn:aws:servicequotas:*:*:elasticloadbalancing/*",
"arn:aws:servicequotas:*:*:ec2/*",
]
},
]
})
}
# Billing & Cost Read-Only
resource "aws_iam_role_policy" "billing_cost_readonly" {
name = "billing-cost-readonly"
role = aws_iam_role.tierzero.id
policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Sid = "CostAnalysis"
Effect = "Allow"
Action = [
"ce:Describe*",
"ce:Get*",
"ce:List*",
"cur:Describe*",
"cur:Get*",
"budgets:Describe*",
"budgets:View*",
]
Resource = "*"
},
]
})
}
# Security, DNS, Messaging & CDN Read-Only
# Covers: IAM, STS, KMS, ACM, WAF, SQS, Route53, CloudFront
resource "aws_iam_role_policy" "security_dns_messaging_readonly" {
name = "security-dns-messaging-readonly"
role = aws_iam_role.tierzero.id
policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Sid = "IAMReadOnly"
Effect = "Allow"
Action = [
"iam:GenerateCredentialReport",
"iam:GenerateServiceLastAccessedDetails",
"iam:Get*",
"iam:List*",
"iam:SimulateCustomPolicy",
"iam:SimulatePrincipalPolicy",
]
Resource = "*"
},
{
Sid = "STSReadOnly"
Effect = "Allow"
Action = [
"sts:GetCallerIdentity",
]
Resource = "*"
},
{
Sid = "KMSReadOnly"
Effect = "Allow"
Action = [
"kms:DescribeKey",
"kms:ListAliases",
]
Resource = "*"
},
{
Sid = "ACMReadOnly"
Effect = "Allow"
Action = [
"acm:DescribeCertificate",
"acm:ListCertificates",
]
Resource = "*"
},
{
Sid = "WAFReadOnly"
Effect = "Allow"
Action = [
"waf:ListWebACLs",
"waf:GetWebACL",
"wafv2:ListWebACLs",
"wafv2:GetWebACL",
]
Resource = "*"
},
{
Sid = "SQSReadOnly"
Effect = "Allow"
Action = [
"sqs:GetQueueAttributes",
"sqs:GetQueueUrl",
"sqs:ListDeadLetterSourceQueues",
"sqs:ListQueues",
"sqs:ListMessageMoveTasks",
"sqs:ListQueueTags",
]
Resource = "*"
},
{
Sid = "Route53ReadOnly"
Effect = "Allow"
Action = [
"route53:Get*",
"route53:List*",
"route53:TestDNSAnswer",
]
Resource = "*"
},
{
Sid = "CloudFrontReadOnly"
Effect = "Allow"
Action = [
"cloudfront:Describe*",
"cloudfront:Get*",
"cloudfront:List*",
"cloudfront-keyvaluestore:Describe*",
"cloudfront-keyvaluestore:Get*",
"cloudfront-keyvaluestore:List*",
]
Resource = "*"
},
{
Sid = "ResourceGroupsAndTags"
Effect = "Allow"
Action = [
"resource-groups:ListGroups",
"resource-groups:ListGroupResources",
"resource-groups:GetGroup",
"resource-groups:GetGroupQuery",
"tag:GetResources",
]
Resource = "*"
},
]
})
}
output "role_arn" {
description = "ARN of the TierZero IAM role — provide this to TierZero"
value = aws_iam_role.tierzero.arn
}
```
Apply with:
```bash theme={null}
terraform init
terraform apply -var="external_id="
```
Then paste the outputted `role_arn` into the TierZero setup wizard.
### Inline Policy Summary
The Terraform configuration above creates **1 managed policy attachment + 5 inline policies**:
| Policy | Services Covered |
| ----------------------------------------- | --------------------------------------------------------------------------------------- |
| **Managed:** `AmazonEKSMCPReadOnlyAccess` | EKS clusters, Kubernetes API |
| `compute-networking-readonly` | EC2, Lambda, ECS, ELB, Auto Scaling, CloudFormation, VPC |
| `storage-database-readonly` | S3, RDS, DynamoDB, EBS, EFS, ElastiCache, Kinesis |
| `monitoring-observability-readonly` | CloudWatch, Logs, Synthetics, RUM, Application Signals, Observability Admin, CloudTrail |
| `billing-cost-readonly` | Cost Explorer, Cost & Usage Reports, Budgets |
| `security-dns-messaging-readonly` | IAM, STS, KMS, ACM, WAF, SQS, Route53, CloudFront |
## Kubernetes (EKS)
If your team uses Amazon EKS, you can grant TierZero access to your Kubernetes clusters for inspecting workloads, pods, and cluster state during investigations. The `AmazonEKSMCPReadOnlyAccess` policy is already included in both the managed policy list and the Terraform configuration above.
### Create an EKS Access Entry
Grant the IAM role access to your EKS cluster:
1. In the AWS Console, go to **EKS → Clusters → *your cluster* → Access**
2. Click **Create access entry**
3. Select the TierZero IAM role as the principal
4. Assign the **AmazonEKSViewPolicy** access policy with **Cluster** scope
5. Click **Create**
Repeat for each EKS cluster you want TierZero to access.
### Allow TierZero's IP Addresses (If Required)
If your EKS cluster's API server endpoint restricts access by IP, you need to add TierZero's outbound IP addresses to the allowlist:
1. In the AWS Console, go to **EKS → Clusters → *your cluster* → Networking**
2. Under **API server endpoint access**, click **Manage**
3. Add TierZero's outbound IP addresses (available in your TierZero dashboard under **Settings → Integrations → AWS**)
This step is only required if your EKS cluster has IP-based restrictions on the API server endpoint. Clusters with public endpoint access enabled without CIDR restrictions do not need this.
## What TierZero Accesses
* **CloudWatch Logs & Metrics**: Query log groups, log streams, metrics, alarms, and X-Ray traces
* **Compute**: EC2 instances, Lambda functions, ECS services, ELB, Auto Scaling groups
* **Storage & Databases**: S3 buckets, RDS instances, DynamoDB tables, EBS volumes, EFS file systems, ElastiCache clusters
* **Networking**: VPCs, subnets, security groups, Route53 DNS, CloudFront distributions
* **EKS clusters**: Kubernetes workloads, pods, and cluster state (if configured)
* **IAM metadata**: Roles, policies, and trust relationships
* **Billing data**: Cost and usage, budgets, invoices
## Security
* TierZero uses **cross-account IAM role assumption**, so no long-lived credentials are stored
* An **External ID** is required to prevent [confused deputy attacks](https://docs.aws.amazon.com/IAM/latest/UserGuide/confused-deputy.html)
* The default policies are **read-only**; write access is opt-in based on your needs
* Revoke access at any time by deleting the IAM role or updating its trust policy
* All API calls are logged in **AWS CloudTrail**
## Troubleshooting
### "Access Denied" Errors
* Verify the IAM role's trust policy allows TierZero's account (`851725519002`) to assume it
* Check that the External ID matches the value shown in the TierZero setup wizard
* Ensure all required policies are attached to the role (check both managed and inline policies)
### EKS Cluster Not Accessible
* Confirm the EKS access entry exists for the TierZero IAM role
* Verify the **AmazonEKSViewPolicy** is assigned with **Cluster** scope
* If using IP restrictions, ensure TierZero's outbound IPs are allowlisted
### Missing Data
* Verify the IAM role has the specific policy for the data type you expect (e.g., `monitoring-observability-readonly` for CloudWatch logs)
* Check that the role ARN entered in TierZero matches the role in your AWS account
* If using the managed policy approach, note that some services (DynamoDB, ElastiCache, EBS/EFS, etc.) require the Terraform setup for coverage
# Bitbucket
Source: https://docs.tierzero.ai/integrations/bitbucket
Code repository and collaboration platform
## Overview
Connect your Bitbucket workspace to give TierZero access to your repositories, commits, pull requests, and pipelines. TierZero uses Bitbucket to correlate code changes with incidents and analyze CI/CD pipeline results.
## Prerequisites
* Administrative access to your Bitbucket workspace
* Permission to create OAuth consumers
## Setup Instructions
### Step 1: Navigate to Integration Settings
1. Log into your TierZero dashboard
2. Go to **Settings → Integrations**
3. Click **Connect** next to Bitbucket
### Step 2: Create an OAuth Consumer
1. In Bitbucket, go to your workspace **Settings → Apps and features → OAuth consumers**
2. Click **Add consumer**
3. Check **This is a private consumer**
4. Set the callback URL to the URL provided in TierZero's setup dialog
5. Grant the following permissions:
| Permission | Level | Purpose |
| ----------------- | ----- | --------------------------------------------------------- |
| **Account** | Read | Required for authentication during the OAuth flow |
| **Repositories** | Write | Read code, push commits, and access repository metadata |
| **Pull requests** | Write | Read and create pull requests, post comments |
| **Pipelines** | Write | Read pipeline status and logs, trigger and stop pipelines |
6. Click **Save**
### Step 3: Enter Credentials
1. Copy the **Key** and **Secret** from the created OAuth consumer
2. Paste them into TierZero's Bitbucket setup fields
3. Click **Connect**
## What TierZero Accesses
* **Repository contents**: Code files, branches, and directory structure
* **Commit history**: Recent commits, diffs, and authors
* **Pull requests**: Open PRs, comments, and approval status
* **Pipelines**: Pipeline run status, logs, and artifacts
## Security
* TierZero uses Bitbucket's OAuth 2.0 flow
* Tokens are encrypted at rest and refreshed automatically
* Revoke access by deleting the OAuth consumer in Bitbucket workspace settings
## Troubleshooting
### OAuth Authorization Fails
* Verify the callback URL matches exactly what's shown in TierZero's setup dialog
* Ensure the OAuth consumer is marked as private
* Check that all required permissions are granted
# Buildkite
Source: https://docs.tierzero.ai/integrations/buildkite
CI/CD platform for running build and deployment pipelines
## Overview
Connect your Buildkite organization to give TierZero access to your CI/CD pipelines, build results, and test suites. TierZero uses Buildkite to analyze build failures, detect flaky tests, and correlate deployments with incidents.
## Prerequisites
* Administrative access to your Buildkite organization
* Permission to create API tokens
## Setup Instructions
### Step 1: Navigate to Integration Settings
1. Log into your TierZero dashboard
2. Go to **Settings → Integrations**
3. Click **Connect** next to Buildkite
### Step 2: Create an API Token
1. In Buildkite, go to **Personal Settings → API Access Tokens**
2. Click **New API Access Token**
3. Name it (e.g., "TierZero Integration")
4. Select your organization
5. Grant **Read** access to the required scopes (builds, pipelines, agents)
6. Click **Create**
7. Copy the token
### Step 3: Enter Credentials
1. Paste the API token into TierZero's setup field
2. Enter your Buildkite organization slug
3. Click **Connect**
## What TierZero Accesses
* **Pipelines**: Pipeline configurations and metadata
* **Builds**: Build results, logs, and timing
* **Jobs**: Individual job results and artifacts
* **Test suites**: Test analytics and flaky test data
* **Annotations**: Build annotations for context
## Integration Resources
After connecting Buildkite, TierZero uses these resource catalogs to understand your pipelines:
* **pipelines.json**: Pipeline names, slugs, descriptions, and relevant annotations/artifacts
* **test\_suites.json**: Test suite configurations
## Capabilities
Buildkite pipelines can be used as [Capabilities](/features/capabilities), which are workflow automations that TierZero can trigger during investigations (e.g., rollback pipelines, diagnostic builds).
## Security
* TierZero uses a scoped API token with minimal permissions
* Tokens are encrypted at rest
* Revoke access by deleting the API token in Buildkite
## Troubleshooting
### Pipeline Not Visible
* Verify the API token has access to the correct organization
* Check that the token has the required read scopes
# Chronosphere
Source: https://docs.tierzero.ai/integrations/chronosphere
Cloud-native observability platform for metrics and monitoring
## Overview
Connect your Chronosphere environment to give TierZero access to your metrics, dashboards, and alerting data. TierZero queries Chronosphere during investigations to analyze service health, correlate metrics with incidents, and surface relevant dashboards.
## Prerequisites
* Administrative access to your Chronosphere environment
* Permission to create service accounts
## Setup Instructions
### Step 1: Create a Chronosphere Service Account
1. In Chronosphere, create a service account following [Chronosphere's service account documentation](https://docs.chronosphere.io/administer/accounts-teams/service-accounts)
2. Use a **Read-only** service account, as TierZero only needs query access
3. Copy the API token that is generated when the service account is created
### Step 2: Navigate to Integration Settings
1. Log into your TierZero dashboard
2. Go to **Settings → Integrations**
3. Click **Connect** next to Chronosphere
### Step 3: Configure the Connection
1. Enter a **Connection Name** (e.g., "Chronosphere-Main")
2. Paste your **API Token** from the Chronosphere service account
3. Enter your **Site Domain** (e.g., `https://myorg.chronosphere.io`)
4. Optionally specify **Dashboard Slugs** to allowlist specific dashboards. If left blank, all dashboards are accessible
5. Click **Save**
### Step 4: Configure Alert Webhooks (Optional)
To receive Chronosphere alerts in TierZero:
1. Open the Chronosphere integration page in TierZero and click your connection
2. Click **Edit** in the Webhooks section
3. Click **+** to create a new webhook
4. In Chronosphere, go to **Alerts → Notifiers** and create a new Notifier
5. Enter the webhook URL from TierZero
6. Enable **Notify when Resolved**
7. Add the Notifier to your team's Notification Policy
## What TierZero Accesses
* **Metrics**: Query and analyze timeseries metrics
* **Dashboards**: Access dashboard configurations and panel data
* **Alerts**: Receive alert notifications via webhook
## Security
* TierZero uses a **read-only service account** with no elevated privileges
* API tokens are encrypted at rest and in transit
* Revoke access at any time by deleting or disabling the service account in Chronosphere
## Troubleshooting
### Connection Fails
* Verify the site domain is correct (include `https://`)
* Check that the API token is valid and the service account is active
* Ensure the service account has read permissions
### Dashboards Not Visible
* If you specified dashboard slugs, verify they match the actual slugs in Chronosphere
* Clear the dashboard slugs field to allow access to all dashboards
# CloudWatch
Source: https://docs.tierzero.ai/integrations/cloudwatch
AWS monitoring service for logs, metrics, and alarms
## Overview
Connect CloudWatch to give TierZero access to your AWS logs, metrics, and alarms. TierZero queries CloudWatch during investigations to analyze log patterns, metric trends, and alarm states.
## Prerequisites
* An [AWS integration](/integrations/aws) configured with IAM role assumption
* CloudWatch log groups and metrics in your AWS account
## Setup Instructions
### Step 1: Configure AWS Integration
If you haven't already, set up the [AWS integration](/integrations/aws) first. CloudWatch access is provided through the AWS IAM role.
### Step 2: Navigate to Integration Settings
1. Go to **Settings → Integrations**
2. Click **Connect** next to CloudWatch
3. Select the AWS region(s) where your CloudWatch data resides
### Step 3: Verify Access
TierZero will validate that it can access CloudWatch using the configured IAM role.
## Required IAM Permissions
If you used the [Terraform setup](/integrations/aws#terraform-setup) for AWS, CloudWatch and Logs permissions are already included in the `monitoring-observability-readonly` inline policy.
If you attached managed policies manually, ensure `CloudWatchReadOnlyAccess` is attached to the role. At a minimum, the role needs:
```json theme={null}
{
"Effect": "Allow",
"Action": [
"logs:GetLogEvents",
"logs:FilterLogEvents",
"logs:DescribeLogGroups",
"logs:DescribeLogStreams",
"cloudwatch:GetMetricData",
"cloudwatch:ListMetrics",
"cloudwatch:DescribeAlarms"
],
"Resource": "*"
}
```
## What TierZero Accesses
* **Log groups and log streams**: Query and filter log events
* **Metrics**: Query metric data and timeseries
* **Alarms**: View alarm configurations and state
## Integration Resources
After connecting CloudWatch, TierZero uses these resource catalogs:
* **log\_groups.json**: List of CloudWatch log groups and their fields
* **metrics.json**: List of CloudWatch metrics to monitor
## Security
* Access is provided through IAM role assumption, so no long-lived credentials are needed
* Read-only access only
* Revoke by updating the IAM role's permissions or trust policy
## Troubleshooting
### "Access Denied" on Log Queries
* Verify the IAM role includes `logs:GetLogEvents` and `logs:FilterLogEvents`
* Check that the role has access to the specific log groups being queried
* Ensure the correct AWS region is selected
# Confluence
Source: https://docs.tierzero.ai/integrations/confluence
Team knowledge base and documentation platform
## Overview
Connect Confluence to give TierZero access to your team's documentation, runbooks, and knowledge base. TierZero searches Confluence during investigations and when answering questions through the Support Agent.
## Prerequisites
* Administrative access to your Atlassian/Confluence instance
* Permission to authorize OAuth applications
## Setup Instructions
### Step 1: Navigate to Integration Settings
1. Log into your TierZero dashboard
2. Go to **Settings → Integrations**
3. Click **Connect** next to Confluence
### Step 2: Authorize via OAuth
1. Click **Connect to Confluence**
2. You'll be redirected to Atlassian to authorize TierZero
3. Select the Confluence site(s) you want to grant access to
4. Review permissions and click **Accept**
5. You'll be redirected back to TierZero
## What TierZero Accesses
* **Pages**: Page content, titles, and metadata
* **Spaces**: Space listings and structure
* **Search**: Full-text search across your Confluence instance
## Security
* TierZero uses Atlassian's OAuth 2.0 flow
* Tokens are automatically refreshed and encrypted at rest
* Revoke access from your Atlassian account settings under **Connected Apps**
## Troubleshooting
### Pages Not Found
* Verify TierZero has access to the correct Confluence site
* Check that the pages are not restricted to specific users or groups that TierZero cannot access
# Custom MCP
Source: https://docs.tierzero.ai/integrations/custom-mcp
Connect external MCP servers to extend TierZero with custom tools
## Overview
Custom MCP servers let you connect external Model Context Protocol servers to TierZero. Once connected, TierZero discovers the tools exposed by that server and can use them during investigations, support conversations, and other agent workflows.
TierZero uses the server description and your MCP-specific instructions to decide when a custom server is relevant. A strong description matters because it is part of the routing signal for when those tools should be used.
TierZero optimizes custom MCP connectors for production use despite claims that "MCP is dead". Simply connect the MCP servers you need for your workflows, and use MCP-specific instructions to give TierZero more context.
## Prerequisites
* Super admin access in TierZero
* A remote MCP server reachable from TierZero over HTTPS
* One supported authentication method: Bearer Token, Custom Headers, or OAuth
* A server that successfully responds to MCP tool discovery
In production, custom MCP server URLs must use `https://`. `http://` URLs are only allowed in development environments.
## Setup Instructions
### Step 1: Open Custom MCP Servers
1. Log into your TierZero dashboard
2. Go to **Settings → Integrations**
3. Open **Custom MCP Servers**
4. Click **Add Server**
### Step 2: Enter Basic Details
1. Add a **Server Name**
2. Add a **Description** that clearly explains what this server handles
Example descriptions:
* `Query and analyze PostgreSQL databases`
* `Inspect internal feature flag configuration and rollout state`
* `Search our internal observability data lake for historical incidents`
Server names must start with a letter and may only contain letters, numbers, and spaces. Hyphens are not allowed in the display name.
### Step 3: Configure the Connection
1. Enter the **Server URL**
2. Choose an **Auth Type**
3. Provide the required credentials
Supported auth types:
* **Bearer Token**: Sends the token as an `Authorization: Bearer ...` header
* **Custom Headers**: Sends one or more custom header key/value pairs
* **OAuth**: Starts an OAuth authorization flow with the MCP provider
### Step 4: Test and Save
For **Bearer Token** and **Custom Headers**:
1. Click **Test Connection**
2. Confirm TierZero can connect and discover tools
3. Click **Add Server**
For **OAuth**:
1. Click **Add Server**
2. Complete the provider's OAuth flow
3. Return to TierZero after authorization completes
If setup succeeds, the server becomes active and TierZero stores the discovered tool list for that endpoint.
## How TierZero Uses Custom MCP Servers
After a server is active, TierZero can call its tools anywhere agent tool use is supported. In practice, TierZero relies on three inputs:
* The **server description** you entered in settings
* The **discovered tool list** from the MCP server
* The **MCP-specific instructions** you maintain in Knowledge & Tuning
## MCP Server Instructions
After you create a custom MCP server, TierZero adds a dedicated file for it under **Knowledge & Tuning → mcp**.
The generated file name is based on the server slug, for example:
* `pg analyze` becomes `pg-analyze.md`
* `feature flags` becomes `feature-flags.md`
Use this file to document:
* When TierZero should use this MCP server
* Which tools are most important
* Any argument conventions or required context
* Safe or unsafe usage patterns
This is the best place to tune how aggressively TierZero should reach for a given custom server.
## Limits and Behavior
* Server names must be unique within the organization
* If connection settings change, TierZero clears the active tool list and requires the connection to be tested again before the updated configuration is trusted
* A server can be saved in an inactive state if creation or update succeeds but tool discovery fails
## Security
* TierZero validates custom MCP URLs and rejects private or internal network targets
* Credentials are encrypted in transit and encrypted at rest. See [TierZero Security](https://tierzero.ai/security) for more detail
* OAuth servers can refresh tokens after setup when supported by the provider
* Access can be removed at any time by deleting the custom MCP server from TierZero
## Troubleshooting
### Test Connection Fails
* Verify the MCP server URL is correct and publicly reachable
* Confirm the auth method matches what the server expects
* For Custom Headers, make sure each header includes both a name and a value
### Server Saves But Stays Inactive
* The server record was created, but TierZero could not discover tools
* Open the server and run **Test Connection** again after fixing the endpoint or credentials
### OAuth Setup Restarts
* If TierZero does not have a valid access token for the server, testing the connection can restart the OAuth flow
* Complete the provider authorization again and return to **Settings → Integrations → Custom MCP Servers**
# Datadog
Source: https://docs.tierzero.ai/integrations/datadog
Monitor and analyze logs, metrics, and traces from Datadog
## Overview
Connect your Datadog account to give TierZero read-only access to your telemetry data. TierZero uses Datadog to query logs, metrics, traces, events, and RUM data during investigations.
## Prerequisites
* Administrative access to your Datadog account
* Permission to create API and Application keys
## Setup Instructions
### Step 1: Navigate to Integration Settings
1. Log into your TierZero dashboard
2. Go to **Settings → Integrations**
3. Click **Connect** next to Datadog
### Step 2: Create an API Key
1. In Datadog, navigate to **Organization Settings → API Keys**
2. Click **New Key**
3. Name it (e.g., "TierZero Integration") and click **Create Key**
4. Copy the API key and paste it into the **API Key** field in TierZero
### Step 3: Create an Application Key
1. In Datadog, navigate to **Organization Settings → Application Keys**
2. Click **New Key**
3. Name it (e.g., "TierZero Integration") and click **Create Key**
4. Edit the scope to include all required permissions:
You must select each permission individually. Missing permissions will prevent TierZero from accessing the corresponding data.
**Required permissions:**
* All `*_read` permissions (for reading metrics, logs, and traces)
* All `logs_read_*` permissions (for log access)
* `timeseries_query` permission (for querying metrics)
5. Copy the Application key and paste it into the **Application Key** field in TierZero
### Step 4: Select Your Datadog Site
Select the site that matches your Datadog account:
| Site | Domain |
| ---- | ------------------- |
| US1 | `datadoghq.com` |
| US3 | `us3.datadoghq.com` |
| US5 | `us5.datadoghq.com` |
| EU1 | `datadoghq.eu` |
| AP1 | `ap1.datadoghq.com` |
| AP2 | `ap2.datadoghq.com` |
### Step 5: Verify Connection
Click **Connect** and verify the integration shows as connected.
## What TierZero Accesses
Once connected, TierZero has read-only access to:
* **Metrics and timeseries data**: query and visualize metrics
* **Logs and log aggregations**: search and analyze log entries
* **APM traces and spans**: distributed tracing data
* **Events**: Datadog events and alerts
* **RUM data**: Real User Monitoring sessions and errors
* **CI/CD pipeline events**: build and deployment telemetry
* **Dashboards and notebooks**: imported resource context
## Integration Resources
After connecting Datadog, TierZero uses these resource catalogs to understand your Datadog setup:
* **logs\_tags.json**: Tags and attributes used in your logs
* **spans\_tags.json**: Tags and attributes used in your spans
* **metrics.json**: List of important metrics
* **dashboards.json**: Imported dashboards
* **notebooks.json**: Imported notebooks
## Security
* TierZero only requires **read-only access**; no write permissions are needed
* API keys are encrypted at rest and in transit
* Revoke access at any time by deleting the keys in your Datadog account
## Troubleshooting
### "Unauthorized" Errors
* Verify the API key and Application key are correct
* Ensure the Application key has all required read permissions
* Check that you selected the correct Datadog site
### Missing Data
* Confirm the Application key has the specific permission for the data type (e.g., `logs_read_data` for logs)
* Verify the data exists in your Datadog account for the time range being queried
# Discord
Source: https://docs.tierzero.ai/integrations/discord
Team messaging via Discord bot integration
## Overview
Connect Discord to enable TierZero to receive alerts, interact with your team, and manage incidents directly in Discord channels. The Discord integration works through a bot that bridges Discord and TierZero via webhooks.
## Prerequisites
* A Discord Developer Application and Bot Token
* An HTTPS endpoint for webhook reception
* A TierZero API key (32-character hex string)
## How It Works
The Discord bot acts as a bridge between Discord and TierZero:
* **Outbound** (Discord → TierZero): Bot forwards mentions, reactions, and server events to TierZero via webhooks
* **Inbound** (TierZero → Discord): TierZero sends messages, reactions, and updates back through the bot
## Setup Instructions
### Step 1: Create a Discord Bot
1. Go to the [Discord Developer Portal](https://discord.com/developers/applications)
2. Create a new application and add a bot
3. Enable the required intents: **Server Members**, **Message Content**, **Guild Messages**, **Guild Message Reactions**
4. Copy the bot token
### Step 2: Configure TierZero
1. Go to **Settings → Integrations** in TierZero
2. Click **Connect** next to Discord
3. Enter your bot token and webhook endpoint URL
4. Copy the TierZero API key provided
### Step 3: Deploy Your Bot
Deploy a bot that:
1. Listens to Discord events (mentions, reactions, server joins/leaves)
2. Forwards events to TierZero's webhook endpoints
3. Exposes webhook endpoints for TierZero to send messages back
### Authentication
All communication between TierZero and your bot uses the header:
```http theme={null}
x-tierzero-discord-key: <32-char-hex-api-key>
```
## Events
### Discord → TierZero
| Event | Trigger | Endpoint |
| ---------------- | ------------------------ | --------------------------------------- |
| `server_joined` | Bot added to a server | `POST /discord/webhooks/server_joined` |
| `bot_mentioned` | User mentions the bot | `POST /discord/webhooks/bot_mentioned` |
| `reaction_added` | User reacts to a message | `POST /discord/webhooks/reaction_added` |
| `server_left` | Bot removed from server | `POST /discord/webhooks/server_left` |
### TierZero → Discord
| Operation | Bot Endpoint |
| -------------- | ------------------------------------- |
| Send message | `POST /webhooks/send_message` |
| Add reaction | `POST /webhooks/add_reaction` |
| Edit message | `POST /webhooks/edit_message` |
| Delete message | `POST /webhooks/delete_message` |
| Get users | `POST /webhooks/get_users` |
| Get channels | `POST /webhooks/get_server_channels` |
| Get messages | `POST /webhooks/get_channel_messages` |
| Get threads | `POST /webhooks/get_channel_threads` |
For detailed webhook specifications, payload schemas, and implementation examples, see the full [Discord Bot reference](/integrations/discord-bot).
## Security
* **HTTPS 1.2+** required for all communications
* **Replay protection**: Each event includes a UUID idempotency key; duplicates are rejected within 24 hours
* Store your API key securely and never expose it in client-side code
## Troubleshooting
### Bot Not Responding
* Verify the bot is online and the webhook endpoint is reachable
* Check that the `x-tierzero-discord-key` header matches
* Ensure the bot has the required Discord intents enabled
# Elastic Cloud
Source: https://docs.tierzero.ai/integrations/elasticsearch-cloud
Search and analyze logs in Elastic Cloud
## Overview
Connect your Elastic Cloud deployment to give TierZero access to your log data. TierZero queries Elasticsearch during investigations to search logs, analyze patterns, and correlate events.
## Prerequisites
* Access to your Elastic Cloud deployment
* Permission to create API keys
## Setup Instructions
### Step 1: Navigate to Integration Settings
1. Log into your TierZero dashboard
2. Go to **Settings → Integrations**
3. Click **Connect** next to Elasticsearch
### Step 2: Create an API Key
1. In Elastic Cloud, open **Stack Management → API Keys**
2. Click **Create API Key**
3. Name it (e.g., "tierzero-read-access")
4. Set the permissions to read-only:
```json theme={null}
{
"role_descriptors": {
"tierzero_reader": {
"cluster": ["monitor"],
"index": [
{
"names": ["*"],
"privileges": ["read", "view_index_metadata"]
}
]
}
}
}
```
5. Click **Create**
6. Copy the **Base64-encoded** API key value
### Step 3: Enter Connection Details
1. Enter your Elastic Cloud **endpoint URL** (e.g., `https://my-deployment.es.us-east-1.aws.found.io:9243`)
2. Paste the **Base64-encoded API key**
3. Click **Connect**
## What TierZero Accesses
* **Log search**: Query logs across indices with time range filtering
* **Index metadata**: Index names and field mappings
* **Aggregations**: Statistical analysis of log data
## Security
* TierZero uses API key authentication with read-only permissions
* API keys are encrypted at rest
* Revoke access by deleting the API key in Elastic Cloud under **Stack Management → API Keys**
## Troubleshooting
### Connection Fails
* Verify the endpoint URL includes the port (typically `:9243` for Elastic Cloud)
* Check that the API key is Base64-encoded (the `encoded` value, not the raw `id:api_key`)
* Ensure the deployment is running and accessible
# Elasticsearch On-Prem
Source: https://docs.tierzero.ai/integrations/elasticsearch-on-prem
Search and analyze logs in self-hosted Elasticsearch
## Overview
Connect your self-hosted Elasticsearch cluster to give TierZero access to your log data. TierZero queries your on-premises Elasticsearch during investigations to search logs, analyze patterns, and correlate events.
## Prerequisites
* Access to your self-hosted Elasticsearch cluster
* Ability to create API keys in Elasticsearch
* Network connectivity between TierZero and your Elasticsearch cluster (port 9200 over HTTPS)
## Setup Instructions
### Step 1: Navigate to Integration Settings
1. Log into your TierZero dashboard
2. Go to **Settings → Integrations**
3. Click **Connect** next to Elasticsearch
### Step 2: Create an API Key
You can create an API key via the Elasticsearch UI or REST API.
**Option 1: Elasticsearch UI**
1. Open **Stack Management → API Keys**
2. Click **Create API Key**
3. Name it (e.g., "tierzero-read-access")
4. Set read-only permissions:
```json theme={null}
{
"role_descriptors": {
"tierzero_reader": {
"cluster": ["monitor"],
"index": [
{
"names": ["*"],
"privileges": ["read", "view_index_metadata"]
}
]
}
}
}
```
5. Click **Create** and copy the **Base64-encoded** API key
**Option 2: REST API**
```bash theme={null}
curl -X POST "https://:9200/_security/api_key" \
-H "Content-Type: application/json" \
-u "elastic:" \
-d '{
"name": "tierzero-read-access",
"role_descriptors": {
"tierzero_reader": {
"cluster": ["monitor"],
"index": [
{
"names": ["*"],
"privileges": ["read", "view_index_metadata"]
}
]
}
}
}'
```
Copy the `encoded` value from the response.
### Step 3: Enter Connection Details
1. Enter your Elasticsearch **cluster endpoint** (e.g., `https://elasticsearch.internal.yourcompany.com:9200`)
2. Paste the **Base64-encoded API key**
3. Click **Connect**
## Network Requirements
Your self-hosted Elasticsearch cluster must be reachable from TierZero over HTTPS:
* **Port**: 9200 (default) or your custom Elasticsearch port
* **Protocol**: HTTPS with a valid TLS certificate
* If your cluster is behind a firewall or VPN, ensure TierZero's IP addresses are allowlisted
* For clusters in AWS, **AWS PrivateLink** is supported. Contact TierZero to set up a private connection without exposing your cluster to the public internet
TierZero requires HTTPS connectivity to your Elasticsearch cluster. Clusters accessible only via internal networks may require AWS PrivateLink, a reverse proxy, or VPN configuration.
## What TierZero Accesses
* **Log search**: Query logs across indices with time range filtering
* **Index metadata**: Index names and field mappings
* **Aggregations**: Statistical analysis of log data
* **Cluster health**: Basic cluster monitoring status
## Security
* TierZero uses API key authentication with **read-only permissions**
* No write, delete, or admin operations are performed
* API keys are encrypted at rest and in transit
* Revoke access by deleting the API key in your Elasticsearch cluster
* Never store credentials in plaintext in configuration files or source control
## Troubleshooting
### Connection Timeout
* Verify the cluster endpoint is reachable from TierZero (check firewall rules)
* Ensure HTTPS is enabled on your Elasticsearch cluster
* Confirm the port number is correct
### Authentication Errors
* Verify the API key is Base64-encoded (the `encoded` value from the creation response)
* Check that the API key has not expired
* Ensure the API key has the required `read` and `monitor` permissions
### TLS Certificate Errors
* Ensure your Elasticsearch cluster has a valid TLS certificate
* Self-signed certificates may require additional configuration
# FireHydrant
Source: https://docs.tierzero.ai/integrations/firehydrant
Incident management and response platform
## Overview
Connect FireHydrant to enable TierZero to receive incident events and participate in your incident response workflow. TierZero can automatically investigate incidents reported through FireHydrant.
## Prerequisites
* Administrative access to your FireHydrant account
* Permission to configure webhook integrations
## Setup Instructions
### Step 1: Navigate to Integration Settings
1. Log into your TierZero dashboard
2. Go to **Settings → Integrations**
3. Click **Connect** next to FireHydrant
### Step 2: Configure Webhook
1. TierZero will display a webhook URL. Copy it
2. In FireHydrant, go to **Settings → Integrations**
3. Add a webhook integration pointing to the TierZero URL
4. Enter the webhook secret provided by TierZero
5. Select the incident events to forward (incident creation, updates, milestones)
### Step 3: Create a Webhook Subscription
1. Back in TierZero, enter the webhook details:
* **Name**: A descriptive name (e.g., "FireHydrant Production")
* **Remote ID**: The integration identifier from FireHydrant
* **Secret**: The webhook authentication secret
Configure webhook subscriptions from the TierZero dashboard to receive these events.
## What TierZero Accesses
* **Incident lifecycle events**: Creation, updates, status changes, milestones
* **Incident metadata**: Severity, affected services, responders
## Using with Alert Agents
After creating a webhook subscription, connect it to an Alert Agent:
1. Create or edit an Alert Agent
2. Select **Webhook** as the alert source
3. Choose your FireHydrant webhook subscription
4. TierZero will automatically investigate incidents reported through FireHydrant
## Using with Incident Agent
FireHydrant incidents can trigger the [Incident Agent](/features/incident-agent), which will auto-join the corresponding Slack channel and begin real-time investigation.
## Security
* Webhook signatures are validated on every incoming event
* Secrets are encrypted at rest and never exposed in API responses
* Revoke by removing the webhook in FireHydrant
## Troubleshooting
### Incidents Not Triggering Investigations
* Verify the webhook URL is correct in FireHydrant
* Check that the webhook secret matches
* Ensure an Alert Agent is connected to the webhook subscription
* Review FireHydrant's webhook delivery logs for errors
# GCP
Source: https://docs.tierzero.ai/integrations/gcp
Connect your Google Cloud Platform account
## Overview
Connect your GCP account to give TierZero access to your Google Cloud infrastructure. TierZero uses GCP integrations for cloud resource context and monitoring data.
## Prerequisites
* Administrative access to your GCP project
* Permission to create service accounts
## Setup Instructions
### Step 1: Navigate to Integration Settings
1. Log into your TierZero dashboard
2. Go to **Settings → Integrations**
3. Click **Connect** next to GCP
### Step 2: Create a Service Account
1. In the GCP Console, go to **IAM & Admin → Service Accounts**
2. Click **Create Service Account**
3. Name it (e.g., "tierzero-integration")
4. Grant the required read-only roles
5. Create a JSON key for the service account
### Step 3: Enter Credentials
1. Upload or paste the service account JSON key in TierZero
2. Click **Connect**
## What TierZero Accesses
* **Cloud resource metadata**: Project and resource configurations
* **Monitoring data**: See [GCP Cloud Monitoring](/integrations/gcp-cloud-monitoring) for detailed metrics and logs access
## Security
* TierZero uses service account impersonation with minimal permissions
* Credentials are encrypted at rest
* Revoke access by deleting the service account or removing its key
## Troubleshooting
### Permission Errors
* Verify the service account has the required roles
* Check that the service account key has not been rotated or deleted
# GCP Cloud Monitoring
Source: https://docs.tierzero.ai/integrations/gcp-cloud-monitoring
Google Cloud monitoring for metrics, logs, and alerts
## Overview
Connect GCP Cloud Monitoring to give TierZero access to your Google Cloud metrics, logs, and alerting data. TierZero queries Cloud Monitoring during investigations to analyze service health and correlate events.
## Prerequisites
* A [GCP integration](/integrations/gcp) configured with a service account
* Cloud Monitoring enabled in your GCP project
## Setup Instructions
### Step 1: Configure GCP Integration
If you haven't already, set up the [GCP integration](/integrations/gcp) first. Cloud Monitoring access is provided through the GCP service account.
### Step 2: Navigate to Integration Settings
1. Go to **Settings → Integrations**
2. Click **Connect** next to GCP Cloud Monitoring
3. Select the GCP project(s) to monitor
### Step 3: Verify Access
TierZero will validate access to Cloud Monitoring using the configured service account.
## Required Roles
The service account must have:
* `roles/monitoring.viewer`: Read metrics and alerting policies
* `roles/logging.viewer`: Read Cloud Logging data
## What TierZero Accesses
* **Metrics**: Custom and system metrics, timeseries data
* **Logs**: Cloud Logging entries and log-based metrics
* **Alerts**: Alerting policy configurations and incidents
## Security
* Access through GCP service account impersonation
* Read-only roles only
* Revoke by removing the monitoring/logging roles from the service account
## Troubleshooting
### Missing Metrics
* Verify the service account has `roles/monitoring.viewer`
* Check that metrics exist in the selected GCP project
* Ensure the correct project is selected in the integration settings
# GitHub
Source: https://docs.tierzero.ai/integrations/github
Code repository and collaboration platform
## Overview
Connect your GitHub organization to give TierZero access to your repositories, commits, deployments, and pull requests. TierZero uses GitHub to correlate code changes with incidents, analyze recent deployments, and generate fix PRs.
## Prerequisites
* Administrative access to your GitHub organization
* Permission to install GitHub Apps
## Setup Instructions
### Step 1: Navigate to Integration Settings
1. Log into your TierZero dashboard
2. Go to **Settings → Integrations**
3. Click **Connect** next to GitHub
### Step 2: Install the GitHub App
1. Click **Install on GitHub**
2. Select your GitHub organization
3. Choose repository access:
* **All repositories**: Access all current and future repositories (recommended for comprehensive coverage)
* **Only select repositories**: Grant access to specific repositories
### Step 3: Complete Installation
1. Review the permissions and click **Install**
2. You'll be redirected back to TierZero
3. The integration should now show as connected
## Permissions
The TierZero GitHub App requests:
### Read Access
* **Actions**: View workflow run results and logs
* **Checks**: View check suite and check run results
* **Code**: Read repository contents, commits, and branches
* **Commit statuses**: View commit status checks
* **Deployments**: View deployment history and status
* **Metadata**: Access basic repository information
### Read and Write Access
* **Pull requests**: Read PR details and discussions, create fix PRs and post review comments
## What TierZero Accesses
* **Repository contents**: Code files, directory structure
* **Commit history**: Recent commits, diffs, and authors
* **Deployment history**: Deploy timestamps, environments, and status
* **Pull requests**: Open PRs, review comments, CI check results
* **GitHub Actions**: Workflow run logs and results
* **Repository list**: Available as imported repository context
## Integration Resources
After connecting GitHub, your repositories are automatically cataloged for TierZero to use during code-aware investigations.
## Modifying Access
You can change which repositories TierZero has access to at any time:
1. Go to your GitHub organization's **Settings → GitHub Apps**
2. Click **Configure** next to TierZero
3. Update repository access
4. Changes take effect immediately
## Security
* TierZero uses the GitHub App model (not personal access tokens)
* Read-only access to everything except pull requests
* Access can be revoked at any time from GitHub's app settings
* Installation-scoped tokens with minimal permissions
## Troubleshooting
### Repository Not Visible
* Verify the repository is included in the GitHub App's repository access settings
* If using "Only select repositories," add the missing repository
### "Not Found" Errors
* The GitHub App may not have access to the repository
* Check that the app is still installed on your organization
# GitLab
Source: https://docs.tierzero.ai/integrations/gitlab
Code repository and DevOps platform
## Overview
Connect your GitLab instance to give TierZero access to your repositories, merge requests, pipelines, and deployments. TierZero uses GitLab to correlate code changes with incidents and analyze CI/CD pipeline results.
## Prerequisites
* Administrative access to your GitLab group or instance
* Permission to create access tokens
## Setup Instructions
### Step 1: Navigate to Integration Settings
1. Log into your TierZero dashboard
2. Go to **Settings → Integrations**
3. Click **Connect** next to GitLab
### Step 2: Create an Access Token
1. In GitLab, go to your group's **Settings → Access Tokens** (or use a personal access token)
2. Create a new token with the following scopes:
* `read_api`: Read access to the API
* `read_repository`: Read access to repository files
3. Set an expiration date and click **Create**
4. Copy the generated token
### Step 3: Enter Credentials
1. Paste the access token into TierZero
2. Enter your GitLab instance URL (for self-hosted instances)
3. Click **Connect**
## What TierZero Accesses
* **Repository contents**: Code files, branches, and directory structure
* **Commit history**: Recent commits, diffs, and authors
* **Merge requests**: Open MRs, discussions, and CI results
* **Pipelines**: CI/CD pipeline runs, jobs, and logs
* **Deployments**: Deployment history and environments
## Security
* TierZero uses scoped access tokens
* Read-only access to repositories and API
* Tokens are encrypted at rest
* Revoke access by deleting the token in GitLab
## Troubleshooting
### Repository Not Visible
* Verify the access token has the `read_api` and `read_repository` scopes
* For group tokens, check that the repository belongs to the token's group
* For self-hosted instances, ensure the GitLab URL is correct
# Grafana Cloud
Source: https://docs.tierzero.ai/integrations/grafana-cloud
Observability platform for metrics, logs, and traces
## Overview
Connect Grafana Cloud to give TierZero access to your metrics, logs, and traces. TierZero queries your Grafana Cloud data sources during investigations.
## Prerequisites
* Administrative access to your Grafana Cloud instance
* Permission to create service account tokens
## Setup Instructions
### Step 1: Navigate to Integration Settings
1. Log into your TierZero dashboard
2. Go to **Settings → Integrations**
3. Click **Connect** next to Grafana Cloud
### Step 2: Create a Service Account Token
1. In Grafana Cloud, go to **Administration → Service Accounts**
2. Create a new service account with **Viewer** role
3. Generate a token for the service account
4. Copy the token
### Step 3: Enter Credentials
1. Paste the service account token into TierZero
2. Enter your Grafana Cloud instance URL
3. Click **Connect**
## What TierZero Accesses
* **Dashboards**: Dashboard configurations and panel queries
* **Data sources**: Prometheus, Loki, Tempo, and other configured data sources
* **Metrics**: Prometheus metrics via data source queries
* **Logs**: Loki log queries
* **Traces**: Tempo trace queries
## Security
* TierZero uses a Grafana service account token with Viewer role
* Read-only access to dashboards and data sources
* Tokens are encrypted at rest
* Revoke by deleting the service account or token in Grafana Cloud
## Troubleshooting
### Dashboard Not Visible
* Verify the service account has at least Viewer permissions
* Check that the dashboard is not restricted to specific teams or folders
# incident.io
Source: https://docs.tierzero.ai/integrations/incident-io
Incident management platform
## Overview
Connect incident.io to enable TierZero to receive incident events and participate in your incident response workflow. TierZero can automatically investigate incidents reported through incident.io.
## Prerequisites
* Administrative access to your incident.io account
* Permission to configure webhook integrations
## Setup Instructions
### Step 1: Navigate to Integration Settings
1. Log into your TierZero dashboard
2. Go to **Settings → Integrations**
3. Click **Connect** next to incident.io
### Step 2: Configure Webhook
1. TierZero will display a webhook URL. Copy it
2. In incident.io, configure a webhook pointing to the TierZero URL
3. Enter the webhook secret provided by TierZero
4. Select the incident events to forward
### Step 3: Create a Webhook Subscription
1. Back in TierZero, enter the webhook details:
* **Name**: A descriptive name (e.g., "incident.io Production")
* **Remote ID**: The integration identifier from incident.io
* **Secret**: The webhook authentication secret
Configure webhook subscriptions from the TierZero dashboard to receive these events.
## What TierZero Accesses
* **Incident lifecycle events**: Creation, updates, status changes
* **Incident metadata**: Severity, affected services, responders
## Using with Incident Agent
incident.io incidents can trigger the [Incident Agent](/features/incident-agent), which auto-joins the corresponding Slack channel and begins real-time investigation.
## Security
* Webhook signatures are validated on every incoming event
* Secrets are encrypted at rest
* Revoke by removing the webhook in incident.io
## Troubleshooting
### Incidents Not Triggering
* Verify the webhook URL is correct in incident.io
* Check that the webhook secret matches
* Ensure an Alert Agent or Incident Agent is configured to respond
# Jira
Source: https://docs.tierzero.ai/integrations/jira
Project management and issue tracking
## Overview
Connect Jira to enable TierZero to create and reference issues during investigations. TierZero can create follow-up tickets, link related issues, and provide context from existing Jira issues.
## Prerequisites
* Administrative access to your Atlassian/Jira instance
* Permission to authorize OAuth applications
## Setup Instructions
### Step 1: Navigate to Integration Settings
1. Log into your TierZero dashboard
2. Go to **Settings → Integrations**
3. Click **Connect** next to Jira
### Step 2: Authorize via OAuth
1. Click **Connect to Jira**
2. You'll be redirected to Atlassian to authorize TierZero
3. Select the Jira site(s) you want to grant access to
4. Review permissions and click **Accept**
5. You'll be redirected back to TierZero
## What TierZero Accesses
* **Issues**: Read and create issues, add comments
* **Projects**: List projects and issue types
* **Search**: Query issues using JQL
## Security
* TierZero uses Atlassian's OAuth 2.0 flow
* Tokens are automatically refreshed and encrypted at rest
* Revoke access from your Atlassian account settings under **Connected Apps**
## Troubleshooting
### Cannot Create Issues
* Verify TierZero has write access to the target project
* Check that the required issue fields are available for the selected issue type
# Langfuse
Source: https://docs.tierzero.ai/integrations/langfuse
LLM observability and analytics platform
## Overview
Connect Langfuse to give TierZero visibility into your LLM application performance. TierZero uses Langfuse data to monitor AI/ML pipeline health, trace requests, and analyze model behavior during investigations.
## Prerequisites
* Access to your Langfuse instance
* Permission to create API keys
## Setup Instructions
### Step 1: Navigate to Integration Settings
1. Log into your TierZero dashboard
2. Go to **Settings → Integrations**
3. Click **Connect** next to Langfuse
### Step 2: Enter Credentials
1. In Langfuse, go to **Settings → API Keys**
2. Create a new API key with read access
3. Copy the public key and secret key
4. Enter your Langfuse host URL, public key, and secret key in TierZero
5. Click **Connect**
## What TierZero Accesses
* **Traces**: LLM request traces and spans
* **Scores**: Quality and evaluation scores
* **Sessions**: User session data
* **Metrics**: Latency, token usage, and cost data
## Security
* TierZero uses API key authentication
* Read-only access to traces and metrics
* Keys are encrypted at rest
* Revoke by deleting the API key in Langfuse
## Troubleshooting
### Connection Fails
* Verify the host URL is correct (include the protocol, e.g., `https://`)
* Check that the public and secret keys are entered correctly
* Ensure the Langfuse instance is accessible from TierZero
# LangSmith
Source: https://docs.tierzero.ai/integrations/langsmith
LLM development and monitoring platform by LangChain
## Overview
Connect LangSmith to give TierZero visibility into your LLM application performance. TierZero uses LangSmith data to monitor AI/ML pipeline health, trace requests, and debug model issues.
## Prerequisites
* Access to your LangSmith account
* Permission to create API keys
## Setup Instructions
### Step 1: Navigate to Integration Settings
1. Log into your TierZero dashboard
2. Go to **Settings → Integrations**
3. Click **Connect** next to LangSmith
### Step 2: Enter Credentials
1. In LangSmith, go to **Settings → API Keys**
2. Create a new API key
3. Copy the API key
4. Paste it into TierZero and click **Connect**
## What TierZero Accesses
* **Traces**: LLM request traces and spans
* **Runs**: Individual LLM invocations and chain runs
* **Feedback**: User feedback and evaluation data
* **Projects**: Project listings and metadata
## Security
* TierZero uses API key authentication
* Read-only access
* Keys are encrypted at rest
* Revoke by deleting the API key in LangSmith
## Troubleshooting
### Missing Traces
* Verify the API key has access to the project containing the traces
* Check that tracing is enabled in your LangSmith application
# Linear
Source: https://docs.tierzero.ai/integrations/linear
Install TierZero as a Linear workspace member to trigger investigations and support workflows from Linear issues
## Overview
TierZero can be installed as a Linear agent — an assignable, mentionable member of your Linear workspace. Once installed, any issue assigned to TierZero (or comment that @mentions it) opens an agent session in Linear's UI. TierZero acknowledges within seconds, streams its thinking and tool calls back as activities, and posts a final response when it's done.
![A Linear issue activity feed: TierZero is assigned the issue, completes an investigation with root cause, and starts drafting a fix PR after a follow-up message]()
Linear is treated as a new channel for the existing [Support Agent](/features/support-agent) product. Issues that match a Support Agent's routing rules run that agent's runbook; everything else falls through to a generic ad-hoc chat session — same as @mentioning TierZero in Slack.
## Prerequisites
* Administrative access to your Linear workspace (required to install the agent)
* Admin role in TierZero with `write:integrations` permission
## Setup Instructions
### Step 1: Install the Agent in Your Workspace
1. In TierZero, go to **Settings → Integrations → Project Management → Linear**
2. Open the **Agent** tab
3. Click **Install Agent**
4. You'll be redirected to Linear to authorize the installation
5. Approve the requested permissions (assignable, mentionable, post agent activities)
6. You'll be redirected back to TierZero with the installation listed under the Agent tab
After installation, TierZero appears as a member of your Linear workspace. It can be assigned to issues, mentioned in comments, and added to teams just like a human user. The agent does not consume a billable Linear seat.
You can install the agent into multiple Linear workspaces from the same TierZero org — each workspace shows up as its own row in the Agent tab.
### Step 2: Route Issues to TierZero (Linear Side)
This step happens entirely in Linear and is up to you. Common patterns:
* **Manual assignment**: Engineers assign individual issues to TierZero as needed
* **Linear automations**: Auto-assign issues that match conditions (e.g. "when issue created in Security team → delegate to TierZero")
* **Labels or dedicated projects**: Route issues via a `tierzero` label or a dedicated project
TierZero is passive — it processes whatever is assigned or @mentioned. How issues arrive is your choice.
### Step 3: Configure a Support Agent for Linear (Optional)
To run a specific runbook on issues that match certain criteria, create a Support Agent with Linear as the source:
1. Go to **Support Agents → Create Support Agent**
2. Choose **Linear** as the source
3. Pick the Linear team this agent should monitor
4. (Optional) Add filter rules — match by labels and/or project. Rules support AND/OR logic.
5. Write filtering instructions / runbook
6. Save
Issues assigned to TierZero in the chosen team that pass the filter will run this Support Agent's runbook. Issues that match no Support Agent fall through to a generic ad-hoc chat session — full tool access, no runbook.
Each Linear team can be bound to at most one Support Agent. Use filter rules within that agent if you need to differentiate behavior by label or project inside the same team.
## Removing the Agent
Removing the agent is a two-sided action — disconnect on the TierZero side, then revoke access on the Linear side.
### Step 1: Disconnect on TierZero
1. **Settings → Integrations → Linear → Agent tab**
2. Click the menu next to the workspace and choose **Disconnect**
This removes the OAuth credentials and stops TierZero from processing webhooks for that workspace.
### Step 2: Revoke Access in Linear
1. In Linear, go to **Settings → API → Authorized applications**
2. Find the TierZero agent and click **Revoke**
Until access is revoked in Linear, the TierZero agent will still appear as an installed application in your workspace.
# New Relic
Source: https://docs.tierzero.ai/integrations/new-relic
Full-stack observability platform
## Overview
Connect New Relic to give TierZero access to your application performance data, logs, metrics, and traces. TierZero queries New Relic during investigations using NRQL to analyze service health and correlate events.
## Prerequisites
* Access to your New Relic account
* Permission to create User API keys
## Setup Instructions
### Step 1: Navigate to Integration Settings
1. Log into your TierZero dashboard
2. Go to **Settings → Integrations**
3. Click **Connect** next to New Relic
### Step 2: Create a User API Key
1. In New Relic, go to **API Keys** (under your profile)
2. Click **Create a key**
3. Select **User** as the key type
4. Name it (e.g., "TierZero Integration")
5. Click **Create**
6. Copy the generated key
### Step 3: Enter Credentials
1. Paste the User API key into TierZero
2. Select your New Relic account ID
3. Click **Connect**
## What TierZero Accesses
* **Applications**: APM application health and performance
* **Metrics**: Custom and system metrics via NRQL queries
* **Logs**: Log entries and attributes
* **Traces and spans**: Distributed tracing data and span attributes
* **Transactions**: Transaction-level performance data
* **Dashboards**: Imported resource context
## Integration Resources
After connecting New Relic, TierZero uses these resource catalogs:
* **applications.json**: List of New Relic applications
* **dashboards.json**: Imported dashboards
* **logs\_attrs.json**: Log attributes
* **metrics.json**: Available metrics
* **spans\_attrs.json**: Span attributes
* **transactions\_attrs.json**: Transaction attributes
## Security
* TierZero uses a User API key for NRQL queries
* Read-only access to your observability data
* Keys are encrypted at rest
* Revoke by deleting the API key in New Relic
## Troubleshooting
### NRQL Query Errors
* Verify the User API key is valid and has not expired
* Check that the account ID is correct
* Ensure the queried data exists in your New Relic account
# Notion
Source: https://docs.tierzero.ai/integrations/notion
Knowledge base and documentation platform
## Overview
Connect Notion to give TierZero access to your team's documentation, runbooks, and internal knowledge base. TierZero searches Notion pages during investigations and when answering questions through the Support Agent.
## Prerequisites
* Administrative access to your Notion workspace
* Permission to create integrations
## Setup Instructions
### Step 1: Navigate to Integration Settings
1. Log into your TierZero dashboard
2. Go to **Settings → Integrations**
3. Click **Connect** next to Notion
### Step 2: Authorize Notion
1. Click **Connect to Notion**
2. You'll be redirected to Notion to authorize TierZero
3. Select the pages and databases you want to share with TierZero
4. Click **Allow access**
5. You'll be redirected back to TierZero
You can share specific pages or entire workspaces. TierZero can only access pages explicitly shared during authorization.
## What TierZero Accesses
* **Pages**: Page content, titles, and metadata
* **Databases**: Database entries and properties
* **Search**: Full-text search across shared pages
## Integration Resources
After connecting Notion, your shared pages are cataloged for use during investigations and support workflows.
## Security
* TierZero uses Notion's OAuth 2.0 integration flow
* Access is limited to explicitly shared pages
* Tokens are encrypted at rest
* Revoke access from Notion's **Settings → Connections**
## Troubleshooting
### Pages Not Found
* Verify the pages are shared with the TierZero integration in Notion
* Check that the Notion connection is still active in **Settings → Integrations**
* If you added new pages after connecting, you may need to re-authorize to share them
# OpsGenie
Source: https://docs.tierzero.ai/integrations/opsgenie
Alert management and on-call scheduling
## Overview
Connect OpsGenie to enable TierZero to receive alert events and participate in your alert response workflow. TierZero can automatically investigate alerts routed through OpsGenie.
## Prerequisites
* Administrative access to your OpsGenie account
* Permission to create API integrations
## Setup Instructions
### Step 1: Navigate to Integration Settings
1. Log into your TierZero dashboard
2. Go to **Settings → Integrations**
3. Click **Connect** next to OpsGenie
### Step 2: Create a Webhook Integration in OpsGenie
1. In OpsGenie, go to **Settings → Integrations → Add Integration**
2. Select **Webhook** integration type
3. Copy the webhook URL provided by TierZero
4. Configure the webhook to send alert events
5. Copy the **Integration ID** and **API key** from OpsGenie
### Step 3: Create a Webhook Subscription in TierZero
1. Back in TierZero, enter the webhook details:
* **Name**: A descriptive name (e.g., "OpsGenie Production Alerts")
* **Remote ID**: The Integration ID from OpsGenie
* **Secret**: The webhook authentication secret
* **API Key**: The OpsGenie API key (required for alert enrichment)
The OpsGenie API key is required at creation time for the webhook to function properly. It cannot be changed after creation.
Configure webhook subscriptions from the TierZero dashboard to receive these events.
## What TierZero Accesses
* **Alert events**: Alert creation, acknowledgment, closure
* **Alert metadata**: Severity, description, tags, affected services
* **Alert enrichment**: Additional context fetched via the OpsGenie API key
## Using with Alert Agents
After creating a webhook subscription, connect it to an Alert Agent:
1. Create or edit an Alert Agent
2. Select **Webhook** as the alert source
3. Choose your OpsGenie webhook subscription
4. TierZero will automatically investigate alerts
## API Reference
For programmatic webhook management, see the [OpsGenie Webhook API](/references/tz_api) reference.
## Security
* Webhook signatures are validated on every incoming event
* API keys and secrets are encrypted at rest and never returned in API responses
* Revoke by removing the webhook integration in OpsGenie
## Troubleshooting
### Alerts Not Triggering Investigations
* Verify the webhook URL is correct in OpsGenie
* Check that both the secret and API key are configured
* Ensure an Alert Agent is connected to the webhook subscription
* Review OpsGenie's integration logs for delivery errors
# Integrations Overview
Source: https://docs.tierzero.ai/integrations/overview
Connect TierZero to your telemetry, code, and operational systems
## Overview
TierZero integrates with your existing infrastructure to provide intelligent investigation, automated response, and proactive monitoring. This page provides an overview of available integrations and setup priorities.
## Available Integrations
| Category | Integrations |
| ----------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **AI Observability** | [Langfuse](/integrations/langfuse), [LangSmith](/integrations/langsmith) |
| **Cloud Infrastructure** | [AWS](/integrations/aws), [Buildkite](/integrations/buildkite), [GCP](/integrations/gcp), [Temporal](/integrations/temporal) |
| **Code & Source Control** | [Bitbucket](/integrations/bitbucket), [GitHub](/integrations/github), [GitLab](/integrations/gitlab) |
| **Communication** | [Discord](/integrations/discord), [Slack](/integrations/slack) |
| **Custom MCP** | [Custom MCP](/integrations/custom-mcp) |
| **Data Warehouse** | [Snowflake](/integrations/snowflake) |
| **Incident Response** | [FireHydrant](/integrations/firehydrant), [incident.io](/integrations/incident-io), [OpsGenie](/integrations/opsgenie), [PagerDuty](/integrations/pagerduty), [Rootly](/integrations/rootly) |
| **Knowledge Base** | [Confluence](/integrations/confluence), [Notion](/integrations/notion) |
| **Project Management** | [Jira](/integrations/jira), [Linear](/integrations/linear) |
| **Telemetry & Observability** | [Chronosphere](/integrations/chronosphere), [CloudWatch](/integrations/cloudwatch), [Datadog](/integrations/datadog), [Elastic Cloud](/integrations/elasticsearch-cloud), [Elasticsearch On-Prem](/integrations/elasticsearch-on-prem), [GCP Cloud Monitoring](/integrations/gcp-cloud-monitoring), [Grafana Cloud](/integrations/grafana-cloud), [New Relic](/integrations/new-relic), [Prometheus](/integrations/prometheus), [Sentry](/integrations/sentry) |
## Integration Priority
For optimal performance, set up integrations in this order:
### Essential (Required)
These three categories are necessary for core functionality:
* **Communication**: [Slack](/integrations/slack) is required for TierZero to interact with your team
* **Observability**: At least one telemetry platform (Datadog, New Relic, etc.) for logs, metrics, and traces
* **Code**: At least one code platform (GitHub, GitLab, Bitbucket) for repository and deployment context
### Recommended
* **Alert & Incident Response**: Connect PagerDuty, OpsGenie, or other incident platforms to enable Alert Agents and Incident Agent
* **Knowledge Base**: Connect Notion or Confluence for runbook and documentation access
* **CI/CD**: Connect Buildkite or Temporal for build and workflow context
### Optional
* **Project Management**: Jira or Linear for ticket creation and tracking
* **Cloud Infrastructure**: AWS or GCP for cloud-specific resource context
* **Data Warehouse**: Snowflake for data warehouse queries
* **AI/ML Observability**: LangSmith or Langfuse for LLM application monitoring
* **Custom MCP**: Connect external MCP servers when you want TierZero to use internal or domain-specific tools
## Setup
To connect an integration:
1. Navigate to **Settings → Integrations** in your TierZero dashboard
2. Click the **Connect** button for the integration you want to add
3. Follow the setup wizard and authorize the required permissions
4. Verify the integration shows as **Connected**
See each integration's dedicated page for detailed setup instructions.
## Security
All integrations follow these principles:
* **Read-only access** whenever possible, with write access only where needed for functionality (e.g., posting Slack messages, creating PRs)
* **Encrypted credentials** at rest and in transit
* **Revocable access** at any time through the provider's settings or TierZero's integration page
* **Minimal permissions**: only the permissions required for functionality are requested
## Resource Context
Many integrations provide structured resource context such as dashboards, pipelines, tags, repositories, and attributes. TierZero uses that metadata alongside live telemetry and code access to make investigations more accurate.
# PagerDuty
Source: https://docs.tierzero.ai/integrations/pagerduty
Incident management and on-call scheduling
## Overview
Connect PagerDuty to enable TierZero to receive incident events and participate in your incident response workflow. TierZero can automatically investigate incidents triggered through PagerDuty.
## Prerequisites
* Administrative access to your PagerDuty account
* Permission to configure webhook subscriptions
## Setup Instructions
### Step 1: Navigate to Integration Settings
1. Log into your TierZero dashboard
2. Go to **Settings → Integrations**
3. Click **Connect** next to PagerDuty
### Step 2: Configure PagerDuty Webhook
1. TierZero will display a webhook URL. Copy it
2. In PagerDuty, go to **Integrations → Generic Webhooks (v3)**
3. Click **New Webhook**
4. Paste the TierZero webhook URL as the endpoint
5. Select the events to forward:
* `incident.triggered`
* `incident.acknowledged`
* `incident.resolved`
6. Optionally filter by service or team
7. Copy the webhook secret
### Step 3: Create a Webhook Subscription in TierZero
1. Back in TierZero, enter the webhook details:
* **Name**: A descriptive name (e.g., "PagerDuty Production")
* **Remote ID**: The webhook subscription ID from PagerDuty
* **Secret**: The webhook signing secret
Configure webhook subscriptions from the TierZero dashboard to receive these events.
## What TierZero Accesses
* **Incident events**: Trigger, acknowledgment, and resolution events
* **Incident metadata**: Severity, service, escalation policy, assigned responders
## Using with Alert Agents
After creating a webhook subscription, connect it to an Alert Agent:
1. Create or edit an Alert Agent
2. Select **Webhook** as the alert source
3. Choose your PagerDuty webhook subscription
4. TierZero will automatically investigate incidents
## Using with Incident Agent
PagerDuty incidents can trigger the [Incident Agent](/features/incident-agent), which auto-joins the corresponding Slack channel and begins real-time investigation.
## Security
* PagerDuty webhook signatures are validated using HMAC-SHA256
* Secrets are encrypted at rest and never returned in API responses
* Revoke by deleting the webhook subscription in PagerDuty
## Troubleshooting
### Incidents Not Triggering
* Verify the webhook URL is correct in PagerDuty
* Check that the webhook secret matches
* Confirm the webhook is not filtered to exclude the triggering service
* Review PagerDuty's webhook delivery logs under **Integrations → Generic Webhooks**
# Prometheus
Source: https://docs.tierzero.ai/integrations/prometheus
Query Prometheus metrics from self-hosted or private network deployments
## Overview
Connect Prometheus to give TierZero access to your metrics during investigations. TierZero queries the Prometheus HTTP API to analyze metric behavior, correlate incidents, and support root-cause analysis.
## Prerequisites
* Access to a Prometheus endpoint that exposes the HTTP API
* A reachable endpoint URL for TierZero (public endpoint or AWS PrivateLink)
* Authentication details, if your deployment requires auth
## Setup Instructions
### Step 1: Navigate to Integration Settings
1. Log into your TierZero dashboard
2. Go to **Settings → Integrations**
3. Click **Connect** next to Prometheus
### Step 2: Choose Network Access Type
In the Prometheus integration form, choose one of the following:
* **Public**: Use a publicly reachable endpoint URL (for example, `http://prometheus.example.com:9090`)
* **Private (PrivateLink)**: Use AWS PrivateLink when Prometheus is only reachable inside your private network/VPC
### Step 3: Configure Connection Details
#### Public Connection
1. Enter the full **Endpoint URL**
2. Keep the endpoint accessible to TierZero
3. Continue to authentication setup
#### Private Connection (AWS PrivateLink)
If Prometheus is private, first create a PrivateLink connection:
1. Go to **Settings → Network**
2. Under **Private Connections**, click **Add Connection**
3. Add TierZero's AWS account principal to your VPC Endpoint Service's **Allowed principals**
4. Enter your **VPC Endpoint Service name** (format: `com.amazonaws.vpce..vpce-svc-...`)
5. Select the AWS region and create the connection
6. If your endpoint service requires acceptance, approve the connection request in AWS
7. Return to **Settings → Integrations → Prometheus**, choose **Private (PrivateLink)**, and select the connection
After selecting a private connection, TierZero uses the PrivateLink DNS endpoint and Prometheus port (typically `9090`).
### Step 4: Configure Authentication
Select the authentication mode used by your Prometheus endpoint:
* **No Authentication**: For endpoints without auth
* **Basic Auth**: Provide Base64-encoded `username:password`
* **Bearer Token**: Provide the token value
* **mTLS (Client Certificate)**: Provide client certificate and client private key in PEM format
### Step 5: Test and Connect
1. Click **Test Connection**
2. Resolve any validation errors shown in the form
3. Click **Connect**
## Network Requirements
* Prometheus HTTP API must be reachable from TierZero
* Default Prometheus port is `9090` (or your custom port)
* Protocol can be HTTP or HTTPS (HTTPS is recommended)
* For private VPC deployments, use **AWS PrivateLink** through **Settings → Network**
If your Prometheus endpoint is only accessible on internal networks, direct public connectivity will fail. Use AWS PrivateLink (or equivalent secure network routing) before connecting the integration.
## What TierZero Accesses
* **Prometheus metrics** via API queries
* **Time-series data** for investigations and comparisons
* **Metadata** needed to evaluate metric behavior during incidents
## Security
* Credentials are encrypted at rest
* TierZero uses provided credentials only for Prometheus queries
* Access can be revoked by rotating credentials, disabling the endpoint, or removing the integration
## Troubleshooting
### Connection Timeout
* Verify the endpoint URL/port is correct
* Confirm TierZero can reach the endpoint (public network or PrivateLink path)
* Check security groups, NACLs, firewalls, and endpoint service policies
### Authentication Errors
* Confirm the selected auth type matches your Prometheus deployment
* For Basic Auth, ensure you provide Base64-encoded `username:password`
* For mTLS, verify certificate/key pair validity and formatting
### TLS or Certificate Errors
* Verify TLS certificate chain and hostname
* If using private/internal certificates over PrivateLink, enable **Allow self-signed certificates** in Advanced Settings
# Rootly
Source: https://docs.tierzero.ai/integrations/rootly
Incident management and response platform
## Overview
Connect Rootly to enable TierZero to receive incident events and participate in your incident response workflow. TierZero can automatically investigate incidents reported through Rootly.
## Prerequisites
* Administrative access to your Rootly account
* Permission to configure webhook integrations
## Setup Instructions
### Step 1: Navigate to Integration Settings
1. Log into your TierZero dashboard
2. Go to **Settings → Integrations**
3. Click **Connect** next to Rootly
### Step 2: Configure Webhook
1. TierZero will display a webhook URL. Copy it
2. In Rootly, configure a webhook pointing to the TierZero URL
3. Enter the webhook secret provided by TierZero
4. Select the incident events to forward
### Step 3: Create a Webhook Subscription
1. Back in TierZero, enter the webhook details:
* **Name**: A descriptive name (e.g., "Rootly Production")
* **Remote ID**: The integration identifier from Rootly
* **Secret**: The webhook authentication secret
Configure webhook subscriptions from the TierZero dashboard to receive these events.
## What TierZero Accesses
* **Incident lifecycle events**: Creation, updates, status changes
* **Incident metadata**: Severity, affected services, responders
* **Dashboard content**: Rootly dashboard panels and data
## Using with Incident Agent
Rootly incidents can trigger the [Incident Agent](/features/incident-agent), which auto-joins the corresponding Slack channel and begins real-time investigation.
## Security
* Webhook signatures are validated on every incoming event
* Secrets are encrypted at rest
* Revoke by removing the webhook in Rootly
## Troubleshooting
### Incidents Not Triggering
* Verify the webhook URL is correct in Rootly
* Check that the webhook secret matches
* Ensure an Alert Agent or Incident Agent is configured to respond
# Sentry
Source: https://docs.tierzero.ai/integrations/sentry
Application error tracking and performance monitoring
## Overview
Connect Sentry to give TierZero access to your application errors, issues, and performance data. TierZero queries Sentry during investigations to identify error patterns, track issue frequency, and correlate errors with deployments.
## Prerequisites
* Administrative access to your Sentry organization
* Permission to create internal integrations or auth tokens
## Setup Instructions
### Step 1: Navigate to Integration Settings
1. Log into your TierZero dashboard
2. Go to **Settings → Integrations**
3. Click **Connect** next to Sentry
### Step 2: Authorize Sentry
1. Click **Connect to Sentry**
2. You'll be redirected to Sentry to authorize TierZero
3. Select the Sentry organization to connect
4. Review permissions and click **Allow**
5. You'll be redirected back to TierZero
## What TierZero Accesses
* **Issues**: Error groups, frequency, and affected users
* **Events**: Individual error occurrences and stack traces
* **Projects**: Project listings and configuration
* **Tags**: Issue tags for filtering and correlation
## Integration Resources
After connecting Sentry, TierZero uses these resource catalogs:
* **projects.json**: List of Sentry projects
* **issues\_tags.json**: Tags used for issue categorization
## Security
* TierZero uses Sentry's OAuth 2.0 authorization flow
* Read-only access to errors and project data
* Tokens are encrypted at rest
* Revoke access from Sentry's **Settings → Integrations → Custom Integrations**
## Troubleshooting
### Issues Not Visible
* Verify TierZero has access to the correct Sentry organization
* Check that the connected token has not expired
* Ensure the projects contain data for the queried time range
# Slack
Source: https://docs.tierzero.ai/integrations/slack
Team messaging and collaboration
## Overview
Slack is TierZero's primary communication channel. The integration enables TierZero to participate in incident channels, respond to questions, deliver investigation findings, and interact with your team directly where they work.
## Prerequisites
* Administrative access to your Slack workspace
* Permission to install Slack apps
## Setup Instructions
### Step 1: Navigate to Integration Settings
1. Log into your TierZero dashboard
2. Go to **Settings → Integrations**
3. Click **Connect** next to Slack
### Step 2: Install the Slack App
1. Click **Add to Slack**
2. You'll be redirected to Slack to authorize the installation
3. Review the permissions and click **Allow**
4. You'll be redirected back to TierZero
The integration should now show as connected.
## Permissions
The TierZero Slack App requests the following permissions:
### Communication & Responses
* **App mentions**: Respond to @TierZero mentions in channels
* **Send messages**: Post investigation findings and responses
* **Customize messages**: Respond with custom formatting
* **Slash commands**: Enable `/remember` to save context and preferences
### Channel Access
* **Join public channels**: Investigate incidents and gather context
* **Read public channels**: Access channel messages for investigation
* **Access channel history**: Gather historical context for investigations
* **Read and write to private channels**: Full functionality in private alert and incident channels
### Direct Messages & Groups
* **Read and write direct messages**: Interact with TierZero via DM
* **Read multi-party DMs**: Support group conversations with TierZero
### Context & Investigation
* **Read message metadata**: Gather context from message metadata
* **Read pinned messages**: Access pinned runbooks and important messages
* **Read and write files**: Download attachments for context, upload investigation findings
### Feedback & User Management
* **Read and add reactions**: Gather thumbs-up/down feedback, acknowledge requests with reactions
* **Read user information and groups**: Triage issues to the correct person or team
## What TierZero Uses Slack For
* **Incident Agent**: Auto-joins incident channels, posts real-time findings
* **Alert Agent**: Delivers investigation results to configured channels
* **Support Agent**: Monitors ask channels and answers questions
* **Digest Agent**: Posts scheduled reports to team channels
* **Copilot**: Interactive conversations via DM or in-channel mentions
## Security
* TierZero follows Slack's OAuth 2.0 authorization flow
* Access can be revoked at any time from your Slack workspace settings (**Settings → Manage Apps**)
* All communication is encrypted in transit
## Troubleshooting
### TierZero Not Responding in a Channel
* Ensure TierZero has been invited to the channel (for private channels)
* Check that no Slack trigger filters are excluding the channel
* Verify the Slack integration is still connected in **Settings → Integrations**
### Missing Message History
* TierZero can only access message history after being added to a channel
* For private channels, TierZero must be explicitly invited
# Snowflake
Source: https://docs.tierzero.ai/integrations/snowflake
Query and inspect your Snowflake data warehouse
## Overview
Connect Snowflake to give TierZero the ability to query your data warehouse during investigations. TierZero can run read-only SQL queries to analyze data, check pipeline health, and investigate data quality issues.
## Prerequisites
* Access to your Snowflake account
* Permission to create users or roles with read access
## Setup Instructions
### Step 1: Navigate to Integration Settings
1. Log into your TierZero dashboard
2. Go to **Settings → Integrations**
3. Click **Connect** next to Snowflake
### Step 2: Enter Connection Details
1. Enter your Snowflake **Account Identifier** (e.g., `xy12345.us-east-1`)
2. Enter a **Username** and **Password** (or key pair) for TierZero
3. Specify the **Warehouse**, **Database**, and **Schema** to use
4. Click **Connect**
We recommend creating a dedicated Snowflake user with read-only access for TierZero.
## What TierZero Accesses
* **SQL queries**: Read-only queries against your specified databases and schemas
* **Table metadata**: Schema information, column types, and row counts
* **Query history**: Recent query patterns for debugging
## Security
* TierZero only executes **read-only queries**; no INSERT, UPDATE, DELETE, or DDL operations are performed
* Credentials are encrypted at rest
* Scope access using Snowflake roles to control which databases and schemas TierZero can query
* Revoke by deleting or disabling the Snowflake user
## Troubleshooting
### Connection Fails
* Verify the account identifier is correct (include the region suffix)
* Check that the username and password are valid
* Ensure the specified warehouse exists and is not suspended
* Confirm the user has access to the specified database and schema
# Temporal
Source: https://docs.tierzero.ai/integrations/temporal
Workflow orchestration and task queue platform
## Overview
Connect Temporal to give TierZero access to your workflow execution data. TierZero uses Temporal to investigate workflow failures, analyze task queue health, and correlate workflow issues with other system events.
## Prerequisites
* Access to your Temporal cluster (Cloud or self-hosted)
* Permission to create API keys or namespace access
## Setup Instructions
### Step 1: Navigate to Integration Settings
1. Log into your TierZero dashboard
2. Go to **Settings → Integrations**
3. Click **Connect** next to Temporal
### Step 2: Enter Connection Details
1. Enter your Temporal **server address** (e.g., `your-namespace.tmprl.cloud:7233`)
2. Provide authentication credentials (API key or mTLS certificates)
3. Specify the namespace(s) to monitor
4. Click **Connect**
## What TierZero Accesses
* **Namespaces**: Namespace configurations and metadata
* **Workflows**: Workflow execution history, status, and results
* **Activities**: Activity task results and failures
* **Task queues**: Queue health and worker status
## Integration Resources
After connecting Temporal, TierZero catalogs your namespaces for use during investigations and workflow analysis.
## Security
* TierZero uses API key or mTLS authentication
* Read-only access to workflow data
* Credentials are encrypted at rest
* Revoke by deleting the API key or revoking the certificate
## Troubleshooting
### Connection Timeout
* Verify the server address is correct and reachable
* For Temporal Cloud, ensure the namespace is active
* Check that the API key or certificate has not expired
* Confirm the namespace name is correct
# Instructions
Source: https://docs.tierzero.ai/knowledge/instructions
Configure TierZero.md and team-specific instructions for TierZero
## Overview
Instructions tell TierZero how to reason about your systems, ownership model, and operating conventions.
TierZero supports two instruction scopes:
* **TierZero.md**: Global guidance that applies across the organization
* **Team instructions**: Team-specific guidance that applies when TierZero is working in that team's context
Together, these give TierZero the baseline context it should always follow plus the local nuances that matter for a specific team.
## TierZero.md
`TierZero.md` is the main instruction file TierZero follows when handling requests across your organization.
### What to Include
A well-structured `TierZero.md` should include sections like:
**Company**
High-level description of what your company does and the products you offer. This helps TierZero understand the business context of issues.
**Customer Blast Radius**
How to correlate logs, traces, metrics, and other observability data with:
* The set of customers affected by an issue
* The subset of environments affected, such as sharded infrastructure by region or cell
**Environment**
How to identify the production environment in logs, tags, traces, and infrastructure naming. Include a brief description of your deployment setup.
**Infrastructure as Code**
Whether infrastructure is deployed as code, where it is defined, and whether there are important production resources not defined in code.
**Code Release and Deployment**
How services are released, how infrastructure changes roll out, and whether non-deployment feature releases such as feature flags are used.
### Sample `TierZero.md`
```markdown theme={null}
### Company
Acme Corp is a SaaS platform that provides project management and collaboration tools for distributed teams. Our core products are:
- **Acme Projects**: Task and project management
- **Acme Docs**: Real-time collaborative documents
- **Acme Chat**: Team messaging and video calls
### Customer Blast Radius
Customers are identified by `org_id` in logs and traces. We use regional sharding:
| Tag/Attribute | Data Source | Description |
|---------------|-------------|-------------|
| `org_id` | Logs, Traces | Customer organization identifier |
| `region` | Logs, Metrics | Geographic region (us-east, eu-west, ap-south) |
| `cell_id` | Logs, Traces | Infrastructure cell within a region |
| `tier` | Logs | Customer tier (free, pro, enterprise) |
Enterprise customers (tier=enterprise) are on dedicated cells. Free/Pro customers share multi-tenant cells.
### Environment
Production runs on AWS EKS across three regions:
- `us-east-1` (primary)
- `eu-west-1` (EU customers)
- `ap-south-1` (APAC customers)
Identify production in observability data:
- Logs: `env:prod*` or `environment:production`
- Kubernetes namespace: `prod-*`
- Database instances: `prod-aurora-*`
### Infrastructure as Code
All infrastructure is defined in Terraform:
- Repository: `acme/infrastructure`
- Directory: `terraform/environments/prod/`
Exceptions not in IaC:
- DNS records managed in Cloudflare UI
- Some legacy Lambda functions in `acme/legacy-functions`
### Code Release and Deployment
- Services deploy via GitHub Actions -> ArgoCD
- Deployments roll out via canary (10% -> 50% -> 100%)
- Feature flags managed in LaunchDarkly
- Flag naming: `feature..`
- Query flags: `source:launchdarkly @flag.key:*`
```
## Team Instructions
Team instructions let you tailor TierZero's behavior for individual teams. Use them when different teams have:
* **Different services and ownership boundaries**
* **Different escalation paths**
* **Different runbook conventions**
* **Different tool preferences**
### How to Set Up Team Instructions
1. Open **Instructions** from the left navigation
2. In the sidebar, click `.md`
3. Edit the markdown for that team
4. Click **Save**
The instructions take effect immediately for all agents associated with that team.
### What to Include
**Service Ownership**
```markdown theme={null}
## Services We Own
- **payment-service**: Handles all payment processing (Go, deployed on EKS)
- **billing-api**: Subscription management and invoicing (Python, deployed on EKS)
- **payment-gateway**: External payment provider integration (Go)
## Key Dependencies
- PostgreSQL (RDS) for transaction data
- Redis for idempotency keys
- Stripe API for payment processing
```
**Escalation Procedures**
```markdown theme={null}
## Escalation
- SEV1: Page @payments-oncall immediately, notify #payments-incidents
- SEV2: Post in #payments-alerts, oncall will triage within 15 minutes
- SEV3: Create a Jira ticket in the PAYMENTS project
```
**Team-Specific Debugging**
```markdown theme={null}
## Common Issues
- Payment timeouts are usually caused by Stripe API latency — check Stripe status page first
- If billing-api returns 500s, check the subscription_events table for stuck records
- Redis connection errors usually resolve with a pod restart — check if the Redis cluster is healthy first
```
**Dashboard and Runbook References**
```markdown theme={null}
## Key Dashboards
- Payment Processing: https://app.datadoghq.com/dashboard/abc-123
- Stripe Integration Health: https://app.datadoghq.com/dashboard/def-456
## Runbooks
- Payment Failover: https://notion.so/team/payment-failover-runbook
- Stripe Outage Response: https://notion.so/team/stripe-outage-playbook
```
## TierZero.md vs Team Instructions
| | TierZero.md | Team Instructions |
| ---------------- | ----------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- |
| **Scope** | Entire organization | Single team |
| **Content** | Architecture overview, global conventions, org-wide context | Team-specific services, escalation paths, debugging tips |
| **When applied** | Every investigation | Requests that pertain to the team's ownership, services, or domain — TierZero determines relevance based on the team's description |
| **Who edits** | Admins | Team members |
## Best Practices
**1. Keep It Complementary**
* Avoid duplicating the same material in both places
* Put global conventions in `TierZero.md` and local nuance in team instructions
**2. Include Real Examples**
* Mention specific service names, dashboard URLs, Slack channels, and identifiers
* Real references help TierZero navigate your systems accurately
**3. Update After Incidents**
* After a significant incident, add new debugging learnings and escalation notes
* This builds institutional knowledge over time
**4. Keep It Focused**
* Include the most important instructions and identifiers
* Reference long runbooks or dashboards instead of copying them wholesale
# Skills
Source: https://docs.tierzero.ai/knowledge/skills
Create and manage skills for TierZero
## Overview
Skills are reusable instruction bundles that extend how TierZero handles specific kinds of work. The new **Skills** workspace lets you create and manage skills for your organization.
## Where to Find It
Open **Skills** from the left navigation.
You need the `read:skills` permission to view the page and `write:skills` to create, edit, or delete skills.
## Skill Types
TierZero supports two kinds of skills:
* **Managed in TierZero**: You author and edit the `SKILL.md` content directly in TierZero
* **Sync from GitHub**: TierZero reads a skill from a directory in a GitHub repository
## Managed Skills
Managed skills are best when you want to create or maintain the skill entirely inside TierZero.
When you create a managed skill:
1. Click **Add Skill**
2. Choose **Manage in TierZero**
3. Enter a name and description
4. TierZero creates a starter `SKILL.md`
5. Edit the raw markdown and save
Managed skills are edited as full `SKILL.md` files, including frontmatter.
### Required Format
A managed skill must be valid `SKILL.md` content and must start with frontmatter like:
```markdown theme={null}
---
name: debug-ooms
description: Use when a service or job is experiencing out-of-memory issues and you need to identify the cause, verify whether memory growth is expected, and suggest the safest fix.
---
## When to Use
Invoke this skill when:
- A pod or process is being OOMKilled
- Memory usage is growing unexpectedly
- A deployment started hitting memory limits after a release
## Investigation Steps
1. Confirm which workload is OOMing and when it started
2. Compare current memory usage against recent deploys or traffic changes
3. Check whether the spike is caused by expected load, a memory leak, or an undersized memory limit
4. Identify the safest mitigation, such as rollback, memory limit adjustment, or code-level fixes
## Common Pitfalls
- Short-lived spikes can look like leaks if you only inspect a narrow time window
- Raising memory limits without checking request volume can hide the real issue
```
At minimum:
* The file must start with `---`
* The frontmatter must have a closing `---`
* The frontmatter must contain both `name` and `description`
When you save a managed skill, TierZero re-validates the file and re-extracts the skill name and description from the frontmatter.
## GitHub-Synced Skills
GitHub-synced skills are best when you already keep skills in your codebase and want TierZero to reference them from there.
When you create a GitHub skill:
1. Click **Add Skill**
2. Choose **Sync from GitHub**
3. Select a connected GitHub organization
4. Select a repository
5. Enter the directory path that contains the skill
6. Review the detected name and description
7. Save the skill
### GitHub Requirements
* You must have an approved GitHub integration connected in TierZero
* The selected directory must exist in the repository
* The directory must contain a `SKILL.md` file
* That `SKILL.md` must contain valid frontmatter with `name` and `description`
### Name and Description Overrides
For GitHub-synced skills, the UI lets you override the displayed name and description in TierZero.
Those overrides are used when the skill is loaded by TierZero, even though the source files come from GitHub.
## What to Put in a Skill
A good skill should teach TierZero how to handle a repeatable class of work.
Useful examples include:
* Step-by-step debugging playbooks for a recurring failure mode
* Deployment or rollback procedures
* Domain-specific investigation guidance
* Safe operating rules for sensitive systems
* How to interpret provider-specific or internal resource names
Skills should be more focused than your main [Instructions](/knowledge/instructions) page. Put broad org or team context in Instructions, and put task-specific guidance in a skill.
## Editing and Deleting
* **Managed skills**: Edit the raw `SKILL.md` content directly in the editor
* **GitHub skills**: Edit the name, description, GitHub org, repository, or directory path in the form view
* **Delete**: Remove a skill from the Skills workspace
For GitHub skills, changing the repository, directory path, or GitHub integration triggers re-validation.
## Best Practices
**1. Keep each skill narrow**
* One skill should cover one workflow or one operational domain
* Split large skills instead of turning them into general-purpose manuals
**2. Make the description precise**
* The description helps TierZero decide when the skill applies
* Write it as a clear statement of when the skill should be used
**3. Prefer procedures over reference dumps**
* Put the critical workflow, decision points, and caveats in the skill
* Link out or summarize long reference material instead of copying everything in
# Digest Prompts
Source: https://docs.tierzero.ai/prompt-library/digest-prompts
Example prompts for scheduled digest agent reports
## Example Digest Prompts
The following are example digest prompts that can be used for reporting. Modify them to work with your specific services, pipelines, and monitoring setup.
### Daily Service Anomaly Report
**Problem Statement:** Proactively detect emerging issues in production services before they escalate into customer-impacting incidents.
**Prompt:**
```markdown theme={null}
Execute the following runbook for service 'api-server':
1. Analyze both errors and warnings for the last 24 hours, compare that with the previous 3 days (i.e. from 4 days ago to 1 day ago), do you see any new issues emerging?
2. For any new issue identified (pick the top 5 if many), find the potential root cause by
a. Querying sample traces if available
b. Examining any recent code or config changes made to the relevant files
3. Generate a report by deep diving into each new issue. The report should cover:
a. Severity
b. Number of occurrences
c. Example error message or trace
d. Impact analysis
e. Potential root cause
f. Related PRs if any
g. Recommended next steps
```
### Weekly CI Pipeline Health Report
**Problem Statement:** Track CI/CD pipeline reliability and identify flaky tests or infrastructure issues that slow down development velocity.
**Prompt:**
```markdown theme={null}
For the 'production-release' pipeline over the most recent whole week Monday 00:00:00 PDT to Sunday 23:59:59 PDT
- Find all failed builds on the 'main' branch, record the build numbers
- For each failed build identified, query the failed jobs and their logs to analyze the failure reason
- Summarize your findings in the following report
Report: Top CI Failures
Generate a table in markdown in the following format:
CI Job Name | Failure Count (# of Builds) | Example Link | List of Build Numbers
... | ... | ... | ...
... | ... | ... | ...
List the top 10 frequently failed jobs in the table.
```
### Log Volume Analysis for Cost Optimization
**Problem Statement:** Identify opportunities to reduce observability costs by analyzing log volume patterns.
**Prompt:**
```markdown theme={null}
I'm conducting a logs analysis to see what is contributing the most to log volume over the last 2 days. Follow the instructions and generate a report for me.
1. Determine the top 10 services for logs by volume over the past 2 days.
- Query logs grouped by service.
- Use the timeframe for the last 2 full days.
2. For each of the top 10 services, determine the top 10 log messages by volume over the past 2 days.
- Query logs for each service grouped by message.
- Use the timeframe for the last 2 full days.
Generate a report explaining what is contributing the most to log volume. In particular, format the final report as follows (including a table)
### Indexed Log Volume Analysis
#### Recommendations
[2-3 bullet points with recommendations for reducing indexed log volume]
#### Top 20 Log Messages by Indexed Log Volume Count
| Service | Message | Daily Indexed Log Volume Count |
| ... | ... | ... |
| ... | ... | ... |
(Table should be sorted by log volume count and include the top 20 messages with highest log volume count. Divide the 2-day log volume count by 2 to get the daily log volume count.)
```
### Weekly Infrastructure Cost Report
**Problem Statement:** Track infrastructure spend and identify cost optimization opportunities.
**Prompt:**
```markdown theme={null}
Generate a weekly infrastructure cost report covering the past 7 days:
1. Query cloud cost data to get total spend by service/resource type.
2. Compare with the previous week to identify any significant increases (>10%).
3. Identify the top 5 cost drivers and any anomalies.
4. Look for optimization opportunities:
- Idle or underutilized resources
- Resources that could be right-sized
- Reserved instance coverage gaps
Format the report as:
### Weekly Cost Summary
- Total spend: $X
- Week-over-week change: +/-X%
### Top Cost Drivers
| Resource | Cost | Change |
| ... | ... | ... |
### Optimization Recommendations
[Bullet points with specific recommendations]
```
# Investigation Runbooks
Source: https://docs.tierzero.ai/prompt-library/investigation-runbooks
Example runbooks for automated alert investigation
## Example Investigation Runbooks
The following are example runbooks that can be used for alert investigations. Each runbook guides TierZero through root cause analysis, impact assessment, and recommended next steps. Modify them to work with your specific services, monitoring setup, and investigation workflows.
### High 500 Errors for API Requests
**Alert Scenario:** API requests are returning 500 errors, indicating server-side failures that could be affecting users.
**Runbook:**
```markdown theme={null}
1. Determine user and account impact.
- Query spans to identify how many unique users and accounts were affected.
- Note the percentage of total active users if available.
- Check if enterprise or high-value customers are impacted.
- Enterprise customer list: https://acme.atlassian.net/wiki/spaces/CS/pages/333444/Enterprise+Customer+Directory
2. Sample error traces for root cause analysis.
- Query at least 5 trace IDs corresponding to the 500 errors.
- For each trace, examine the error spans to identify the failing service and error message.
3. Check for related code changes.
- If the error contains a version or commit hash, look for commits in the past 3 days that could be related.
- Reference: https://acme.atlassian.net/wiki/spaces/ENG/pages/123456/Deployment+History
4. Review the service dashboard for anomalies.
- Dashboard: https://app.datadoghq.com/dashboard/abc-123/api-server-health
```
### Kubernetes Pod in CrashLoopBackOff
**Alert Scenario:** Kubernetes pods are repeatedly crashing and restarting, indicating application-level failures or resource constraints.
**Runbook:**
```markdown theme={null}
1. Query Kubernetes events for the affected pods.
- Use the cluster and namespace from the alert context.
- Look for OOMKilled, FailedScheduling, or other error events.
2. Assess blast radius.
- Determine how many pods and replicas are affected.
- Check if the issue is isolated to a single node or spread across the cluster.
- Identify which services and endpoints are degraded as a result.
3. Check application logs for errors.
- Query logs for the affected pods during the alert time range.
- Start with error-level logs, then check info-level if no errors found.
4. Review resource metrics.
- Check CPU throttling and memory utilization for the affected pods.
- Check node health if pod issues are widespread.
- Dashboard: https://app.datadoghq.com/dashboard/xyz-789/kubernetes-cluster-health
5. Reference the K8s troubleshooting guide.
- Runbook: https://acme.atlassian.net/wiki/spaces/SRE/pages/789012/K8s+Troubleshooting
```
### High Database CPU Utilization
**Alert Scenario:** Database cluster is experiencing high CPU utilization, which could indicate inefficient queries, increased traffic, or resource constraints.
**Runbook:**
```markdown theme={null}
1. Query database metrics to understand CPU utilization levels.
- Get the DB cluster identifier from the alert.
- Check the trend over the past hour to understand if this is a spike or gradual increase.
2. Determine downstream impact.
- Query errors and latency for services that depend on this database.
- Count unique affected users and accounts from dependent service spans.
- Check if the issue is impacting a specific region or globally.
- Regional dashboard: https://app.datadoghq.com/dashboard/geo-001/regional-health
3. Identify calling applications and slow queries.
- Query spans to find which applications are making the most database calls.
- Look for queries with high average duration.
4. Check for traffic changes.
- Compare current traffic volume to the previous period.
- Look for any unusual patterns or traffic sources.
5. Review database dashboard and documentation.
- Dashboard: https://app.datadoghq.com/dashboard/def-456/rds-performance
- Scaling guide: https://acme.atlassian.net/wiki/spaces/INFRA/pages/345678/Database+Scaling+Procedures
```
### Service Latency Degradation
**Alert Scenario:** Service latency has increased beyond acceptable thresholds.
**Runbook:**
```markdown theme={null}
1. Identify the scope of latency degradation.
- Check if latency is elevated across all endpoints or specific routes.
- Determine if the issue is regional or global by checking error distribution by region.
- If isolated to one region: note for targeted investigation.
- If global: escalate as widespread outage.
2. Assess user impact.
- Query to count unique affected users and accounts experiencing elevated latency.
- Check if enterprise customers are disproportionately affected.
- Enterprise customer list: https://acme.atlassian.net/wiki/spaces/CS/pages/333444/Enterprise+Customer+Directory
3. Trace slow requests.
- Sample traces with high duration to identify bottlenecks.
- Look for slow spans in downstream services or databases.
4. Check for recent changes.
- Review recent deployments to the affected service.
- Check for config changes or feature flag updates.
- Deployment log: https://acme.atlassian.net/wiki/spaces/ENG/pages/567890/Recent+Deployments
5. Review service dependencies.
- Check health of upstream and downstream services.
- Service dependency map: https://app.datadoghq.com/apm/service-map
```
# Support Agent Instructions
Source: https://docs.tierzero.ai/prompt-library/support-agent-instructions
Example instructions for configuring support agent behavior
## Example Support Agent Instructions
Support agent instructions guide how TierZero responds to questions in your Slack channels. These instructions help the agent understand what types of questions to answer, how to investigate, and how to format responses.
### Platform Team Support (#ask-infra)
**Use Case:** Product engineering teams asking the platform team questions about infrastructure, permissions, and Kubernetes.
**Instructions:**
```markdown theme={null}
### Request Types and Response Guidelines
#### IAM Access Permissions
For requests involving AWS IAM roles, permissions, or access issues:
**Pre-requisites (ask user if not provided):**
- {{ aws_account }}: The AWS account (e.g., production, staging)
- {{ service_or_resource }}: The service or resource they need access to
**Steps:**
1. Search the terraform-iam repository for the role definition matching {{ service_or_resource }}
2. Query AWS IAM to check the user's current role assignments in {{ aws_account }}
3. If a permission change is needed, link to the access request template and explain next steps
4. Ask user to submit an access request if elevated permissions are required
**Relevant Resources:**
- Access request template: https://acme.atlassian.net/wiki/spaces/INFRA/pages/123456/Access+Request+Template
- IAM roles documentation: https://acme.atlassian.net/wiki/spaces/INFRA/pages/789012/IAM+Roles+Guide
#### Kubernetes Troubleshooting
For requests involving pod failures, deployment issues, or cluster problems:
**Pre-requisites (ask user if not provided):**
- {{ cluster }}: The Kubernetes cluster name
- {{ namespace }}: The namespace where the issue is occurring
- {{ pod_or_deployment }}: The pod or deployment name experiencing issues
**Steps:**
1. Query Kubernetes events for {{ namespace }} in {{ cluster }} to identify recent errors
2. Query pod logs for {{ pod_or_deployment }} to find error messages
3. Check CPU and memory utilization for the affected workloads
4. If OOMKilled, recommend increasing memory limits and link to resource configuration docs
5. Ask user to run: `kubectl describe pod {{ pod_or_deployment }} -n {{ namespace }}` for additional details if needed
**Relevant Resources:**
- K8s troubleshooting runbook: https://acme.atlassian.net/wiki/spaces/SRE/pages/345678/K8s+Troubleshooting
- Cluster dashboard: https://app.datadoghq.com/dashboard/k8s-001/cluster-health
### For All Other Types of Requests
Do not respond.
```
### Engineering Support (#ask-eng)
**Use Case:** Support and sales teams asking engineering questions about feature releases and product capabilities.
**Instructions:**
```markdown theme={null}
### Request Types and Response Guidelines
#### Feature Release Timeline
For requests about when features shipped or will ship:
**Pre-requisites (ask user if not provided):**
- {{ feature_name }}: The name or description of the feature
**Steps:**
1. Search release notes and changelog for {{ feature_name }} to find if it has already shipped
2. If shipped, provide the release version and date
3. If not shipped, check the product roadmap in Notion for planned timeline
4. Search GitHub PRs for {{ feature_name }} to find current development status
5. Provide the current status and expected timeline if available
**Relevant Resources:**
- Release notes: https://acme.atlassian.net/wiki/spaces/PRODUCT/pages/111222/Release+Notes
- Product roadmap: https://www.notion.so/acme/Product-Roadmap-abc123
### For All Other Types of Requests
Do not respond.
```
# Claude Code
Source: https://docs.tierzero.ai/references/mcp/claude-code
Install the TierZero plugin in Claude Code and authenticate via OAuth.
The TierZero plugin for Claude Code wraps the [MCP server](/references/mcp/overview) with a `/tierzero` slash command and a skill that auto-loads when production context would help. Authentication is OAuth — there's nothing to copy or paste.
## Install
1. Add the TierZero marketplace:
```
/plugin marketplace add TierZeroAI/tierzero-claude-code
```
2. Install the plugin:
```
/plugin install tierzero@tierzero
```
3. Run any `/tierzero` command. The first time, Claude Code prompts you to authenticate — run `/mcp` and approve the connection in your browser.
```
/tierzero What errors occurred in payment-service in the last hour?
```
![Claude Code authenticating the TierZero MCP server and fetching an investigation]()
## Project-wide installation
To enable the TierZero plugin for your whole team, add this to your project's `.claude/settings.json` and commit it:
```json theme={null}
{
"extraKnownMarketplaces": {
"tierzero": {
"source": {
"source": "github",
"repo": "TierZeroAI/tierzero-claude-code"
}
}
},
"enabledPlugins": {
"tierzero@tierzero": true
}
}
```
Anyone who clones the project will have the plugin available; each user authenticates individually on first use.
## Usage
With the plugin installed, Claude Code automatically loads the `tierzero` skill when your task would benefit from production context — it can query TierZero in the background while it continues working on your code.
You can also explicitly start an investigation:
```text theme={null}
/tierzero What errors occurred in payment-service in the last hour?
```
Or pull existing context from a TierZero chat, investigation, or digest:
```text theme={null}
/tierzero:fetch-context https://app.tierzero.ai/chat/c/
```
See the [overview](/references/mcp/overview) for a full description of `tierzero_ask` and `tierzero_fetch_context`.
# Cursor
Source: https://docs.tierzero.ai/references/mcp/cursor
Install TierZero from the Cursor MCP marketplace and authenticate via OAuth.
Cursor publishes TierZero in its MCP marketplace. Install once and Cursor agents can call the [TierZero MCP tools](/references/mcp/overview) directly. Authentication is OAuth — no tokens to copy.
## Install
1. Open the [TierZero listing on the Cursor marketplace](https://cursor.com/marketplace/tierzero).
2. Click **Install** — Cursor adds the `tierzero` server to your MCP config.
3. Use a TierZero-backed action from any agent. Cursor prompts **Connect with tierzero** with an **Authenticate** button. Click it and complete the OAuth flow in your browser.
![Cursor agent calling /tierzero-fetch and prompting to authenticate the TierZero MCP server]()
Once connected, Cursor agents can call `tierzero_ask` and `tierzero_fetch_context` whenever production context is relevant. The first call after a fresh install prompts for authentication; subsequent calls use the stored token.
## Usage
Ask the Cursor agent something that would benefit from production context — it will call `tierzero_ask` automatically. To pull a specific TierZero chat, investigation, or digest into the conversation, paste its URL and ask the agent to fetch context.
See the [overview](/references/mcp/overview) for what each tool does and the arguments it accepts.
# Other MCP clients
Source: https://docs.tierzero.ai/references/mcp/other
Connect any MCP-compliant client to the TierZero MCP server using OAuth or a Personal Access Token.
Any MCP client that supports remote HTTP transport can connect to TierZero. Pick whichever auth method your client supports.
## Server details
* **URL:** `https://api.tierzero.ai/mcp/`
* **Transport:** Streamable HTTP
* **Auth:** OAuth 2.1 *or* Personal Access Token
## Option 1: OAuth (recommended)
If your client supports OAuth 2.1 with dynamic client registration, point it at `https://api.tierzero.ai/mcp/` and let it auto-discover the authorization server via `/.well-known/oauth-protected-resource`. The client will handle the browser-based login flow on first use.
This is the same flow Claude Code and Cursor use. No tokens to manage, and access is tied to the user that approved the connection.
## Option 2: Personal Access Token
Use a PAT for clients that don't support OAuth, for headless setups (CI, scripts), or when you want long-lived auth tied to a specific user.
### Generate a token
1. Go to [auth.tierzero.ai/account/api\_keys/](https://auth.tierzero.ai/account/api_keys/)
2. Create a new personal API key
3. Save it somewhere your client can read it, for example as an environment variable:
```bash theme={null}
export TIERZERO_PAT_TOKEN=""
```
Treat the PAT like a password. It grants access to your TierZero data. Never commit it to version control or paste it into shared chat.
### Configure your client
Send the PAT in the `Authorization` header on every request to `https://api.tierzero.ai/mcp/`:
```
Authorization: Bearer
```
The exact config field depends on the client. As an example, for Claude Code's manual MCP config:
```bash theme={null}
claude mcp add tierzero-server https://api.tierzero.ai/mcp/ \
--transport http \
--header "Authorization: Bearer ${TIERZERO_PAT_TOKEN}"
```
Or in a JSON config like Cursor's:
```json theme={null}
{
"mcpServers": {
"tierzero-server": {
"type": "http",
"url": "https://api.tierzero.ai/mcp/",
"headers": {
"Authorization": "Bearer ${env:TIERZERO_PAT_TOKEN}"
}
}
}
}
```
### Rotating or revoking a PAT
Generate a new token at the [API keys page](https://auth.tierzero.ai/account/api_keys/) and update your client config. Revoke the old one from the same page. There is no automatic expiry — tokens stay valid until you revoke them.
# Overview
Source: https://docs.tierzero.ai/references/mcp/overview
Connect TierZero as a remote MCP server to MCP-compliant clients such as Claude Code, Cursor, or any HTTP/OAuth-aware client.
## What it is
TierZero exposes a remote Model Context Protocol (MCP) server. AI coding assistants — Claude Code, Cursor, and any other MCP-compliant client — can call it directly to query your production context while you work.
* **Endpoint:** `https://api.tierzero.ai/mcp/`
* **Transport:** Streamable HTTP
* **Auth:** OAuth 2.1 (recommended) or Personal Access Token
## What it does
The server exposes two tools.
### `tierzero_ask`
Runs a TierZero investigation against your connected production systems and returns evidence-backed findings. It can:
* Query logs, metrics, traces, and spans from connected integrations such as Datadog, New Relic, CloudWatch, Grafana Cloud, and Sentry
* Inspect GitHub repositories, CI/CD systems such as BuildKite and GitHub Actions, and workflow orchestration in Temporal
* Retrieve documentation and runbooks from connected knowledge sources such as Confluence, Notion, and Slack
* Correlate findings across telemetry, code, workflows, and documentation
![Claude Code automatically loading the TierZero skill and running a TierZero production context query in the background]()
`tierzero_ask` invokes the TierZero agent, which may take several minutes for non-trivial queries. Clients that support MCP progress notifications will display progress updates during execution.
### `tierzero_fetch_context`
Loads an existing TierZero chat, investigation, digest, or artifact into your IDE so you can act on it without starting a new investigation.
![Claude Code fetching existing TierZero chat context and using it to answer a follow-up question]()
Arguments:
* `url`: A TierZero chat URL (`https://app.tierzero.ai/chat/c/`), investigation URL (`https://app.tierzero.ai/investigations/`), digest run URL (`https://app.tierzero.ai/digests/run/`), or raw GlobalID
* `include_sources`: Optional boolean. Set to `true` to include source metadata in the response
## Choosing a client
Pick the setup guide that matches your client:
Plugin install + OAuth. Recommended for Claude Code users.
Cursor marketplace install + OAuth.
Any HTTP MCP client. OAuth 2.1 or PAT.
## Troubleshooting
### Authentication didn't pop up
Run `/mcp` in Claude Code, or open the MCP panel in Cursor settings, and click **Authenticate** on the `tierzero` server.
### Token expired or revoked
Reconnect from the same place: `/mcp` in Claude Code, or the Tools & MCP panel in Cursor. The client will start a fresh OAuth flow. If you're using a PAT, generate a new one at [auth.tierzero.ai/account/api\_keys/](https://auth.tierzero.ai/account/api_keys/) and update your client config.
### Connection issues
Confirm `https://api.tierzero.ai/mcp/` is reachable from your network and that your client supports HTTP transport. Older MCP clients that only support stdio transport are not compatible with the remote server.
# Terraform Provider
Source: https://docs.tierzero.ai/references/terraform-provider
Manage TierZero alert agents with Terraform
## Overview
TierZero enables customers to manage alert agents using Terraform, allowing you to define and version control your incident response automation as infrastructure-as-code.
## Documentation
The TierZero Terraform Provider is published on the Terraform Registry. For the latest version and complete documentation, visit:
[TierZero Terraform Provider Documentation](https://registry.terraform.io/providers/TierZeroAI/tierzero/latest/docs)
# API Reference
Source: https://docs.tierzero.ai/references/tz_api
Use TierZero's REST API to programmatically ask questions and integrate TierZero insights into your applications.
## Overview
The TierZero API allows you to programmatically ask questions to the TierZero AI agent and receive structured responses. This enables you to integrate TierZero's intelligent analysis capabilities into your applications, automation workflows, and custom tools.
The API uses standard HTTP methods with JSON payloads and requires authentication via organization API keys. All API responses are formatted in JSON and include relevant interaction IDs for tracking and follow-up.
## Authentication
All API requests require authentication using an organization API key. Include your API key in the request header:
```
X-TierZero-Org-Api-Key: your-api-key-here
```
API keys can be generated from your TierZero dashboard. Each key is associated with a specific organization and has access to that organization's resources and context.
## Resource Identifiers
The TierZero API uses **Global IDs** to identify resources like digests, alert agents, teams, and notification integrations. Global IDs are opaque string identifiers that you receive in API responses and use in subsequent requests.
**Example Global IDs:**
```json theme={null}
{
"digest_id": "R3JhcGhRTEpvYjoxMjM=",
"alert_id": "R3JhcGhRTEFsZXJ0OjQ1Ng==",
"notification_integration_id": "R3JhcGhRTE5vdGlmaWNhdGlvbkludGVncmF0aW9uOjEyMzQ="
}
```
## Endpoints
### Alert Agent Management
#### Create Alert Agent (`POST /api/v1/alert-responders`)
Create a new alert agent that automatically investigates incoming alerts.
##### Request
**Method:** `POST`
**Path:** `/api/v1/alert-responders`
**Content-Type:** `application/json`
**Headers:**
* `X-TierZero-Org-Api-Key` (required): Your organization API key
* `Content-Type`: `application/json`
**Request Body:**
For **webhook-based alerts** (PagerDuty, OpsGenie, FireHydrant, Rootly):
```json theme={null}
{
"team_name": "string", // Required: Team name
"name": "string", // Required: Alert agent name
"webhook_sources": [ // Required: Webhook sources to monitor (see "List Webhook Subscriptions" below)
{
"type": "string", // Webhook type: PAGERDUTY, OPSGENIE, FIREHYDRANT, or ROOTLY
"remote_id": "string" // External webhook ID (use "List Webhook Subscriptions" to find valid remote_id values)
}
],
"matching_criteria": { // Required: Criteria for matching alerts (provide text_matches OR match_rules)
"text_matches": ["string"], // Option 1: Array of wildcard patterns (see "Matching Criteria" below)
"match_rules": { // Option 2: Structured rules with AND/OR logic (see "Matching Criteria" below)
"operator": "AND",
"conditions": [...]
}
},
"runbook": { // Optional: Investigation runbook (uses default if not provided)
"investigation_prompt": "string" // Optional: Main investigation prompt
},
"notification_integration_ids": [ // Optional: Notification integration Global IDs (use "List Notification Integrations" below)
"string"
]
}
```
For **Slack alerts**:
```json theme={null}
{
"team_name": "string", // Required: Team name
"name": "string", // Required: Alert agent name
"slack_channel_id": "string", // Required: Slack channel ID (must start with 'C' for public channels or 'G' for private channels)
"matching_criteria": { // Required: Criteria for matching alerts (provide text_matches OR match_rules)
"text_matches": ["string"], // Option 1: Array of wildcard patterns (see "Matching Criteria" below)
"match_rules": { // Option 2: Structured rules with AND/OR logic (see "Matching Criteria" below)
"operator": "AND",
"conditions": [...]
},
"slack_bot_app_user_id": "string" // Optional: Filter by specific bot/sender ID (e.g., "B01234567")
},
"runbook": { // Optional: Investigation runbook (uses default if not provided)
"investigation_prompt": "string" // Optional: Main investigation prompt
},
"notification_integration_ids": [ // Optional: Notification integration Global IDs (use "List Notification Integrations" below)
"string"
]
}
```
**Note:** If `runbook` or `runbook.investigation_prompt` is not provided, a default investigation prompt will be used: *"Please investigate the issue and explain the root cause to the best of your abilities!"*
**Idempotency:** If an alert agent with the same name exists for the organization and team, returns the existing responder instead of creating a duplicate.
##### Matching Criteria
The `matching_criteria` field controls which alerts trigger this agent. You can provide **either** `text_matches` or `match_rules` — the API will auto-generate the other format and return both in the response.
**Option 1: `text_matches`** — Simple wildcard patterns (recommended for Terraform/Pulumi)
A flat list of patterns using `*` as a wildcard. All patterns must match (implicit AND).
| Pattern | Meaning | Example matches |
| ------------ | ---------------------- | ---------------------------- |
| `*stg/*` | Contains `stg/` | `"deploy stg/api failed"` |
| `release/*` | Starts with `release/` | `"release/v2.1 rollback"` |
| `*-critical` | Ends with `-critical` | `"db-critical"` |
| `*error*` | Contains `error` | `"connection error timeout"` |
| `*foo*bar*` | Regex: `foo.*bar` | `"foo-123-bar"` |
```json theme={null}
"matching_criteria": {
"text_matches": ["*stg/*", "*release/*"]
}
```
**Option 2: `match_rules`** — Structured rules with AND/OR logic
A tree of conditions supporting `AND`/`OR` operators and typed conditions. Max nesting depth of 2.
Condition types:
* `contains` — substring match (case-insensitive)
* `starts_with` — prefix match
* `ends_with` — suffix match
* `regex` — regular expression match
```json theme={null}
"matching_criteria": {
"match_rules": {
"operator": "OR",
"conditions": [
{"type": "contains", "value": "stg/"},
{"type": "contains", "value": "release/"}
]
}
}
```
Nested example (alerts containing "error" AND either "stg/" or "release/"):
```json theme={null}
"matching_criteria": {
"match_rules": {
"operator": "AND",
"conditions": [
{"type": "contains", "value": "error"},
{
"operator": "OR",
"conditions": [
{"type": "contains", "value": "stg/"},
{"type": "contains", "value": "release/"}
]
}
]
}
}
```
**Precedence:** If both `text_matches` and `match_rules` are provided, `match_rules` takes precedence and `text_matches` will be regenerated from it.
##### Response
**Status Code:** `200 OK`
```json theme={null}
{
"id": "string", // Alert agent Global ID
"url": "string", // Link to alert agent details page
"name": "string", // Alert agent name
"status": "ACTIVE" // Status (ACTIVE or PAUSED)
}
```
#### Get Alert Agent (`GET /api/v1/alert-responders/{alert_id}`)
Retrieve detailed information about a specific alert agent.
##### Request
**Method:** `GET`
**Path:** `/api/v1/alert-responders/{alert_id}`
**Headers:**
* `X-TierZero-Org-Api-Key` (required): Your organization API key
**Path Parameters:**
* `alert_id` (required): Alert agent Global ID
##### Response
**Status Code:** `200 OK`
```json theme={null}
{
"id": "string", // Alert agent Global ID
"organization_name": "string", // Organization name
"team_name": "string", // Team name
"name": "string", // Alert agent name
"runbook": { // Investigation runbook (null if not active)
"investigation_prompt": "string" // Main investigation prompt
},
"matching_criteria": { // Alert matching criteria (both formats are always returned)
"text_matches": ["string"], // Wildcard patterns
"match_rules": { // Structured rules
"operator": "AND",
"conditions": [...]
},
"slack_bot_app_user_id": "string" // Optional: Bot/sender ID filter (Slack alerts only)
},
"webhook_sources": [ // Configured webhook sources (empty array for Slack alerts)
{
"type": "string", // Webhook type: PAGERDUTY, OPSGENIE, FIREHYDRANT, or ROOTLY
"remote_id": "string" // External webhook ID
}
],
"slack_channel_id": "string", // Slack channel ID (null for webhook-based alerts)
"notification_integration_ids": [ // Notification integration Global IDs (only present when configured)
"string"
],
"status": "ACTIVE", // Status (ACTIVE or PAUSED)
"created_at": "string", // ISO 8601 timestamp
"updated_at": "string" // ISO 8601 timestamp
}
```
#### List Alert Agents (`GET /api/v1/alert-responders`)
List all alert agents for your organization, optionally filtered by team.
##### Request
**Method:** `GET`
**Path:** `/api/v1/alert-responders`
**Headers:**
* `X-TierZero-Org-Api-Key` (required): Your organization API key
**Query Parameters:**
* `team_name` (optional): Filter by team name
##### Response
**Status Code:** `200 OK`
```json theme={null}
{
"alert_responders": [
{
"id": "string", // Alert agent Global ID
"name": "string", // Alert agent name
"team_name": "string",// Team name
"status": "ACTIVE", // Status (ACTIVE or PAUSED)
"url": "string" // Link to alert agent details page
}
]
}
```
#### Update Alert Agent (`PUT /api/v1/alert-responders/{alert_id}`)
Update an existing alert agent's configuration.
##### Request
**Method:** `PUT`
**Path:** `/api/v1/alert-responders/{alert_id}`
**Content-Type:** `application/json`
**Headers:**
* `X-TierZero-Org-Api-Key` (required): Your organization API key
* `Content-Type`: `application/json`
**Path Parameters:**
* `alert_id` (required): Alert agent Global ID
**Request Body:**
For **webhook-based alerts**:
```json theme={null}
{
"name": "string", // Optional: New name
"runbook": { // Optional: New runbook
"investigation_prompt": "string" // Main investigation prompt
},
"matching_criteria": { // Optional: New matching criteria (provide text_matches OR match_rules)
"text_matches": ["string"], // Option 1: Wildcard patterns
"match_rules": { // Option 2: Structured rules (see "Matching Criteria" above)
"operator": "AND",
"conditions": [...]
}
},
"webhook_sources": [ // Optional: New webhook sources (see "List Webhook Subscriptions" below)
{
"type": "string",
"remote_id": "string" // External webhook ID (use "List Webhook Subscriptions" to find valid remote_id values)
}
],
"notification_integration_ids": [ // Optional: New notification integration Global IDs (use "List Notification Integrations" below)
"string"
]
}
```
For **Slack alerts**:
```json theme={null}
{
"name": "string", // Optional: New name
"runbook": { // Optional: New runbook
"investigation_prompt": "string" // Main investigation prompt
},
"matching_criteria": { // Optional: New matching criteria (provide text_matches OR match_rules)
"text_matches": ["string"], // Option 1: Wildcard patterns
"match_rules": { // Option 2: Structured rules (see "Matching Criteria" above)
"operator": "AND",
"conditions": [...]
},
"slack_bot_app_user_id": "string" // Optional: Bot/sender ID filter
},
"slack_channel_id": "string", // Optional: New Slack channel ID (must start with 'C' or 'G')
"notification_integration_ids": [ // Optional: New notification integration Global IDs (use "List Notification Integrations" below)
"string"
]
}
```
All fields are optional. Only provided fields will be updated.
**Note:** Cannot mix alert types - don't specify `webhook_sources` for Slack alerts or `slack_channel_id` for webhook-based alerts.
##### Response
**Status Code:** `200 OK`
```json theme={null}
{
"id": "string", // Alert agent Global ID
"url": "string", // Link to alert agent details page
"name": "string", // Alert agent name
"status": "ACTIVE" // Status (ACTIVE or PAUSED)
}
```
#### Delete Alert Agent (`DELETE /api/v1/alert-responders/{alert_id}`)
Delete an alert agent.
##### Request
**Method:** `DELETE`
**Path:** `/api/v1/alert-responders/{alert_id}`
**Headers:**
* `X-TierZero-Org-Api-Key` (required): Your organization API key
**Path Parameters:**
* `alert_id` (required): Alert agent Global ID
##### Response
**Status Code:** `200 OK`
```json theme={null}
{
"success": true
}
```
**Idempotency:** Returns success even if the alert agent is already deleted.
#### Enable Alert Agent (`POST /api/v1/alert-responders/{alert_id}/enable`)
Enable a paused alert agent to start processing alerts.
##### Request
**Method:** `POST`
**Path:** `/api/v1/alert-responders/{alert_id}/enable`
**Headers:**
* `X-TierZero-Org-Api-Key` (required): Your organization API key
**Path Parameters:**
* `alert_id` (required): Alert agent Global ID
##### Response
**Status Code:** `200 OK`
```json theme={null}
{
"id": "string", // Alert agent Global ID
"status": "ACTIVE" // Updated status
}
```
#### Disable Alert Agent (`POST /api/v1/alert-responders/{alert_id}/disable`)
Disable an alert agent to temporarily stop processing alerts.
##### Request
**Method:** `POST`
**Path:** `/api/v1/alert-responders/{alert_id}/disable`
**Headers:**
* `X-TierZero-Org-Api-Key` (required): Your organization API key
**Path Parameters:**
* `alert_id` (required): Alert agent Global ID
##### Response
**Status Code:** `200 OK`
```json theme={null}
{
"id": "string", // Alert agent Global ID
"status": "PAUSED" // Updated status
}
```
#### List Webhook Subscriptions (`GET /api/v1/webhook-subscriptions`)
Discover available webhook subscriptions for your organization. Use this endpoint to find valid `webhook_sources` when creating or updating alert agents.
##### Request
**Method:** `GET`
**Path:** `/api/v1/webhook-subscriptions`
**Headers:**
* `X-TierZero-Org-Api-Key` (required): Your organization API key
##### Response
**Status Code:** `200 OK`
```json theme={null}
{
"webhook_subscriptions": [
{
"type": "string", // Webhook type: PAGERDUTY, OPSGENIE, FIREHYDRANT, ROOTLY, or SLACK
"remote_id": "string", // External webhook ID
"name": "string" // Human-readable name
}
]
}
```
#### List Notification Integrations (`GET /api/v1/notification-integrations`)
Discover available notification integrations for your organization. Use this endpoint to find valid `notification_integration_ids` when creating or updating alert agents.
##### Request
**Method:** `GET`
**Path:** `/api/v1/notification-integrations`
**Headers:**
* `X-TierZero-Org-Api-Key` (required): Your organization API key
**Query Parameters:**
* `kind` (optional): Filter by integration kind. Allowed values:
* `DISCORD_WEBHOOK`: Discord webhook integrations
* `SLACK_ALERT`: Slack alert integrations (Auto-Detect)
##### Response
**Status Code:** `200 OK`
```json theme={null}
{
"notification_integrations": [
{
"id": "string", // Notification integration Global ID
"name": "string", // Human-readable name
"kind": "string", // Integration kind (DISCORD_WEBHOOK or SLACK_ALERT)
"created_at": "string" // ISO 8601 timestamp
}
]
}
```
### Investigation Management
#### Get Investigation (`GET /api/v1/investigations/{investigation_id}`)
Retrieve details about a completed investigation (alert agent job run).
##### Request
**Method:** `GET`
**Path:** `/api/v1/investigations/{investigation_id}`
**Headers:**
* `X-TierZero-Org-Api-Key` (required): Your organization API key
**Path Parameters:**
* `investigation_id` (required): Investigation Global ID (GraphQL JobRun ID)
##### Response
**Status Code:** `200 OK`
```json theme={null}
{
"organization_name": "string", // Organization name
"team_name": "string", // Team name (null if not assigned)
"id": "string", // Investigation Global ID
"name": "string", // Investigation name
"status": "string", // Status: PENDING, IN_PROGRESS, ERRORED, COMPLETED, or STOPPED
"started_at": "string", // ISO 8601 timestamp (null if not started)
"completed_at": "string", // ISO 8601 timestamp (null if not completed)
"prompt": "string", // Investigation prompt
"event_payload": "string", // Original event/alert payload
"result": "string" // Formatted investigation result with numbered source citations and sources appended
}
```
**Note:** The `result` field contains the investigation analysis with:
* Source markup (`††toolu_...††`) replaced with numbered citations (`[1]`, `[2]`, etc.)
* Timestamp markers (`§`) removed
* A "Sources:" section appended at the end with clickable links to data sources
**Example Response:**
```json theme={null}
{
"organization_name": "Acme Corp",
"team_name": "Production",
"id": "R3JhcGhRTEpvYlJ1bjoxMjM=",
"name": "Critical API Error Investigation",
"status": "COMPLETED",
"started_at": "2025-01-15T10:30:00+00:00",
"completed_at": "2025-01-15T10:35:00+00:00",
"prompt": "Investigate this critical error and provide root cause analysis",
"event_payload": "{\"alert_id\": \"PD123\", \"service\": \"api-gateway\", \"severity\": \"critical\"}",
"result": "The investigation shows a spike in 500 errors [1] caused by database connection timeouts [2].\n\nSources:\n- [1]: [service:api-gateway status:500](https://datadog.com/logs/...)\n- [2]: [query:database timeout](https://datadog.com/logs/...)"
}
```
### Digest Agent Management
#### Create Digest Agent (`POST /api/v1/digests`)
Create a new scheduled digest agent job that periodically generates reports based on configured sections.
##### Request
**Method:** `POST`
**Path:** `/api/v1/digests`
**Content-Type:** `application/json`
**Headers:**
* `X-TierZero-Org-Api-Key` (required): Your organization API key
* `Content-Type`: `application/json`
**Request Body:**
```json theme={null}
{
"team_name": "string", // Required: Name of the team
"name": "string", // Required: Name for the digest agent
"sections": [ // Required: Array of digest agent sections
{
"section": "string", // Section name
"prompt": "string" // Prompt/query for this section
}
],
"schedule": "string", // Required: Cron expression (e.g., "0 9 * * 1-5")
"slack_channel_id": "string" // Optional: Slack channel ID for notifications
}
```
##### Response
**Status Code:** `200 OK`
```json theme={null}
{
"id": "string", // Digest Agent ID
"url": "string" // Link to digest agent details page
}
```
#### Get Digest Agent (`GET /api/v1/digests/{digest_id}`)
Retrieve details about a specific digest agent job.
##### Request
**Method:** `GET`
**Path:** `/api/v1/digests/{digest_id}`
**Headers:**
* `X-TierZero-Org-Api-Key` (required): Your organization API key
**Path Parameters:**
* `digest_id` (required): Digest Agent ID returned from create/update operations
##### Response
**Status Code:** `200 OK`
```json theme={null}
{
"organization_name": "string", // Organization name
"team_name": "string", // Team name
"id": "string", // Digest Agent ID
"name": "string", // Digest Agent name
"sections": [ // Digest Agent sections
{
"section": "string", // Section name
"prompt": "string" // Prompt/query for this section
}
],
"schedule": "string", // Cron expression
"slack_channel_id": "string" // Optional: Slack channel ID if configured
}
```
#### Update Digest Agent (`PUT /api/v1/digests/{digest_id}`)
Update an existing digest agent job's configuration.
##### Request
**Method:** `PUT`
**Path:** `/api/v1/digests/{digest_id}`
**Content-Type:** `application/json`
**Headers:**
* `X-TierZero-Org-Api-Key` (required): Your organization API key
* `Content-Type`: `application/json`
**Path Parameters:**
* `digest_id` (required): Digest Agent ID
**Request Body:**
```json theme={null}
{
"name": "string", // Optional: New name for the digest agent
"sections": [ // Optional: Updated sections array
{
"section": "string",
"prompt": "string"
}
],
"schedule": "string", // Optional: Updated cron expression
"slack_channel_id": "string" // Optional: Updated Slack channel ID
}
```
All fields are optional. Only provided fields will be updated.
##### Response
**Status Code:** `200 OK`
```json theme={null}
{
"id": "string", // Digest Agent ID
"url": "string" // Link to digest agent details page
}
```
#### Delete Digest Agent (`DELETE /api/v1/digests/{digest_id}`)
Delete a digest agent job.
##### Request
**Method:** `DELETE`
**Path:** `/api/v1/digests/{digest_id}`
**Headers:**
* `X-TierZero-Org-Api-Key` (required): Your organization API key
**Path Parameters:**
* `digest_id` (required): Digest Agent ID
##### Response
**Status Code:** `200 OK`
```json theme={null}
{
"success": true
}
```
#### Run Digest Agent (`POST /api/v1/digests/{digest_id}/run`)
Manually trigger a digest agent job to run immediately or schedule it for later.
##### Request
**Method:** `POST`
**Path:** `/api/v1/digests/{digest_id}/run`
**Content-Type:** `application/json`
**Headers:**
* `X-TierZero-Org-Api-Key` (required): Your organization API key
* `Content-Type`: `application/json`
**Path Parameters:**
* `digest_id` (required): Digest Agent ID
**Request Body:**
```json theme={null}
{
"run_sync": false, // Optional: Wait for completion (default: false)
"scheduled_runtime": 1735689600 // Optional: Unix timestamp in seconds to schedule the run
}
```
**Field Descriptions:**
* `run_sync` (optional): If `true`, runs synchronously and returns results in the response. Default is `false`.
* `scheduled_runtime` (optional): Unix timestamp in seconds. If provided, schedules the digest agent to run at this time instead of running immediately. Must be in the future.
**Note:** Notifications are always sent to the configured Slack channels and Discord webhooks for the digest agent.
##### Response (Immediate Run - Async Mode)
**Status Code:** `200 OK`
```json theme={null}
{
"id": "string", // Digest Agent ID
"url": "string" // Link to digest agent details page
}
```
##### Response (Immediate Run - Sync Mode with run\_sync: true)
**Status Code:** `200 OK`
```json theme={null}
{
"id": "string", // Digest Agent ID
"url": "string", // Link to digest agent details page
"results": [ // Array of section results
{
"section": "string", // Section name
"prompt": "string", // Section prompt
"result": "string" // Generated content
}
]
}
```
##### Response (Scheduled Run)
**Status Code:** `200 OK`
```json theme={null}
{
"id": "string", // Digest Agent ID
"url": "string", // Link to digest agent details page
"scheduled_runtime": 1735689600 // Unix timestamp when the digest agent is scheduled to run
}
```
#### Cron Expression Reference
Digest Agent schedules use cron expressions with five fields:
```
* * * * *
│ │ │ │ └─ Day of week (0=Mon, 6=Sun)
│ │ │ └─── Month (1-12)
│ │ └───── Day of month (1-31)
│ └─────── Hour (0-23)
└───────── Minute (0-59)
```
**Note:** Day of week starts at 0 for Monday, which differs from standard cron.
**Common patterns:**
* Weekdays at 9 AM: `0 9 * * 0-4`
* Daily at 9 AM: `0 9 * * *`
* Every Monday at 10 AM: `0 10 * * 0`
* Every 6 hours: `0 */6 * * *`
**Special characters:** `*` (any), `,` (list), `-` (range), `/` (step)
### Interaction Management
#### Get Interaction Results (`GET /api/v1/interactions/{interaction_id}`)
Retrieve the results of a previous TierZero interaction using its interaction ID. This endpoint allows you to fetch results later, which is useful for polling scheduled actions or accessing historical interactions.
For scheduled actions, this endpoint returns the current status and results when complete.
##### Request
**Method:** `GET`
**Path:** `/api/v1/interactions/{interaction_id}`
**Headers:**
* `X-TierZero-Org-Api-Key` (required): Your organization API key
* `Accept`: `application/json`
**Path Parameters:**
* `interaction_id` (required): The interaction ID returned from a previous `POST /api/v1/interactions` call
##### Response
###### Status Response (for async interactions in progress)
**Status Code:** `200 OK`
```json theme={null}
{
"interaction_id": "string", // The interaction ID for this interaction
"status": "PENDING|IN_PROGRESS|FAILED", // Current processing status
"url": "string" // Link to interaction details page in TierZero UI
}
```
###### Complete Response (for completed interactions)
**Status Code:** `200 OK`
```json theme={null}
{
"question": "string", // The original question that was asked
"content": "string", // TierZero's response in formatted markdown
"sources": [ // Array of source information that supported the response
{
"log_type_name": "string", // Name of the log type or data source
"query": "string", // JSON string containing the query parameters
"start": "string", // ISO 8601 timestamp for query start time
"end": "string", // ISO 8601 timestamp for query end time
"link": "string", // Direct URL to view the data in the source system
"artifact_id": "string|null", // Associated artifact identifier, if any
"tool_id": "string", // Internal tool identifier used for the query
"type": "string", // Type of source (e.g., "DatadogLogQuery")
"source_id": "number", // Numeric identifier for the source
"display_text": "string" // Human-readable description of the query
}
],
"interaction_id": "string", // The interaction ID for this interaction
"status": "COMPLETED", // Processing status
"url": "string" // Link to interaction details page in TierZero UI
}
```
**Status Values:**
* `PENDING`: Request received and queued for processing
* `IN_PROGRESS`: Currently being processed by TierZero
* `COMPLETED`: Processing complete, results available
* `FAILED`: Processing failed due to an error
##### Error Responses
**401 Unauthorized:**
```json theme={null}
{
"detail": "Unauthorized"
}
```
**404 Not Found:**
```json theme={null}
{
"detail": "Interaction not found"
}
```
**500 Internal Server Error:**
```json theme={null}
{
"detail": "Error message describing the issue"
}
```
#### Create Interaction (`POST /api/v1/interactions`)
Submit a question to the TierZero AI agent and receive an intelligent response based on your organization's knowledge catalog and observability data.
##### Request
**Method:** `POST`
**Path:** `/api/v1/interactions`
**Content-Type:** `application/json`
**Headers:**
* `X-TierZero-Org-Api-Key` (required): Your organization API key
* `Content-Type`: `application/json`
* `Accept`: `application/json`
**Request Body:**
```json theme={null}
{
"question": "string", // Required: The question to ask TierZero (or prompt for scheduled actions)
"context": [ // Optional: Previous conversation context
{
"interaction_id": "string", // ID from previous API call
"content": "string" // Additional context content
}
],
"user_id": "string", // Optional: User identifier for tracking
"use_tools": boolean, // Optional: Enable tool usage (default: true)
"scheduled_runtimes": [1735689600, 1735776000], // Optional: List of Unix timestamps for scheduled execution (each must be in the future, max 24 hours)
"notifications": { // Optional: Notification configuration
"slack": { // Optional: Slack notifications
"slack_channel_ids": ["C01234567"]
},
"discord": { // Optional: Discord notifications
"discord_webhook_urls": ["https://discord.com/api/webhooks/123456789/abc..."]
},
"web": { // Optional: Webhook callback
"callback_url": "string",
"custom_headers": { // Optional: Custom HTTP headers for callback
"Authorization": "Bearer token",
"X-Custom-Header": "value"
}
}
}
}
```
**Field Descriptions:**
* `question` (required): The question or prompt you want TierZero to analyze. For scheduled actions, this becomes the prompt that runs at the scheduled time.
* `context` (optional): Array of previous interactions to provide conversation context
* `user_id` (optional): Identifier for the user making the request, useful for tracking and analytics
* `use_tools` (optional): If `false`, TierZero will respond without using any tools (no data queries, no integrations). Default is `true`. Set to `false` for faster responses when you only need analysis based on existing knowledge
* `scheduled_runtimes` (optional): List of Unix timestamps (in seconds) for when to execute. Each timestamp must be in the future (max 24 hours). Maximum of 10 timestamps allowed per request. Returns `interaction_id` and status `SCHEDULED` for polling via GET endpoint.
* **Legacy Parameter**: `scheduled_runtime` (singular, integer) is still supported for backwards compatibility but **deprecated**. When using the legacy parameter, the API returns the old response format with top-level `interaction_id`, `scheduled_runtime`, `url`, and `status` fields instead of the `interactions` array. New integrations should use `scheduled_runtimes` (plural, array).
* `notifications` (optional): Nested object containing notification configurations:
* `slack` (optional): Slack notification settings
* `slack_channel_ids`: Array of Slack channel IDs (e.g., `["C01234567"]`). Each ID should start with 'C' (public) or 'G' (private)
* `discord` (optional): Discord notification settings
* `discord_webhook_urls`: Array of Discord webhook URLs
* `web` (optional): Webhook callback settings for async responses
* `callback_url`: URL where results will be sent
* `custom_headers` (optional): Dictionary of custom HTTP headers for callbacks (e.g., `{"Authorization": "Bearer token"}`). If not provided, defaults to `{"X-TierZero-Org-Api-Key": "your-api-key"}`
**Execution Modes:**
* **Synchronous (default)**: Returns result immediately in the response. No `scheduled_runtimes` or `notifications.web.callback_url` provided. Status: `COMPLETED`.
* **Asynchronous with callback**: Provide `notifications.web.callback_url` without `scheduled_runtimes`. Returns `interaction_id` immediately, executes in background, sends results to callback URL when done. Status: `SCHEDULED`.
* **Scheduled**: Provide `scheduled_runtimes` (list of future timestamps). Returns `interaction_id` immediately, executes at specified times. Poll via GET endpoint or use notification channels. Status: `SCHEDULED`.
**Note for Scheduled Actions:**
* A unique name is automatically generated for each scheduled action
* Scheduled actions are limited to at most 24 hours in the future
* Maximum of 10 scheduled runtimes allowed per request
* Notification targets (via `notifications` object) are optional - you can query results using `GET /api/v1/interactions/{interaction_id}`
##### Response
###### Synchronous Response (no callback\_url, no scheduled\_runtime)
**Status Code:** `200 OK`
```json theme={null}
{
"question": "string", // The original question that was asked
"content": "string", // TierZero's response in formatted markdown
"sources": [ // Array of source information that supported the response
{
"log_type_name": "string", // Name of the log type or data source
"query": "string", // JSON string containing the query parameters
"start": "string", // ISO 8601 timestamp for query start time
"end": "string", // ISO 8601 timestamp for query end time
"link": "string", // Direct URL to view the data in the source system
"artifact_id": "string|null", // Associated artifact identifier, if any
"tool_id": "string", // Internal tool identifier used for the query
"type": "string", // Type of source (e.g., "DatadogLogQuery")
"source_id": "number", // Numeric identifier for the source
"display_text": "string" // Human-readable description of the query
}
],
"interaction_id": "string", // Unique identifier for this interaction
"url": "string" // Link to interaction details page in TierZero UI
}
```
###### Asynchronous Response (with callback\_url, no scheduled\_runtimes)
**Status Code:** `200 OK`
```json theme={null}
{
"interactions": [
{
"interaction_id": "abc-123",
"scheduled_runtime": 1735689600,
"url": "https://app.tierzero.ai/interactions/abc-123"
}
],
"status": "SCHEDULED" // Job scheduled for background execution
}
```
**Note**: When using only `callback_url` without `scheduled_runtimes`, the job executes immediately in the background and results are sent to the callback URL. Status is `SCHEDULED`.
###### Scheduled Action Response (with scheduled\_runtimes)
**Status Code:** `200 OK`
```json theme={null}
{
"interactions": [
{
"interaction_id": "abc-123",
"scheduled_runtime": 1735689600,
"url": "https://app.tierzero.ai/interactions/abc-123"
},
{
"interaction_id": "def-456",
"scheduled_runtime": 1735776000,
"url": "https://app.tierzero.ai/interactions/def-456"
}
],
"status": "SCHEDULED" // Jobs scheduled for future execution
}
```
**Note**: When using `scheduled_runtimes`, jobs execute at the specified future times. The callback URL (if provided) will receive results when each scheduled job completes.
###### Legacy Response Format (DEPRECATED - using `scheduled_runtime` singular parameter)
**Status Code:** `200 OK`
```json theme={null}
{
"interaction_id": "abc-123",
"scheduled_runtime": 1735689600,
"url": "https://app.tierzero.ai/interactions/abc-123",
"status": "SCHEDULED"
}
```
**Note**: This format is maintained for backwards compatibility when using the deprecated `scheduled_runtime` (singular) parameter. New integrations should use `scheduled_runtimes` (plural, array) which returns the `interactions` array format shown above.
**Important Behavioral Notes:**
1. **One Interaction Per Scheduled Runtime**: When using `scheduled_runtimes`, each timestamp creates a **separate interaction** with its own unique `interaction_id`. This is a 1:1 mapping - one timestamp = one job = one interaction.
2. **Tracking Individual Executions**:
* Poll each interaction independently: `GET /api/v1/interactions/{interaction_id}`
* Each interaction has its own lifecycle and status
* Store ALL returned `interaction_id`s if you need to track completion
3. **Callback Behavior** (when using `callback_url`):
* You will receive **multiple callbacks** - one per scheduled runtime
* Each callback includes the specific `interaction_id` for that execution
* Callback payload structure:
```json theme={null}
{
"interaction_id": "abc-123",
"question": "your question",
"content": "response content",
"sources": []
}
```
4. **Client Implementation Requirements**:
* **MUST** store all `interaction_id`s from the `interactions` array
* **MUST** handle multiple callbacks when using `callback_url` (one per scheduled runtime)
* Each callback's `interaction_id` corresponds to one of the returned interactions
##### Context Handling
The `context` field allows you to provide conversation history for more contextual responses:
1. **Interaction ID Context**: If a context item includes an `interaction_id` from a previous API call, TierZero will retrieve the full conversation history for that interaction
2. **Content-Only Context**: If only `content` is provided, it will be treated as a user message in the conversation context
3. **Mixed Context**: You can combine both types of context items in a single request
##### Error Responses
**400 Bad Request:**
```json theme={null}
{
"detail": "Invalid payload" | "Invalid question" | "Context must be a list" | "Invalid callback_url: {url}"
}
```
**401 Unauthorized:**
```json theme={null}
{
"detail": "Unauthorized"
}
```
**500 Internal Server Error:**
```json theme={null}
{
"detail": "Error message describing the issue"
}
```
### Billing
#### Get Credit Usage by Team (`POST /api/v1/billing/credit-usage`)
Retrieve credit usage data grouped by team and month for a given time range. Rows with `team_id: null` represent unattributed usage from automated features (alert responders, incidents, digests, internal agents).
##### Request
**Method:** `POST`
**Path:** `/api/v1/billing/credit-usage`
**Content-Type:** `application/json`
**Headers:**
* `X-TierZero-Org-Api-Key` (required): Your organization API key
* `Content-Type`: `application/json`
**Authentication:** This endpoint uses an **organization API key**, consistent with the other programmatic endpoints. The key scopes the response to that organization's usage.
**Request Body:**
```json theme={null}
{
"start": "string", // Required: Start date in ISO 8601 format (e.g., "2026-05-01T00:00:00Z")
"end": "string" // Required: End date in ISO 8601 format (e.g., "2026-07-01T00:00:00Z")
}
```
**Validation:**
* `end` must be after `start`
* Maximum time range is 366 days
##### Response
**Status Code:** `200 OK`
```json theme={null}
{
"organization_id": 11,
"start": "2026-05-01T00:00:00+00:00",
"end": "2026-07-01T00:00:00+00:00",
"data": [
{
"team_id": 7, // Team ID (null for unattributed usage)
"team_name": "Default", // Team name (null for unattributed usage)
"month": "2026-05", // Year-month
"credits": 4406.90 // Credit usage for this team/month
},
{
"team_id": null,
"team_name": null,
"month": "2026-05",
"credits": 9869.56
}
],
"monthly_totals": { // Aggregated credits per month (all teams + unattributed)
"2026-05": 39565.56,
"2026-06": 83043.01
}
}
```
**Data Fields:**
* `data`: Array of per-team, per-month credit usage records, ordered by month then credits descending
* `data[].team_id` / `data[].team_name`: The team. `null` values represent unattributed usage from automated features (alert responders, incident investigations, digests, internal agents, support agents)
* `data[].credits`: Credit usage rounded to 2 decimal places
* `monthly_totals`: Convenience object summing all credits (attributed + unattributed) per month
##### Error Responses
**400 Bad Request:**
```json theme={null}
{
"detail": "Start date required" | "End date required" | "End date must be after start date" | "Time range cannot exceed 366 days" | "Invalid start or end timestamp format. Use ISO 8601."
}
```
**401 Unauthorized:**
```json theme={null}
{
"detail": "Missing API key" | "Invalid API key"
}
```
## Usage Examples
### Alert Agent Management Examples
#### Create an Alert Agent
```bash theme={null}
# First, discover available webhook sources
curl -X GET "https://api.tierzero.ai/api/v1/webhook-subscriptions" \
-H "X-TierZero-Org-Api-Key: your-api-key"
# Discover available notification integrations
curl -X GET "https://api.tierzero.ai/api/v1/notification-integrations" \
-H "X-TierZero-Org-Api-Key: your-api-key"
# Create webhook-based alert agent (PagerDuty example)
curl -X POST "https://api.tierzero.ai/api/v1/alert-responders" \
-H "X-TierZero-Org-Api-Key: your-api-key" \
-H "Content-Type: application/json" \
-d '{
"team_name": "Production",
"name": "Critical Error Alert",
"matching_criteria": {
"text_matches": ["critical", "error", "fatal"]
},
"webhook_sources": [
{
"type": "PAGERDUTY",
"remote_id": "PXXXXXX"
}
],
"runbook": {
"investigation_prompt": "Analyze this alert and provide detailed root cause analysis with remediation steps"
},
"notification_integration_ids": ["R3JhcGhRTE5vdGlmaWNhdGlvbkludGVncmF0aW9uOjEyMw=="]
}'
# Create Slack alert agent
curl -X POST "https://api.tierzero.ai/api/v1/alert-responders" \
-H "X-TierZero-Org-Api-Key: your-api-key" \
-H "Content-Type: application/json" \
-d '{
"team_name": "Production",
"name": "Database Alerts",
"slack_channel_id": "C01234567",
"matching_criteria": {
"text_matches": ["database", "connection", "timeout"],
"slack_bot_app_user_id": "B01234567"
},
"runbook": {
"investigation_prompt": "Investigate database issues and provide remediation steps"
},
"notification_integration_ids": ["R3JhcGhRTE5vdGlmaWNhdGlvbkludGVncmF0aW9uOjEyMw=="]
}'
```
#### List Alert Agents
```bash theme={null}
# List all alert agents
curl -X GET "https://api.tierzero.ai/api/v1/alert-responders" \
-H "X-TierZero-Org-Api-Key: your-api-key"
# List alert agents for a specific team
curl -X GET "https://api.tierzero.ai/api/v1/alert-responders?team_name=Production" \
-H "X-TierZero-Org-Api-Key: your-api-key"
```
#### Get Alert Agent Details
```bash theme={null}
curl -X GET "https://api.tierzero.ai/api/v1/alert-responders/{alert_id}" \
-H "X-TierZero-Org-Api-Key: your-api-key"
```
#### Update Alert Agent
```bash theme={null}
# Update with text_matches (simple wildcards)
curl -X PUT "https://api.tierzero.ai/api/v1/alert-responders/{alert_id}" \
-H "X-TierZero-Org-Api-Key: your-api-key" \
-H "Content-Type: application/json" \
-d '{
"name": "Updated Alert Name",
"matching_criteria": {
"text_matches": ["*critical*", "*emergency*"]
}
}'
# Update with match_rules (structured AND/OR logic)
curl -X PUT "https://api.tierzero.ai/api/v1/alert-responders/{alert_id}" \
-H "X-TierZero-Org-Api-Key: your-api-key" \
-H "Content-Type: application/json" \
-d '{
"matching_criteria": {
"match_rules": {
"operator": "OR",
"conditions": [
{"type": "contains", "value": "stg/"},
{"type": "contains", "value": "release/"}
]
}
}
}'
```
#### Enable/Disable Alert Agent
```bash theme={null}
# Enable alert agent
curl -X POST "https://api.tierzero.ai/api/v1/alert-responders/{alert_id}/enable" \
-H "X-TierZero-Org-Api-Key: your-api-key"
# Disable alert agent
curl -X POST "https://api.tierzero.ai/api/v1/alert-responders/{alert_id}/disable" \
-H "X-TierZero-Org-Api-Key: your-api-key"
```
#### Delete Alert Agent
```bash theme={null}
curl -X DELETE "https://api.tierzero.ai/api/v1/alert-responders/{alert_id}" \
-H "X-TierZero-Org-Api-Key: your-api-key"
```
### Investigation Management Examples
#### Get Investigation Details
```bash theme={null}
# Get investigation by Global ID
curl -X GET "https://api.tierzero.ai/api/v1/investigations/{investigation_id}" \
-H "X-TierZero-Org-Api-Key: your-api-key"
```
**Example Response:**
```json theme={null}
{
"organization_name": "Acme Corp",
"team_name": "Production",
"id": "R3JhcGhRTEpvYlJ1bjoxMjM=",
"name": "Critical API Error Investigation",
"status": "COMPLETED",
"started_at": "2025-01-15T10:30:00+00:00",
"completed_at": "2025-01-15T10:35:00+00:00",
"prompt": "Investigate this critical error and provide root cause analysis",
"event_payload": "{\"alert_id\": \"PD123\", \"service\": \"api-gateway\", \"severity\": \"critical\"}",
"result": "The investigation identified a spike in 500 errors [1] caused by database connection pool exhaustion [2].\n\nRoot Cause:\nThe database connection pool reached its maximum capacity of 50 connections due to a sudden traffic spike.\n\nSources:\n- [1]: [service:api-gateway status:500](https://app.datadoghq.com/logs?query=service:api-gateway+status:500)\n- [2]: [query:connection pool timeout](https://app.datadoghq.com/logs?query=database+connection+pool)"
}
```
### Digest Agent Management Examples
#### Create a Daily Digest Agent
```bash theme={null}
curl -X POST "https://api.tierzero.ai/api/v1/digests" \
-H "X-TierZero-Org-Api-Key: your-api-key" \
-H "Content-Type: application/json" \
-d '{
"team_name": "Engineering",
"name": "Daily Production Health",
"sections": [
{
"section": "Error Summary",
"prompt": "Summarize production errors from the last 24 hours"
},
{
"section": "Performance Metrics",
"prompt": "Report on API response times and throughput"
}
],
"schedule": "0 9 * * 1-5",
"slack_channel_id": "C01234567"
}'
```
#### Get Digest Agent Details
```bash theme={null}
curl -X GET "https://api.tierzero.ai/api/v1/digests/{digest_id}" \
-H "X-TierZero-Org-Api-Key: your-api-key"
```
#### Update a Digest Agent
```bash theme={null}
curl -X PUT "https://api.tierzero.ai/api/v1/digests/{digest_id}" \
-H "X-TierZero-Org-Api-Key: your-api-key" \
-H "Content-Type: application/json" \
-d '{
"name": "Updated Daily Report",
"schedule": "0 10 * * 1-5"
}'
```
#### Run a Digest Agent Immediately
```bash theme={null}
# Run asynchronously (default)
curl -X POST "https://api.tierzero.ai/api/v1/digests/{digest_id}/run" \
-H "X-TierZero-Org-Api-Key: your-api-key" \
-H "Content-Type: application/json"
# Run synchronously and get results immediately
curl -X POST "https://api.tierzero.ai/api/v1/digests/{digest_id}/run" \
-H "X-TierZero-Org-Api-Key: your-api-key" \
-H "Content-Type: application/json" \
-d '{
"run_sync": true
}'
```
#### Schedule a Digest Agent for Later
```bash theme={null}
# Schedule digest agent to run at a specific time (Unix timestamp)
curl -X POST "https://api.tierzero.ai/api/v1/digests/{digest_id}/run" \
-H "X-TierZero-Org-Api-Key: your-api-key" \
-H "Content-Type: application/json" \
-d '{
"scheduled_runtime": 1735689600
}'
```
#### Delete a Digest Agent
```bash theme={null}
curl -X DELETE "https://api.tierzero.ai/api/v1/digests/{digest_id}" \
-H "X-TierZero-Org-Api-Key: your-api-key"
```
### Scheduled Action Examples
Scheduled actions use the same `/api/v1/interactions` endpoint with additional parameters.
#### Create Scheduled Action with Slack Notification
Schedule a TierZero agent to run at specific times and post results to a Slack channel:
```bash theme={null}
curl -X POST "https://api.tierzero.ai/api/v1/interactions" \
-H "X-TierZero-Org-Api-Key: your-api-key" \
-H "Content-Type: application/json" \
-d '{
"question": "Analyze system performance metrics from the last 7 days and identify any anomalies",
"scheduled_runtimes": [1735689600, 1735776000],
"notifications": {
"slack": {
"slack_channel_ids": ["C01234567"]
}
}
}'
```
#### Create Scheduled Action with Discord Notification
Schedule an action to send results to a Discord webhook:
```bash theme={null}
curl -X POST "https://api.tierzero.ai/api/v1/interactions" \
-H "X-TierZero-Org-Api-Key: your-api-key" \
-H "Content-Type: application/json" \
-d '{
"question": "Review security alerts from the last 24 hours and summarize any critical findings",
"scheduled_runtimes": [1735776000],
"notifications": {
"discord": {
"discord_webhook_urls": ["https://discord.com/api/webhooks/123456789/abc..."]
}
}
}'
```
#### Create Scheduled Action with Multiple Notifications
Send results to both Slack and Discord:
```bash theme={null}
curl -X POST "https://api.tierzero.ai/api/v1/interactions" \
-H "X-TierZero-Org-Api-Key: your-api-key" \
-H "Content-Type: application/json" \
-d '{
"question": "Generate a comprehensive report on infrastructure utilization and cost optimization opportunities",
"scheduled_runtimes": [1736208000],
"notifications": {
"slack": {
"slack_channel_ids": ["C01234567", "C02345678"]
},
"discord": {
"discord_webhook_urls": ["https://discord.com/api/webhooks/123456789/abc...", "https://discord.com/api/webhooks/987654321/def..."]
}
}
}'
```
### Billing Examples
#### Get Credit Usage by Team
```bash theme={null}
curl -X POST "https://api.tierzero.ai/api/v1/billing/credit-usage" \
-H "X-TierZero-Org-Api-Key: your-api-key" \
-H "Content-Type: application/json" \
-d '{
"start": "2026-05-01T00:00:00Z",
"end": "2026-07-01T00:00:00Z"
}'
```
### Interaction Examples
#### Basic Question
```bash theme={null}
curl -X POST "https://api.tierzero.ai/api/v1/interactions" \
-H "X-TierZero-Org-Api-Key: your-api-key" \
-H "Content-Type: application/json" \
-d '{
"question": "What errors occurred in our application in the last hour?"
}'
```
### Question with Context
```bash theme={null}
curl -X POST "https://api.tierzero.ai/api/v1/interactions" \
-H "X-TierZero-Org-Api-Key: your-api-key" \
-H "Content-Type: application/json" \
-d '{
"question": "Are these errors related to the deployment we discussed?",
"context": [
{
"interaction_id": "123e4567-e89b-12d3-a456-426614174000"
}
],
"user_id": "user123"
}'
```
### Asynchronous Request with Callback
```bash theme={null}
curl -X POST "https://api.tierzero.ai/api/v1/interactions" \
-H "X-TierZero-Org-Api-Key: your-api-key" \
-H "Content-Type: application/json" \
-d '{
"question": "Analyze the performance impact of our latest deployment",
"notifications": {
"web": {
"callback_url": "https://your-app.com/webhooks/tierzero"
}
}
}'
```
### Scheduled Request with Polling
```bash theme={null}
# Schedule request for future execution
curl -X POST "https://api.tierzero.ai/api/v1/interactions" \
-H "X-TierZero-Org-Api-Key: your-api-key" \
-H "Content-Type: application/json" \
-d '{
"question": "Analyze the performance impact of our latest deployment",
"scheduled_runtimes": [1735689600],
"notifications": {
"slack": {
"slack_channel_ids": ["C01234567"]
}
}
}'
# Response: {"interactions": [{"interaction_id": "123e4567-e89b-12d3-a456-426614174000", "scheduled_runtime": 1735689600, "url": "https://app.tierzero.ai/interactions/123e4567-e89b-12d3-a456-426614174000"}], "status": "SCHEDULED"}
# Poll for results using the interaction_id from the response
curl -X GET "https://api.tierzero.ai/api/v1/interactions/123e4567-e89b-12d3-a456-426614174000" \
-H "X-TierZero-Org-Api-Key: your-api-key" \
-H "Accept: application/json"
```
### Request Without Tools (Knowledge-Only Response)
```bash theme={null}
# Request a response based only on existing knowledge, without querying data sources
curl -X POST "https://api.tierzero.ai/api/v1/interactions" \
-H "X-TierZero-Org-Api-Key: your-api-key" \
-H "Content-Type: application/json" \
-d '{
"question": "What is our standard deployment process?",
"use_tools": false
}'
```
### Retrieve Interaction Results
```bash theme={null}
curl -X GET "https://api.tierzero.ai/api/v1/interactions/123e4567-e89b-12d3-a456-426614174000" \
-H "X-TierZero-Org-Api-Key: your-api-key" \
-H "Accept: application/json"
```
## Response Format
TierZero responses are formatted as markdown and may include:
* **Analysis summaries** with key findings
* **Relevant log excerpts** and error messages
* **Metric trends** and performance data
* **Recommendations** for investigation or resolution
* **Links to relevant dashboards** and tools
The `interaction_id` can be used in subsequent API calls to maintain conversation context and build upon previous analyses.
## Rate Limits
API requests are subject to rate limiting based on your organization's plan. Rate limit information is included in response headers:
* `X-RateLimit-Limit`: Maximum requests per time window
* `X-RateLimit-Remaining`: Remaining requests in current window
* `X-RateLimit-Reset`: Time when the rate limit resets
## Best Practices
1. **Provide Context**: Use the `context` field to maintain conversation history for better responses
2. **Descriptive Questions**: Ask specific, detailed questions for more accurate analysis
3. **Choose the Right Mode**:
* Use **synchronous mode** for immediate responses (no `callback_url` or `scheduled_runtime`)
* Use **scheduled mode with polling** (`scheduled_runtime`) for time-based execution when you need full control over result retrieval
* Use **async mode with callbacks** (`callback_url`) for fire-and-forget scenarios with webhook handling
4. **Optimize Tool Usage**:
* Set `use_tools: true` (default) when you need TierZero to query data sources, run integrations, or perform real-time analysis
* Set `use_tools: false` for faster responses when you only need answers based on existing knowledge (e.g., documentation, runbooks, best practices)
5. **Track Interactions**: Store interaction IDs to enable follow-up questions and conversation threading
6. **Handle Errors**: Implement proper error handling for all HTTP status codes
7. **Respect Rate Limits**: Monitor rate limit headers and implement backoff strategies
8. **Poll Efficiently**: For scheduled mode, use reasonable polling intervals (2-5 seconds) and implement timeouts
9. **User Tracking**: Use the `user_id` field for analytics, rate limiting, and audit trails
## Integration Examples
### Python
```python theme={null}
import requests
import json
import time
# Alert Agent Management Functions
def create_alert_responder(team_name, name, matching_criteria, api_key,
webhook_sources=None, slack_channel_id=None,
runbook=None, notification_integration_ids=None):
"""
Create a new alert agent.
Args:
team_name: Team name
name: Alert agent name
matching_criteria: Matching criteria dict (text_matches, slack_bot_app_user_id)
api_key: Organization API key
webhook_sources: Optional list of webhook sources (for webhook-based alerts)
slack_channel_id: Optional Slack channel ID (for Slack alerts)
runbook: Optional runbook configuration
notification_integration_ids: Optional notification integration IDs
Note: Must specify either webhook_sources OR slack_channel_id (not both)
"""
url = "https://api.tierzero.ai/api/v1/alert-responders"
headers = {
"X-TierZero-Org-Api-Key": api_key,
"Content-Type": "application/json"
}
payload = {
"team_name": team_name,
"name": name,
"matching_criteria": matching_criteria
}
if webhook_sources:
payload["webhook_sources"] = webhook_sources
if slack_channel_id:
payload["slack_channel_id"] = slack_channel_id
if runbook:
payload["runbook"] = runbook
if notification_integration_ids:
payload["notification_integration_ids"] = notification_integration_ids
response = requests.post(url, headers=headers, json=payload)
response.raise_for_status()
return response.json()
def get_alert_responder(alert_id, api_key):
"""Get alert agent details"""
url = f"https://api.tierzero.ai/api/v1/alert-responders/{alert_id}"
headers = {
"X-TierZero-Org-Api-Key": api_key
}
response = requests.get(url, headers=headers)
response.raise_for_status()
return response.json()
def list_alert_responders(api_key, team_name=None):
"""List all alert agents, optionally filtered by team"""
url = "https://api.tierzero.ai/api/v1/alert-responders"
headers = {
"X-TierZero-Org-Api-Key": api_key
}
params = {}
if team_name:
params["team_name"] = team_name
response = requests.get(url, headers=headers, params=params)
response.raise_for_status()
return response.json()
def update_alert_responder(alert_id, api_key, name=None, matching_criteria=None,
webhook_sources=None, slack_channel_id=None,
runbook=None, notification_integration_ids=None):
"""
Update an existing alert agent.
Args:
alert_id: Alert agent Global ID
api_key: Organization API key
name: Optional new name
matching_criteria: Optional new matching criteria
webhook_sources: Optional new webhook sources (for webhook-based alerts only)
slack_channel_id: Optional new Slack channel ID (for Slack alerts only)
runbook: Optional new runbook
notification_integration_ids: Optional new notification integration IDs
Note: Cannot mix alert types - don't specify webhook_sources for Slack alerts
or slack_channel_id for webhook-based alerts
"""
url = f"https://api.tierzero.ai/api/v1/alert-responders/{alert_id}"
headers = {
"X-TierZero-Org-Api-Key": api_key,
"Content-Type": "application/json"
}
payload = {}
if name:
payload["name"] = name
if matching_criteria:
payload["matching_criteria"] = matching_criteria
if webhook_sources:
payload["webhook_sources"] = webhook_sources
if slack_channel_id:
payload["slack_channel_id"] = slack_channel_id
if runbook:
payload["runbook"] = runbook
if notification_integration_ids:
payload["notification_integration_ids"] = notification_integration_ids
response = requests.put(url, headers=headers, json=payload)
response.raise_for_status()
return response.json()
def enable_alert_responder(alert_id, api_key):
"""Enable an alert agent"""
url = f"https://api.tierzero.ai/api/v1/alert-responders/{alert_id}/enable"
headers = {
"X-TierZero-Org-Api-Key": api_key
}
response = requests.post(url, headers=headers)
response.raise_for_status()
return response.json()
def disable_alert_responder(alert_id, api_key):
"""Disable an alert agent"""
url = f"https://api.tierzero.ai/api/v1/alert-responders/{alert_id}/disable"
headers = {
"X-TierZero-Org-Api-Key": api_key
}
response = requests.post(url, headers=headers)
response.raise_for_status()
return response.json()
def delete_alert_responder(alert_id, api_key):
"""Delete an alert agent"""
url = f"https://api.tierzero.ai/api/v1/alert-responders/{alert_id}"
headers = {
"X-TierZero-Org-Api-Key": api_key
}
response = requests.delete(url, headers=headers)
response.raise_for_status()
return response.json()
def list_webhook_subscriptions(api_key):
"""List available webhook subscriptions"""
url = "https://api.tierzero.ai/api/v1/webhook-subscriptions"
headers = {
"X-TierZero-Org-Api-Key": api_key
}
response = requests.get(url, headers=headers)
response.raise_for_status()
return response.json()
def list_notification_integrations(api_key, kind=None):
"""
List available notification integrations.
Args:
api_key: Organization API key
kind: Optional filter by kind. Allowed values: DISCORD_WEBHOOK, SLACK_ALERT
"""
url = "https://api.tierzero.ai/api/v1/notification-integrations"
headers = {
"X-TierZero-Org-Api-Key": api_key
}
params = {}
if kind:
params["kind"] = kind
response = requests.get(url, headers=headers, params=params)
response.raise_for_status()
return response.json()
# Investigation Management Functions
def get_investigation(investigation_id, api_key):
"""Get investigation details by Global ID"""
url = f"https://api.tierzero.ai/api/v1/investigations/{investigation_id}"
headers = {
"X-TierZero-Org-Api-Key": api_key
}
response = requests.get(url, headers=headers)
response.raise_for_status()
return response.json()
# Investigation usage example
investigation = get_investigation("R3JhcGhRTEpvYlJ1bjoxMjM=", "your-api-key")
print(f"Investigation: {investigation['name']}")
print(f"Status: {investigation['status']}")
print(f"Result:\n{investigation['result']}")
# Alert agent usage examples
# First, discover available webhook sources
webhook_subs = list_webhook_subscriptions("your-api-key")
print(f"Available webhook subscriptions: {len(webhook_subs['webhook_subscriptions'])}")
# Discover available notification integrations
notification_integrations = list_notification_integrations("your-api-key")
print(f"Available notification integrations: {len(notification_integrations['notification_integrations'])}")
# Create webhook-based alert agent (PagerDuty example)
webhook_alert = create_alert_responder(
team_name="Production",
name="Critical Error Alert",
matching_criteria={"text_matches": ["critical", "error", "fatal"]},
webhook_sources=[{"type": "PAGERDUTY", "remote_id": "PXXXXXX"}],
api_key="your-api-key",
runbook={
"investigation_prompt": "Analyze this alert and provide detailed root cause analysis with remediation steps"
},
notification_integration_ids=["R3JhcGhRTE5vdGlmaWNhdGlvbkludGVncmF0aW9uOjEyMw=="]
)
print(f"Created webhook alert agent: {webhook_alert['url']}")
# Create Slack alert agent
slack_alert = create_alert_responder(
team_name="Production",
name="Database Alerts",
matching_criteria={
"text_matches": ["database", "connection", "timeout"],
"slack_bot_app_user_id": "B01234567" # Optional: filter by bot ID
},
slack_channel_id="C01234567",
api_key="your-api-key",
runbook={
"investigation_prompt": "Investigate database issues and provide remediation steps"
},
notification_integration_ids=["R3JhcGhRTE5vdGlmaWNhdGlvbkludGVncmF0aW9uOjEyMw=="]
)
print(f"Created Slack alert agent: {slack_alert['url']}")
# Get alert details
alert_details = get_alert_responder(webhook_alert["id"], "your-api-key")
print(f"Alert status: {alert_details['status']}")
# List all alert agents
all_alerts = list_alert_responders("your-api-key")
print(f"Total alert agents: {len(all_alerts['alert_responders'])}")
# Update with text_matches (simple wildcards)
updated = update_alert_responder(
webhook_alert["id"],
"your-api-key",
name="Updated Critical Alert",
matching_criteria={"text_matches": ["*critical*", "*emergency*"]}
)
print(f"Updated alert agent: {updated['name']}")
# Update with match_rules (structured AND/OR logic)
updated_rules = update_alert_responder(
webhook_alert["id"],
"your-api-key",
matching_criteria={
"match_rules": {
"operator": "OR",
"conditions": [
{"type": "contains", "value": "stg/"},
{"type": "contains", "value": "release/"}
]
}
}
)
print(f"Updated alert agent with match_rules: {updated_rules['name']}")
# Update Slack alert agent
updated_slack = update_alert_responder(
slack_alert["id"],
"your-api-key",
slack_channel_id="C98765432",
matching_criteria={
"text_matches": ["*critical*", "*database*"],
"slack_bot_app_user_id": "B99999999"
}
)
print(f"Updated Slack alert agent: {updated_slack['name']}")
# Disable alert agent temporarily
disable_alert_responder(webhook_alert["id"], "your-api-key")
print("Alert agent disabled")
# Re-enable it
enable_alert_responder(webhook_alert["id"], "your-api-key")
print("Alert agent enabled")
# Digest Agent Management Functions
def create_digest(team_name, name, sections, schedule, api_key, slack_channel_id=None):
"""Create a new digest agent job"""
url = "https://api.tierzero.ai/api/v1/digests"
headers = {
"X-TierZero-Org-Api-Key": api_key,
"Content-Type": "application/json"
}
payload = {
"team_name": team_name,
"name": name,
"sections": sections,
"schedule": schedule
}
if slack_channel_id:
payload["slack_channel_id"] = slack_channel_id
response = requests.post(url, headers=headers, json=payload)
response.raise_for_status()
return response.json()
def get_digest(digest_id, api_key):
"""Get digest agent details by digest ID"""
url = f"https://api.tierzero.ai/api/v1/digests/{digest_id}"
headers = {
"X-TierZero-Org-Api-Key": api_key
}
response = requests.get(url, headers=headers)
response.raise_for_status()
return response.json()
def update_digest(digest_id, api_key, name=None, sections=None, schedule=None, slack_channel_id=None):
"""Update an existing digest agent"""
url = f"https://api.tierzero.ai/api/v1/digests/{digest_id}"
headers = {
"X-TierZero-Org-Api-Key": api_key,
"Content-Type": "application/json"
}
payload = {}
if name:
payload["name"] = name
if sections:
payload["sections"] = sections
if schedule:
payload["schedule"] = schedule
if slack_channel_id:
payload["slack_channel_id"] = slack_channel_id
response = requests.put(url, headers=headers, json=payload)
response.raise_for_status()
return response.json()
def run_digest(digest_id, api_key, run_sync=False, scheduled_runtime=None):
"""
Manually run a digest agent job immediately or schedule it for later.
Args:
digest_id: Digest Agent ID
api_key: Organization API key
run_sync: If True, runs synchronously and returns results
scheduled_runtime: Optional Unix timestamp (seconds) to schedule the run
Note: Notifications are always sent to configured channels/webhooks
"""
url = f"https://api.tierzero.ai/api/v1/digests/{digest_id}/run"
headers = {
"X-TierZero-Org-Api-Key": api_key,
"Content-Type": "application/json"
}
payload = {}
if run_sync:
payload["run_sync"] = run_sync
if scheduled_runtime:
payload["scheduled_runtime"] = scheduled_runtime
response = requests.post(url, headers=headers, json=payload)
response.raise_for_status()
return response.json()
def delete_digest(digest_id, api_key):
"""Delete a digest agent job"""
url = f"https://api.tierzero.ai/api/v1/digests/{digest_id}"
headers = {
"X-TierZero-Org-Api-Key": api_key
}
response = requests.delete(url, headers=headers)
response.raise_for_status()
return response.json()
# Scheduled Actions Functions
def create_scheduled_action(
question,
scheduled_runtimes,
api_key,
slack_channel_ids=None,
discord_webhook_urls=None
):
"""
Create a scheduled action that runs a TierZero agent at specific times.
Args:
question: Question/prompt to send to TierZero agent
scheduled_runtimes: List of Unix timestamps (seconds) when action should run
api_key: Organization API key
slack_channel_ids: Optional list of Slack channel IDs for notifications
discord_webhook_urls: Optional list of Discord webhook URLs for notifications
Note:
Notification targets are optional. If not provided, query results using GET /api/v1/interactions/{interaction_id}.
Returns:
Dictionary with:
- interactions (list of dicts): One entry per scheduled runtime
- status (str): SCHEDULED or ACCEPTED
Each interaction dict contains: interaction_id, scheduled_runtime, and url.
"""
url = "https://api.tierzero.ai/api/v1/interactions"
headers = {
"X-TierZero-Org-Api-Key": api_key,
"Content-Type": "application/json"
}
payload = {
"question": question,
"scheduled_runtimes": scheduled_runtimes
}
# Build nested notifications structure
notifications = {}
if slack_channel_ids:
notifications["slack"] = {"slack_channel_ids": slack_channel_ids}
if discord_webhook_urls:
notifications["discord"] = {"discord_webhook_urls": discord_webhook_urls}
if notifications:
payload["notifications"] = notifications
response = requests.post(url, headers=headers, json=payload)
response.raise_for_status()
return response.json()
# Scheduled Action usage example
scheduled_action_result = create_scheduled_action(
question="Analyze system performance metrics from the last 7 days and identify any anomalies or trends",
scheduled_runtimes=[1735689600, 1735776000], # Unix timestamps for when to run
api_key="your-api-key",
slack_channel_ids=["C01234567"]
)
print(f"Status: {scheduled_action_result['status']}") # "SCHEDULED"
print(f"Created {len(scheduled_action_result['interactions'])} scheduled interactions:")
for interaction in scheduled_action_result['interactions']:
print(f" - ID: {interaction['interaction_id']}")
print(f" Scheduled at: {interaction['scheduled_runtime']}")
print(f" URL: {interaction['url']}")
# Store all interaction_ids for tracking individual executions
interaction_ids = [i['interaction_id'] for i in scheduled_action_result['interactions']]
# Digest Agent usage example
digest_result = create_digest(
team_name="Engineering",
name="Daily Production Health",
sections=[
{
"section": "Error Summary",
"prompt": "Summarize production errors from the last 24 hours"
},
{
"section": "Performance Metrics",
"prompt": "Report on API response times and throughput"
}
],
schedule="0 9 * * 1-5",
api_key="your-api-key",
slack_channel_id="C01234567"
)
print(f"Created digest: {digest_result['url']}")
# Run the digest immediately and asynchronously
run_digest(digest_result["id"], "your-api-key")
# Run the digest synchronously to get immediate results
run_result = run_digest(
digest_result["id"],
"your-api-key",
run_sync=True
)
for section_result in run_result.get("results", []):
print(f"\n{section_result['section']}:")
print(section_result['result'])
# Schedule the digest to run at a specific time (e.g., tomorrow at 9 AM)
import time
from datetime import datetime, timedelta
tomorrow_9am = datetime.now() + timedelta(days=1)
tomorrow_9am = tomorrow_9am.replace(hour=9, minute=0, second=0, microsecond=0)
scheduled_result = run_digest(
digest_result["id"],
"your-api-key",
scheduled_runtime=int(tomorrow_9am.timestamp())
)
print(f"Digest Agent scheduled for: {datetime.fromtimestamp(scheduled_result['scheduled_runtime'])}")
# Billing Functions
def get_credit_usage(start, end, api_key):
"""
Get credit usage grouped by team and month.
Args:
start: Start date in ISO 8601 format (e.g., "2026-05-01T00:00:00Z")
end: End date in ISO 8601 format (e.g., "2026-07-01T00:00:00Z")
api_key: Organization API key
Returns:
Dictionary with organization_id, start, end, data (per-team/month credits),
and monthly_totals
"""
url = "https://api.tierzero.ai/api/v1/billing/credit-usage"
headers = {
"X-TierZero-Org-Api-Key": api_key,
"Content-Type": "application/json"
}
response = requests.post(url, headers=headers, json={"start": start, "end": end})
response.raise_for_status()
return response.json()
# Billing usage example
usage = get_credit_usage("2026-05-01T00:00:00Z", "2026-07-01T00:00:00Z", "your-api-key")
print(f"Monthly totals: {usage['monthly_totals']}")
for row in usage["data"]:
team = row["team_name"] or "(Unattributed)"
print(f" {row['month']} | {team}: {row['credits']} credits")
# Interaction Management Functions
def create_interaction_sync(question, api_key, context=None, user_id=None, use_tools=True):
"""Create a synchronous interaction"""
url = "https://api.tierzero.ai/api/v1/interactions"
headers = {
"X-TierZero-Org-Api-Key": api_key,
"Content-Type": "application/json"
}
payload = {"question": question, "use_tools": use_tools}
if context:
payload["context"] = context
if user_id:
payload["user_id"] = user_id
response = requests.post(url, headers=headers, json=payload)
response.raise_for_status()
return response.json()
def create_interaction_async(question, api_key, context=None, user_id=None, use_tools=True):
"""Create an asynchronous interaction"""
url = "https://api.tierzero.ai/api/v1/interactions"
headers = {
"X-TierZero-Org-Api-Key": api_key,
"Content-Type": "application/json"
}
payload = {"question": question, "async": True, "use_tools": use_tools}
if context:
payload["context"] = context
if user_id:
payload["user_id"] = user_id
response = requests.post(url, headers=headers, json=payload)
response.raise_for_status()
return response.json()
def get_interaction_results(interaction_id, api_key):
"""Get interaction results by ID"""
url = f"https://api.tierzero.ai/api/v1/interactions/{interaction_id}"
headers = {
"X-TierZero-Org-Api-Key": api_key,
"Accept": "application/json"
}
response = requests.get(url, headers=headers)
response.raise_for_status()
return response.json()
def wait_for_completion(interaction_id, api_key, timeout=60, poll_interval=2):
"""Poll for interaction completion"""
start_time = time.time()
while time.time() - start_time < timeout:
result = get_interaction_results(interaction_id, api_key)
if result.get("status") == "COMPLETED":
return result
elif result.get("status") == "FAILED":
raise Exception(f"Interaction failed: {interaction_id}")
print(f"Status: {result.get('status')}, waiting...")
time.sleep(poll_interval)
raise TimeoutError(f"Interaction {interaction_id} did not complete within {timeout} seconds")
# Synchronous usage
result = create_interaction_sync(
"What are the top error patterns in production today?",
"your-api-key",
user_id="analyst123"
)
print(result["content"])
# Asynchronous usage
async_result = create_interaction_async(
"Analyze the performance impact of our latest deployment",
"your-api-key",
user_id="analyst123"
)
print(f"Started async interaction: {async_result['interaction_id']}")
# Wait for completion
final_result = wait_for_completion(async_result["interaction_id"], "your-api-key")
print(final_result["content"])
# Request without tools (knowledge-only response)
knowledge_result = create_interaction_sync(
"What is our standard deployment process?",
"your-api-key",
use_tools=False
)
print(knowledge_result["content"])
```
### Node.js
```javascript theme={null}
const axios = require('axios');
async function createInteraction(question, apiKey, options = {}) {
const url = 'https://api.tierzero.ai/api/v1/interactions';
const payload = {
question,
...options
};
try {
const response = await axios.post(url, payload, {
headers: {
'X-TierZero-Org-Api-Key': apiKey,
'Content-Type': 'application/json'
}
});
return response.data;
} catch (error) {
console.error('TierZero API Error:', error.response?.data || error.message);
throw error;
}
}
async function getInteractionResults(interactionId, apiKey) {
const url = `https://api.tierzero.ai/api/v1/interactions/${interactionId}`;
try {
const response = await axios.get(url, {
headers: {
'X-TierZero-Org-Api-Key': apiKey,
'Accept': 'application/json'
}
});
return response.data;
} catch (error) {
console.error('TierZero API Error:', error.response?.data || error.message);
throw error;
}
}
// Usage
createInteraction(
"Show me recent deployment failures",
"your-api-key",
{ user_id: "devops-team" }
).then(result => {
console.log(result.content);
// Later, retrieve the same results using the interaction ID
return getInteractionResults(result.interaction_id, "your-api-key");
}).then(retrievedResult => {
console.log(retrievedResult.content);
});
```