Ops & Chief of Staff · Detect & Respond
Incident context assembled before the engineer opens their laptop
When a P1 fires, Doe pulls the PagerDuty alert, correlates Datadog metrics and deploy diffs, maps the blast radius via New Relic, and delivers a brief with the likely cause and rollback steps within 2 minutes.
Works acrossPagerDutyDatadogNew Relic
What you get.
Alert details from PagerDuty, correlated metrics from Datadog, and service health from New Relic assembled into an incident brief with the likely cause, blast radius, and suggested rollback steps before the on-call engineer starts investigating.
The first 30 minutes of every incident are wasted on context-gathering
PagerDuty fires at 2 AM. The on-call engineer opens Datadog, then New Relic, then checks the deploy log. They cross-reference timestamps, look for correlated service errors, and try to figure out what changed. Meanwhile, the service is down and the incident channel is filling up with "any update?" messages.
The last P1 took 47 minutes to resolve. 28 of those minutes were spent gathering context: which service, what changed, when it started. The actual fix took 19 minutes.
What changes.
- 01Context-gathering timeBefore · 20-30 minutes opening tabs and cross-referencing timestampsWith Doe · Incident brief ready within 2 minutes of the PagerDuty alert
- 02Deploy correlationBefore · Engineer manually checks deploy logs for recent changesWith Doe · Recent deploys correlated with metric changes automatically
- 03Blast radius visibilityBefore · Downstream services discovered as they start failingWith Doe · Affected downstream services identified in the initial brief
- 04Runbook accessBefore · Engineer searches Confluence for the right runbook mid-incidentWith Doe · Relevant runbook linked in the brief based on the service and failure type
How Doe builds the incident brief
- 01Reads the P1 alert with service, escalation, and historyPagerDutyP1 alert fired at 2:14 AM. Service: payment-api. Escalation policy: on-call SRE (J. Park). Alert details: "payment-api response time exceeded 5s threshold for 3 consecutive checks." Previous incident on this service: 11 days ago (bad config deploy).
- 02Pulls metrics, error rates, and recent deploys for the affected serviceDatadogpayment-api CPU spiked to 94% at 2:11 AM (baseline: 35%). Error rate jumped from 0.2% to 12.4%. Last deploy to payment-api: 2:08 AM by deploy-bot (commit: "add retry logic to payment processor"). 2 other services showing elevated error rates: checkout-api and billing-service (both downstream of payment-api).
- 03Checks service health, transaction traces, and upstream dependenciesNew Relicpayment-api Apdex dropped to 0.12 (normal: 0.94). Transaction trace shows the retry loop in processPayment() averaging 8.3s per call (normal: 120ms). Upstream dependency map confirms checkout-api and billing-service are healthy independently but failing on calls to payment-api.
- 04Assembles the incident brief with cause, blast radius, and next stepsDoeIncident brief: the payment-api deploy at 2:08 AM introduced a retry loop in processPayment() consuming CPU and cascading errors to checkout-api and billing-service. Suggested action: roll back commit abc123. Deploy diff, Datadog and New Relic dashboards, and the payment-api rollback runbook linked.
- 05RecurringTriggered by PagerDuty alertEvery P1 triggers a brief within 2 minutes: alert context, correlated metrics, deploy diffs, blast radius, and rollback steps. Incident brief available in Doe.
Up and running in under ten minutes.
- 01Connect your toolsOne-click OAuth for each integration. No API keys, no engineering.
- 02Describe what you need“When a P1 alert fires in PagerDuty, pull the alert details, grab CPU, error rate, and recent deploys from Datadog, check service health in New Relic, and assemble an incident brief with the likely cause, affected services, and suggested rollback steps. Link the deploy diff and the relevant runbook.”
- 03It runs on scheduleTriggered on every P1 PagerDuty alert. Brief available within 2 minutes.
Before you delegate.
- 01How fast does the brief appear after the alert fires?Typically within 2 minutes. The bottleneck is API response times from Datadog and New Relic, not processing. Most briefs are ready before the on-call engineer finishes reading the PagerDuty notification.
- 02What if the root cause is not a deploy?Doe still provides the brief with metric anomalies, error rate changes, and service dependencies. If no recent deploy correlates, it says so and surfaces other potential causes like traffic spikes or upstream failures.
- 03Does Doe take any automated remediation action?No. Doe assembles context and suggests next steps, but it does not roll back deploys or restart services. The on-call engineer makes every remediation decision.
- 04What data sources does the brief check?PagerDuty for alert details and escalation history, Datadog for metrics and deploy events, and New Relic for service health and transaction traces. You can add additional sources like AWS CloudWatch or custom deploy tracking systems.
- 05Can it handle multiple simultaneous incidents?Yes. Each P1 alert generates its own brief independently. If two services fail at the same time, Doe also cross-references them to flag whether they share a common cause.