Anthropic Put Claude in the On-Call Channel: When Systems Alert at Night, It Investigates First
Manual investigation often took more than an hour. Claude Tag now stays in Slack 24/7 and delivers its first evidence-backed analysis in a median of 14 minutes.
- Claude Tag takes over the continuous evidence gathering and initial reporting after a CI/CD alert fires, while humans keep control over production changes.
What's different about traditional on-call vs. Claude as first responder
Anthropic engineer Sachin Malhotra shared a CI/CD on-call setup his team has been running for months. The idea: keep Claude Tag in the Slack on-call and alerting channels, where it acts as the 24/7 first responder.
It's not just speed — it's who keeps doing the first round of investigation
Pick a mode to focus; side-by-side is the default.
- Passive interruptionPaged at night, forced to context-switch immediately.
- System-by-systemManually opening dashboards, logs, code, deploy history, and the incident channel.
- The threshold trapToo sensitive means alert fatigue; too loose and incidents slip through.
- Always on, 24/7Sits in Slack on-call and alerting channels, picking up the first signal automatically.
- Search in parallelMultiple agents checking dashboards, logs, code, cluster state, and past incidents at once.
- Delivered with receiptsOutputs a timeline, root cause hypotheses, and recommendations for the engineer to judge.
Anthropic reports that Claude's median time to deliver an evidence-based analysis is 14 minutes. The fastest case pinpointed the root cause within 4 minutes of the initial alert. That's the first-round investigation result, not the time to full incident resolution.
One real incident started at 10 p.m., when roughly 44 tests in a new service never ran. Claude traced the issue to a read-mode feature flag that had been enabled that day. A human performed the rollback; 3 minutes later, Claude re-checked and confirmed the rules were restored and error rates were back to baseline.
It's not a bot — it's an on-call system
Claude's ability to stay on call doesn't come from a single universal prompt. It depends on channel context, Git knowledge, tool integrations, and multi-agent investigation working together.
It remembers, can look things up, comes back on schedule, and acts by rules
Keeps on-call channel context and reacts to incidents and human prompts in real time.
Reads dashboards, logs, pager, code, cluster state, and the incident channel.
Re-runs on events or a schedule, e.g. generating a handoff every Monday.
Puts investigation, escalation, routing, and learnings under Git review.
Investigation methods and incident experience are stored in Git. lessons.md records root cause, fix, and takeaways from each incident; different failure modes have their own runbooks, and shadow divergence failures even have a 617-line investigation skill.
What's worth copying isn't the 617-line count itself — it's how that skill was created. Engineers worked through real incidents step by step with Claude, then turned the process into a skill afterward. One-off lessons go into lessons.md first; only when the same pattern repeats does it get promoted to a formal runbook. The most important principle inside: check the data before forming theories. Configuration tells you where things might go wrong, but only metrics tell you what actually happened.
How Claude handles an incident
Triage first, then gather evidence; assist the fix, then verify and learn
Click through the four phases to see what it reads, does, and hands off.
Alert channels, human reports, internal incident pages, plus traffic and alert data after a new service ships.
Deterministic monitoring triggers the signal; Claude uses ONCALL.md and live context to judge severity and escalation path.
Either pages someone immediately or writes a morning log entry for daytime follow-up.
lessons.md, failure playbooks, Grafana, logs, PagerDuty, GitHub, Kubernetes, Slack.
Execution agents each read their source of truth; data before speculation. A human can jump in anytime with counterexamples or extra hypotheses.
A SITREP with evidence links: what happened, likely root cause, and what to do next.
A verified root cause hypothesis, feature flags, cluster state, and code or config diffs.
A high-privilege internal agent can ramp or roll back canaries; Claude can also suggest drain/cordon, scaling, or open a fix PR.
A reviewable action plan or PR; the on-call engineer owns any production change.
The same set of dashboards, logs, tests, and the incident thread, post-fix.
Confirms metrics are back to baseline, records root cause and fix, and promotes repeating patterns into playbooks.
Verification results, lessons.md updates, daily and weekly handoffs, plus the ci-weather public status.
There are two key boundaries here. First, monitoring and alerts remain deterministic systems; the agent adds context to decide whether to escalate. Second, Claude won't always be right on the first pass — humans can raise counterexamples or add hypotheses at any time, sending it back to the data to verify again.
These four steps form a complete loop: triage and escalate alerts, investigate evidence in parallel, assist with fixes, then confirm metrics are healthy and write the experience back. The ci-weather tool also aggregates multiple incidents, build metrics, merge queues, and deployment delays into a shared status report, so other engineers don't need to keep asking whether CI is okay.
That status report isn't a one-and-done Claude output, though. The Anthropic team iterated on the format repeatedly, because whether the report reads well depends on the team's own communication habits — that's a human communication problem, not just a technical pipeline. Claude also generates daily and weekly handoffs so the next on-call can pick things up.
One permission boundary should be made explicit: Anthropic also has internal Claude Code Agents with engineer-level access that can automatically adjust canary traffic for feature flags. The public oncall-kit is read-only by default, and production changes still require a human.
How other teams can get started
Anthropic says the basic setup took a few hours, not days. The shortest path has four steps:
- Enable Claude Team or Enterprise and add Claude Tag to the Slack on-call channel;
- Connect monitoring, logs, PagerDuty, GitHub, Kubernetes, and other connectors, and configure Claude Code Remote;
- Generate
lessons.md, runbooks, and escalation rules from your team's own historical incidents; - Start with read-only Claude investigation and diagnostic output, then add permissions only after humans verify reliability.
The official setup kit also includes a ~10-minute fictitious incident drill, so you can understand the flow without connecting production systems.
In the end, what this saves engineers isn't just a few manual queries. It removes mechanical investigation, nighttime interruptions, and repetitive incident communication, giving people time back for longer-term architecture and reliability work.
System alert at night? Claude starts investigating first
Anthropic keeps Claude Tag in its CI/CD Slack channels. It triages alerts, investigates in parallel, and files the first evidence report; humans retain production control. Median: 14 minutes. Fastest root-cause finding: 4 minutes.
Claude gathers evidence before waking a human
The old wayWakes an engineer, who checks each system in sequence. Investigations can exceed an hour.
Claude first responseAlways in Slack; several agents search at once. Median first evidence report: 14 minutes.
Fastest caseRoot cause in the first report within 4 minutes—not full resolution.
Claude does the mechanical digging; humans step in for production decisions.
Four layers make it an on-call system
ClaudeSITREP
Channel contextSlack supplies alerts, discussion, and human counterexamples.
Evidence linksGrafana, Datadog, Kubernetes, PagerDuty, and GitHub supply live data.
Git knowledge layerOne-off lessons enter lessons.md; recurring patterns become investigation skills.
Parallel investigationAgents check metrics, logs, and changes in parallel; the lead agent writes the SITREP.
Metrics first; configuration and theories second.
One loop: detect, investigate, fix, verify, learn
11 DetectRule-based monitoring alerts; Claude pages now or logs it for the morning.
22 InvestigateRead recent lessons, then check metrics, logs, code, and changes in parallel.
33 FixClaude suggests fixes or PRs; humans approve and execute production changes.
44 Verify and learnRecheck metrics; write root cause and lessons back into lessons.md.
Claude can be wrong; humans add counterexamples and send it back to the data.
44 tests skipped: Claude finds, a human rolls back, Claude checks
10 p.m. alertAbout 44 tests stop running.
Root cause foundClaude points to that day's read-mode feature flag.
Human applies the fixThe engineer rolls it back; the public kit does not change production.
3 minutes laterClaude confirms the rules, tests, and error rate are back to baseline.
Copy the investigation loop first, not the production keys
Do this nowConnect Slack and read-only tools; turn incident history into lessons, runbooks, and escalation rules.
Stay read-onlyLet Claude diagnose; humans verify. The kit includes a roughly 10-minute drill.
Earn permissions graduallyInternal agents can adjust canary traffic; the public kit is read-only, and humans execute production changes.
Transferable pattern: rule-based alerts, contextual investigation, human production control, and lessons that feed the next response.
Nighttime alert: who starts investigating first?
Alert44 tests missing for the new service!
ClaudeLet me check the evidence first.
Night calls used to wake a human. Claude responds first now.
Scouts gather evidence in parallel; the lead agent merges it into a SITREP.
ClaudeThis is the first evidence analysis, not a completed fix.
EngineerCheck that counterexample too.
Claude can report early—and can be wrong. The human can always pull the investigation back to data.
ClaudeEvidence points to the read-mode flag turned on today. Suggest rollback.
EngineerI'll execute the production change.
Claude finds and suggests; humans own production calls.
Close the loop by rechecking metrics after the fix.
ClaudeLog the incident first; promote it to a handbook after repeats.
EngineerStart read-only; expand access only after it proves reliable.
What transfers is not automated production changes, but evidence gathering, human gates, and lessons that feed the next response.