MTTR, MTTD, MTBF: The Incident Metrics That Actually Matter
MTTR dropped from 40 min to 10 min. But that's only 70% of the picture. The real win: engineers sleeping through on-call shifts. Mean time metrics are the most tracked reliability numbers in engineering - and the most misunderstood. This guide covers what each one actually measures, how to calculate them correctly, and how to use them to drive real improvement instead of just better-looking dashboards.
Jake Davids
MTTR, MTTD, MTBF: The Incident Metrics That Actually Matter
MTTR dropped from 40 min to 10 min. But that's only 70% of the picture. The real win: engineers sleeping through on-call shifts.
Mean time metrics are the most widely tracked reliability numbers in engineering - and the most misunderstood. Teams track MTTR religiously without agreeing on what "resolution" means. They celebrate MTBF improvements while the same incident keeps happening. They ignore MTTD entirely, then wonder why MTTR won't come down.
This guide covers what each metric actually measures, how to calculate them correctly, and - more importantly - how to use them to drive real reliability improvement.
The MTTx Family: A Quick Reference
| Metric | Full Name | What It Measures |
|---|---|---|
| MTTD | Mean Time to Detect | How long before you know something is wrong |
| MTTA | Mean Time to Acknowledge | How long before someone starts working on it |
| MTTR | Mean Time to Resolve/Repair/Restore | How long to fix it (varies by definition - see below) |
| MTBF | Mean Time Between Failures | How often failures occur |
| MTBR | Mean Time Between Repairs | Recovery time between maintenance events |
Most discussions focus on MTTR and MTBF. MTTD is the underappreciated metric that explains why the others don't improve.
MTTD: Mean Time to Detect
MTTD measures the time between when a problem starts and when your monitoring detects it.
MTTD = Time of detection - Time failure began
Where "time failure began" is when the system actually started misbehaving - not when the alert fired.
Why MTTD is often invisible
Most teams don't measure MTTD because it requires knowing when failures actually start, which isn't always obvious from alerts. Your error rate alert fires at 10:47pm. But the errors started at 10:31pm at a low rate that didn't cross your threshold. Your MTTD for that incident was 16 minutes, not 0.
Alert thresholds that are too conservative (set high to avoid false positives) increase MTTD. Every minute of MTTD adds directly to customer-facing impact time.
Improving MTTD
The levers for improving MTTD:
Lower alert thresholds - carefully. Threshold reduction increases alert noise. The right approach is tiered alerting: low threshold triggers a Slack notification (investigate when available), higher threshold triggers a page (wake someone up). This improves detection time without increasing on-call burden.
SLO-based burn rate alerts. Rather than alerting on a fixed error rate, alert when your error budget is burning significantly faster than normal. A 2% error rate might not be page-worthy. A 2% error rate that will exhaust your monthly SLO budget in 8 hours is.
Synthetic monitoring. Actively test critical user flows every minute. Synthetic monitors catch failures that real traffic patterns might not generate at sufficient volume to trigger threshold alerts immediately.
Anomaly detection. Statistical baselines for metrics can detect unusual patterns before they cross hard thresholds. Modern monitoring tools including Datadog have built-in anomaly detection - use it for metrics where you have enough history to establish a reliable baseline.
MTTA: Mean Time to Acknowledge
MTTA measures from alert firing to an engineer acknowledging the incident.
MTTA = Time of acknowledgement - Time of alert
High MTTA usually indicates one of three things: the alert fired outside business hours and the on-call rotation isn't working, the on-call engineer was alert-fatigued and deprioritized the page, or the escalation chain has a gap.
MTTA benchmarks
Most mature teams target:
- P1 incidents: MTTA under 5 minutes, 24/7
- P2 incidents: MTTA under 15 minutes during business hours
- P3 incidents: MTTA same business day
MTTA above 15 minutes for P1 incidents suggests an on-call process problem - either the paging isn't working, or engineers aren't treating P1 alerts as P1.
MTTR: The Most Misunderstood Metric
MTTR is the most tracked reliability metric and the one with the most inconsistent definition. Before comparing MTTR numbers between teams, make sure you're comparing the same thing.
The three definitions of MTTR
Mean Time to Restore - the most common definition: time from incident detection to service restoration. This is the customer-facing impact time. Service is restored when users can use it normally again, even if the root cause hasn't been fully diagnosed.
Mean Time to Repair - time from incident detection to permanent fix. This includes root cause investigation and a lasting solution, not just a rollback or workaround.
Mean Time to Recover - sometimes used interchangeably with restore, sometimes defined as time from failure start (not detection) to restoration. Be explicit about which you mean.
Most SLO frameworks use Mean Time to Restore - because customer impact is what matters for reliability targets. Mean Time to Repair is more useful for postmortem tracking and reliability improvement.
Calculating MTTR correctly
MTTR = Sum of all incident durations / Number of incidents
Where incident duration = time of resolution - time of detection.
For meaningful averages, calculate MTTR separately by severity level. Aggregating P1 and P4 incidents produces a number that reflects neither. Your P1 MTTR is the number that matters most - it represents your worst customer-facing incidents.
What makes MTTR high
MTTR breaks down into components that point to different interventions:
Context gathering time (20-30 min in poorly tooled teams): The time from alert to starting actual diagnosis. Engineer checks Datadog for the metrics, GitHub for recent deployments, Slack for what the team knows, PagerDuty for the escalation history, the runbook for known issues.
This is the component with the most room for improvement and the clearest tooling solution. When context is pre-assembled, this drops from 20-30 minutes to 2-3 minutes. OpsBrief pulls together the Datadog metrics, GitHub deployments, Slack context, and relevant runbooks automatically when an incident opens - cutting this component dramatically.
Diagnosis time (varies widely): Once the engineer has context, how long does it take to identify the root cause? This depends on system complexity, engineer experience, and tooling quality. Dependency graphs help here - knowing which services are affected and what they depend on turns a complex multi-service incident into a traceable chain.
Remediation time: The actual fix - rolling back a deployment, scaling up a service, clearing a queue. Often the fastest component for well-prepared teams.
Verification time: Confirming the fix worked and the system is stable. Frequently skipped under time pressure, which leads to premature resolution calls and incidents re-opening.
MTTR targets by severity
Industry benchmarks vary widely by organization size and system complexity. Realistic targets for mature SRE practices:
| Severity | Target MTTR |
|---|---|
| P1/SEV-1 | Under 15 minutes |
| P2/SEV-2 | Under 1 hour |
| P3/SEV-3 | Under 4 hours |
| P4/SEV-4 | Under 24 hours |
If your P1 MTTR is currently 40-50 minutes (a common baseline), the context gathering component is usually the largest opportunity. Cutting that from 25 minutes to 3 minutes gets you to 20 minutes. Better diagnostic tooling and runbooks get you the rest of the way.
MTBF: Mean Time Between Failures
MTBF measures the average time between incidents - a measure of how often failures occur.
MTBF = Total operational time / Number of failures
High MTBF means failures are infrequent. Low MTBF means the same systems are failing repeatedly.
When MTBF is misleading
MTBF improvement is easily gamed. If you raise your alert thresholds, fewer things count as incidents, and MTBF goes up - even though the system's reliability hasn't improved.
More importantly, MTBF averages obscure patterns. An average of 14 days between failures sounds reasonable. But if you're having 3 failures in week 1 and 0 in weeks 2-4, you don't have a random failure distribution - you have a recurring problem triggered by something specific (weekly deployment? Batch job? Traffic pattern?).
MTBF is most useful as a trend indicator over time, and most useful when broken down by service rather than aggregated.
The relationship between MTBF and MTTR
The total time your system is unavailable is approximately:
Unavailability = Incident frequency x Average MTTR
(1 / MTBF) x Average incident duration
You can improve availability by improving either MTBF (fewer incidents) or MTTR (shorter incidents). Both matter, but they require different investments.
Improving MTTR is usually faster - it's primarily a tooling and process problem. Improving MTBF requires identifying and fixing root causes of recurring failures - a harder, more systemic problem.
Teams that focus exclusively on MTTR often plateau because they're optimizing response to recurring incidents rather than eliminating them.
The Metric That Connects Them: Recurring Incident Rate
Most teams track MTTR and MTBF separately without looking at the relationship between them.
The recurring incident rate - what percentage of your incidents are the same incident you've had before - connects both metrics:
- High recurring incident rate + low MTTR: Team is fast at fixing incidents but not preventing them. Good at firefighting, not prevention.
- Low recurring incident rate + high MTTR: Incidents are novel and complex. Investment in better tooling and runbooks will help.
- High recurring incident rate + high MTTR: The same incidents keep happening and take a long time to fix. This is where the most improvement potential exists.
Identifying recurring incidents requires tracking incident patterns over time, not just individual incidents. OpsBrief's heat map shows which services fail most frequently and which incidents share the same root cause signature - making the recurring incident pattern visible and actionable.
Teams that act on recurring incident patterns typically reduce their incident volume by 30-50%. That's not a MTTR improvement - it's a MTBF improvement that makes MTTR almost irrelevant for those failure modes because they stop occurring.
Postmortem Metrics: Are They Working?
The purpose of postmortems is to improve MTBF and prevent recurrence. Whether they're working is measurable:
Postmortem action item completion rate: What percentage of postmortem action items actually get completed? Industry data suggests this is below 30% for most teams. Uncompleted action items are the mechanism through which recurring incidents happen.
Time to postmortem: How long after an incident does the postmortem happen? Postmortems lose value quickly - details fade, context is lost, the team moves on. Target: postmortems within 48 hours of resolution.
Recurrence rate for post-mortemed incidents: If an incident has a postmortem and action items, does it happen again? If yes, the postmortem didn't work.
The most common postmortem failure mode is not the quality of the analysis - it's the time cost of writing it. When postmortems take 90 minutes of engineering time, they get deprioritized. When they take 10 minutes because the timeline is already captured automatically, they happen consistently.
OpsBrief's auto-timeline capture records every action, decision, deployment, and metric change during an incident. By the time the incident is resolved, the postmortem timeline is already written. The team's job is analysis and action items, not reconstruction.
Building an Incident Metrics Dashboard
The metrics worth tracking on a regular basis:
Weekly:
- P1 MTTR (trend and individual incidents)
- MTTA for P1 incidents
- Total incident count by severity
Monthly:
- MTTR by severity (trend)
- MTBF by top services
- Recurring incident rate
- Error budget status (consumed vs. remaining)
- On-call shift quality (alerts received, actions required vs. noise)
Quarterly:
- Postmortem completion rate
- Action item completion rate
- Year-over-year reliability trends
- On-call satisfaction (surveyed)
The incident calendar in OpsBrief provides a visual timeline of incidents over time - useful for spotting seasonal patterns, deployment-related clusters, and service-specific trends that are invisible in aggregate metrics.
The Number That Matters Most
MTTR is the metric most teams optimize because it's the most visible. But if you have to pick one metric to improve reliability:
Reduce your recurring incident rate.
A team with 40-minute MTTR and 30% recurring incidents will have worse reliability outcomes than a team with 45-minute MTTR and 5% recurring incidents. The latter team is spending their time on novel problems and learning from each one. The former is repeatedly fixing the same problems faster and faster.
Fix the pattern, and MTTR becomes less important because the incidents that drive it become less frequent. Fix only MTTR, and you get faster at a problem you should be eliminating.
Want visibility into your incident patterns, recurring failures, and on-call metrics in one place? OpsBrief surfaces heat maps, incident timelines, and MTTR trends across your Datadog, PagerDuty, and GitHub data - without requiring manual incident logging.
