Skip to content

Toil in SRE: What It Is and How to Reduce It (2026)

Quick answer: In site reliability engineering (SRE), toil is the manual, repetitive operational work that keeps a service running but leaves it no better afterward, and that grows as the service grows. Google's SRE book defines it by six traits: manual, repetitive, automatable, tactical, without enduring value, and scaling linearly with service growth. Google caps toil at 50% of each SRE's time. Toil reduction means measuring this work, then removing it through automation, better alerting and self-service, so engineers spend their time on work that improves the system.

Restart the stuck worker. Rotate the expiring certificate. Approve another access request. Acknowledge the disk-space alert that fires every Tuesday and resolves itself by Wednesday. None of these tasks is hard. That's the problem: they're easy enough to keep doing by hand, and frequent enough to quietly consume a team.

In site reliability engineering, that kind of work has a name: toil. It's not a synonym for "boring work" or "operations." Google gave it a precise definition so that teams could measure it, cap it and engineer it away.

This guide covers Google's definition, what toil is and isn't, examples by area, the 50% cap, how to measure toil, and how to reduce it, with a separate section on on-call toil, where most teams feel it first.

What is toil in SRE?

Google's Site Reliability Engineering book (Chapter 5, written by Vivek Rau) defines toil as operational work tied to running a production service that shows six traits: it's manual, repetitive, automatable and tactical, it leaves no enduring value, and it grows linearly with the service. A task doesn't need all six traits to count, but the more it has, the more clearly it's toil.

Trait

What it means

Quick test

Manual

A human runs it, even if that means running a script by hand

Does someone have to start it?

Repetitive

It's done again and again, not once

Have you done this more than twice this quarter?

Automatable

A machine could do it as well as a person

Does it need human judgment, or just human hands?

Tactical

It's interrupt-driven and reactive, not planned

Did a page, ticket or request trigger it?

No enduring value

The service is in the same state afterward

Is anything permanently better once it's done?

Scales with the service

Twice the traffic or users means twice the work

Will this grow if the service doubles?

Toil vs. overhead vs. engineering

The most common mistake is calling all unpleasant work "toil." Google separates three kinds of work:

Kind of work

Definition

Examples

Is it toil?

Toil

Operational work with the six traits above

Manually restarting a service, resizing disks by hand, handling the same alert every week

Yes

Overhead

Administrative work not tied to running a service

Team meetings, goal setting, performance reviews, HR paperwork

No

Engineering

Work that permanently improves the service or removes future work

Writing automation, fixing a root cause, improving monitoring, redesigning for scale

No, and it's what removes toil

Two more distinctions matter. Grungy work isn't automatically toil: a painful one-off cleanup of legacy code has enduring value, so it's engineering. And toil isn't always bad in small amounts. A little predictable, repetitive work can be calming, and new team members often learn a system through it. The problem is when it grows unchecked and crowds out everything else.

Examples of toil

Toil hides in every part of operations. These are the patterns SRE teams report most often, grouped by where they show up, with the engineering work that usually removes them.

Area

Toil example

Engineering fix

Alerts and on-call

Acknowledging alerts that resolve on their own; triaging 40 duplicate alerts for one failure; paging the wrong team and re-routing by hand

Alert deduplication and correlation; tuning or deleting non-actionable alerts; service-based routing

Incident response

Creating the incident channel, inviting responders and posting the first status update by hand every time

Automated incident creation with roles, channel and status templates

Deployments

Running the same release checklist step by step; manually promoting builds between environments

CI/CD pipelines with automated checks and rollback

Capacity

Resizing disks, adding nodes or bumping quotas when a threshold is hit

Autoscaling; capacity forecasts; quota policies

Recurring fixes

Restarting a service that leaks memory; clearing a queue that backs up every week

Fixing the root cause through a postmortem action item

Access and requests

Granting database access, creating accounts, resetting environments on request

Self-service tools with approval workflows

Certificates and secrets

Renewing TLS certificates and rotating credentials by calendar reminder

Automated renewal and rotation

Data and reports

Pulling the same weekly metrics by hand for a reliability review

Dashboards and scheduled reports

A useful pattern: if a fix appears in two postmortems in a row, or a runbook step is followed exactly the same way every time, it's a candidate for automation. A runbook is often the first draft of the script that will replace it.

Google's 50% toil cap

Minimum toil from on-call by rotation size compared with Google’s 50% toil cap

Google's SRE organization sets an explicit goal: keep toil below 50% of each SRE's time. The other half, at least, goes to engineering work that makes the service more reliable or removes future toil.

The cap exists because toil grows on its own. Every new service, customer and feature adds a little operational work. Without a limit, an SRE team slowly turns into an operations team with no time left to engineer its way out. Google also treats the cap as a hiring promise: SRE is sold to new engineers as an engineering role, not an ops role.

On-call sets a floor. Google's SRE book points out that on-call alone guarantees a minimum level of toil. An engineer who takes one primary and one secondary week per rotation cycle spends at least 2 of every 6 weeks on-call in a six-person rotation, a floor of about 33%. In an eight-person rotation the floor drops to 25%. Small rotations therefore leave very little room under the cap, which is one reason to keep on-call rotations at a healthy size.

What the data shows. In Google's quarterly surveys, SREs reported spending about 33% of their time on toil on average, under the 50% target. The average hides wide variation: some engineers reported almost none, others as much as 80%. Google's engineers named interrupts (non-urgent requests and messages) as their largest source of toil, followed by urgent on-call response, then releases and pushes.

Industry data outside Google points the same way, and the trend is rising. Catchpoint's SRE Report 2025 found that the median share of time SREs spend on operational work rose from 25% to 30%, the first increase in five years, despite wider use of AI tools. The SRE Report 2026, based on 418 responses, puts median toil at 34% of working time, with roughly half of respondents saying AI has reduced their toil load. In other words, the typical team now sits at about the level Google reports for its own SREs, and toil is growing faster than tooling is removing it.

What happens when a team goes over 50%? Two responses are common:

  • Push the work back. The SRE book's introduction describes redirecting excess operational work, such as tickets and pages, to the product development team that owns the service until the SRE team is back under the cap. This gives developers a direct incentive to build services that are easier to run.

  • Prioritize the fix. Many teams also move toil reduction projects ahead of new feature work, much like an exhausted error budget moves reliability work ahead of releases.

Google measures the cap over a few quarters, not week by week, because toil is spiky. A bad month above 50% isn't a failure. A team that stays above it for several quarters needs to step back and find out why.

The 50% figure isn't magic, and a team can choose a lower target. What matters is that the cap is written down, measured and enforced, so the conversation is about numbers rather than about who's most frustrated.

How to measure toil

Example toil inventory: five recurring tasks taking 30.2 hours a month

You can't cap what you don't measure. Most teams combine three methods, from lightest to most precise:

  1. Survey. Ask engineers each quarter roughly what share of their time went to toil, overhead and engineering. Google's SRE teams use this approach. It's quick, and the trend over several quarters matters more than any single answer.

  2. Tag the work. Label tickets, pages and tasks as toil when they're created or closed. Pages and incident timelines are especially useful because the timestamps are already recorded.

  3. Build a toil inventory. List recurring tasks with their frequency and time per occurrence. This is the most useful method, because it tells you not only how much toil there is but which tasks to remove first.

Example toil inventory

An illustrative month for a three-person platform team (about 480 working hours):

Task

Times per month

Minutes each

Hours per month

Acknowledge disk-space alerts that resolve on their own

60

5

5.0

Restart a worker with a memory leak

12

20

4.0

Grant database access on request

30

15

7.5

Renew TLS certificates by hand

4

45

3.0

Run the manual release checklist

16

40

10.7

Total

30.2

These five tasks alone take about 30 hours a month, roughly 6% of the team's time. Add on-call interruptions, ad hoc requests and recurring incident cleanup, and teams are often surprised by how quickly the total approaches the cap.

Measure two numbers per task, not one: time spent, and interruptions caused. A five-minute task that pages someone at 3 a.m. costs far more than its five minutes, because of the lost sleep and the context switch the next morning.

Toil reduction: how to reduce toil in SRE

Toil reduction is engineering work like any other: it needs a backlog, a priority order and time set aside for it. The fastest teams follow the same sequence.

1. Decide what to remove first

Rank the toil inventory by how much time automation would save compared with what it costs to build:

Payback period = engineering hours to automate ÷ toil hours saved per month

From the example inventory: the manual release checklist takes 10.7 hours a month. If automating it takes 40 engineering hours, it pays for itself in under four months, and every month after that is pure gain. The TLS renewal saves 3 hours a month, but if automation takes 8 hours it pays back even faster. Start with short payback periods, then weight tasks that interrupt people out of hours more heavily.

2. Pick the right fix

Not all toil should be automated. In order of preference:

Strategy

When to use it

Example

Eliminate

The task shouldn't exist at all

Delete an alert nobody acts on instead of automating its acknowledgment

Fix the root cause

The task repairs the same failure repeatedly

Fix the memory leak instead of scripting the restart

Automate

The task is needed and fully predictable

Automated certificate renewal; autoscaling

Self-service

Another team needs something you can safely let them do

Access requests through an approval workflow

Reduce frequency

The task can't be removed yet

Batch weekly requests instead of handling each one immediately

Automating a task that shouldn't exist is a common trap: you've built and now maintain code for work that had no value in the first place.

3. Protect the time

Toil reduction loses to feature work unless the time is reserved. Common approaches are a fixed share of each sprint for reliability work, a rotating "toil owner" for each cycle, or making toil projects automatic follow-ups from postmortem action items.

4. Measure again

Repeat the survey or inventory each quarter. Toil reduction is working when the total goes down even as the service grows, the reverse of toil's natural linear growth.

Automation has a cost too

Every script, bot and pipeline must be maintained, and automation that fails silently creates worse toil than the manual task did. Keep automations simple, monitor them like any other service, and document what they do in the runbook they replaced. For incident workflows specifically, see automated incident management.

How to reduce on-call toil

On-call is where toil hurts most, because it arrives as interruptions, often at night. A page that needs no action is pure toil: manual, repetitive, tactical and with no enduring value. It also feeds directly into on-call burnout.

The biggest source of on-call toil is noise. In ITOC360's analysis of one million production alerts, 60–80% needed no human action. Six changes remove most of it:

  1. Delete or downgrade non-actionable alerts. Review every alert that fired last month. If nobody did anything in response, delete it, raise its threshold, or turn it into a ticket instead of a page. See how to reduce alert fatigue.

  2. Deduplicate and correlate. One failure should produce one page, not forty. Alert deduplication merges repeats; alert correlation groups related alerts from different tools into a single incident.

  3. Alert on symptoms, not causes. Page on user impact, such as error budget burn rate, rather than on CPU or memory thresholds. Burn-rate alerts page only when users are affected; see error budgets.

  4. Route to the owner the first time. Every page that goes to the wrong team creates two units of toil: the wrong person's interruption and the manual re-route.

  5. Attach the runbook. A page that links to its runbook saves the responder the search, and the runbook steps followed every time show you exactly what to automate next.

  6. Automate the response scaffolding. Opening the incident channel, assigning roles, posting the first status update and recording the timeline should happen without anyone typing.

A useful on-call toil metric is pages per shift that required no action. Track it per service and per week. When it rises, the alerting rules for that service are the first place to look.

Where ITOC360 fits

ITOC360 removes the on-call and incident toil described above, so SRE time goes to engineering instead:

  • Less alert noise. ITOC360 Alerting deduplicates and correlates alerts from 50+ monitoring sources before anyone is paged, so one failure becomes one incident.

  • The right person the first time. ITOC360 On-Call routes each incident by service ownership and live schedules, and escalates automatically when nobody acknowledges.

  • No manual setup. ITOC360 Incident Response creates the incident, assigns roles and records the timeline. With ChatOps, responders acknowledge and act from Slack or Microsoft Teams.

  • Toil you can measure. Every page, acknowledgment and escalation is timestamped, so "pages per shift that required no action" becomes a report rather than a guess.

Frequently asked questions

What is toil in SRE?

In site reliability engineering, toil is operational work that is manual, repetitive, automatable, tactical, has no enduring value, and grows linearly as the service grows. Examples include restarting services by hand, acknowledging alerts that resolve themselves, and handling routine access requests.

What is Google's SRE definition of toil?

Google's SRE book defines toil as operational work, tied to running a production service, that is manual, repetitive, automatable and tactical, adds no lasting value, and grows in step with the service. Not every task needs all six traits; the more it has, the more clearly it's toil.

What is Google's 50% toil cap?

Google's SRE teams aim to keep toil below 50% of each engineer's time, leaving at least half for engineering work. When a team exceeds the cap, excess operational work is pushed back to the development team and toil reduction moves up the priority list.

What is toil reduction?

Toil reduction is the practice of measuring repetitive operational work and removing it, by eliminating unnecessary tasks, fixing root causes, automating what remains, or offering self-service. The goal is for toil to stay flat or shrink even as the service grows.

What is the difference between toil and overhead?

Toil is operational work tied to running a service, such as manual restarts or routine pages. Overhead is administrative work not tied to a service, such as meetings, planning and performance reviews. Both take time away from engineering, but only toil can be engineered away.

How do you measure toil?

Three common methods: a quarterly survey of how engineers spend their time, tagging tickets and pages as toil, and a toil inventory that lists each recurring task with its frequency and duration. The inventory is the most useful, because it shows which tasks to remove first.

How do you reduce on-call toil for SRE teams?

Delete or downgrade alerts that never need action, deduplicate and correlate the rest, alert on user-facing symptoms such as error budget burn rate, route pages to the service owner, and automate incident setup. Track pages per shift that required no action.

Is all toil bad?

No. Small amounts of predictable, repetitive work can be useful, especially for new team members learning a system. Toil becomes a problem when it grows unchecked and crowds out the engineering work that would remove it.

Sources