Network Outage Triage and Change Control Playbook

First five minutes

1. Record exact start time and reported scope.
2. Identify what still works.
3. Check recent changes.
4. Check power and major uplinks.
5. Check monitoring/syslog for the same timestamp.

Define blast radius

One user, one VLAN, one switch, one site or every site? Scope is one of the fastest ways to choose the correct layer to investigate.

Freeze unnecessary change

During a major outage, stop unrelated changes. Multiple simultaneous changes destroy the evidence chain.

Reversible troubleshooting

Prefer changes that can be rolled back quickly. Save configuration before editing. Record every command or UI change with time and reason.

Rollback criteria

Before a change, decide what result means success and what result means roll back. "Try this and see" is not a change plan.

After recovery

• Confirm service from the user's perspective.
• Collect final logs/counters.
• Write the cause and contributing factors.
• Record permanent corrective actions.
• Update documentation and monitoring.

Client benefit

A disciplined triage process reduces downtime and prevents the troubleshooting process itself from causing further service loss.

Evidence package

Capture topology state, route tables, interface state, alarms and recent configuration changes before disruptive recovery steps. Screenshots are useful, but text exports are easier to compare later.

Communication

Separate technical work from stakeholder updates. Give users a scope, current impact and next update time without guessing at a root cause before evidence exists.

Post-incident review

Distinguish root cause from contributing factors. A failed uplink may be the trigger, while missing LACP redundancy, absent monitoring and stale documentation made the outage longer.

Layered triage

Once the scope is known, work from dependencies upward: power, physical links, switching, routing, DNS, authentication and application. Jumping directly to application settings while the distribution switch is down wastes time.

Known recent changes

Check firewall changes, switch replacements, firmware upgrades, ISP work, cabling moves and power maintenance near the incident start time. Correlation is not proof, but it is high-value evidence.

Emergency change discipline

An emergency does not remove the need to record changes. Write down what was changed and when, even if the formal ticket is completed later. This lets the team reverse unsuccessful actions.

Service restoration vs root cause

Restoring service can be the first priority. Root-cause investigation may continue afterward using preserved logs and failed components. Do not destroy evidence during a rushed cleanup.

Lessons into engineering

If one fibre cut took down the site because both uplinks shared the same duct, the permanent action is route diversity, not simply replacing the fibre faster next time.