Runbooks — Public Safety / NG911 (Mission-Critical Communications)

1. Onboarding Runbook (New PSAP / Emergency Call Center)

Objective: Integrate a new Public Safety Answering Point (PSAP) or regional call center into the NG911 ecosystem.

Step Sequence:

  1. Pre-Validation
    • Provision DIA/MPLS circuits with diverse physical paths.
    • Confirm ESInet/NGCS interconnection agreements.
    • Inventory CAD, radio gateways, and recording systems.
  2. Edge Deployment
    • Install SD-WAN appliances for DIA/MPLS + LTE/5G failover.
    • Configure VLANs/VRFs: CAD, call handling, admin IT.
    • Deploy redundant SBCs for SIP call routing.
  3. Zero-Touch Provisioning (ZTP)
    • Devices register with orchestrator.
    • Apply NG911 templates (QoS for voice, geo-pin for calls).
  4. Security & Compliance Enrollment
    • PKI/ZTNA roles: call-taker, dispatcher, supervisor.
    • Enable SIEM logging for all call routing.
    • Tag logs for CJIS compliance.
  5. Functional Tests
    • Place test 911 calls (wired, mobile, VoIP).
    • Validate CAD integration with AVL/GPS.
    • Confirm recordings geo-replicated.
  6. Handover
    • PSAP marked “Production” in CMDB.
    • NOC/SOC monitoring thresholds enabled (call success rate, CAD uptime).

2. Failover Runbook (Primary Call Path Loss)

Objective: Ensure call-taking and dispatch continuity during carrier or trunk outage.

Step Sequence:

  1. Detection
    • AIOps flags call setup failures >1%.
    • Synthetic 911 call probe fails.
  2. Automatic Failover
    • SD-WAN/SBC reroutes calls to alternate ESInet path.
    • LTE/5G backup ensures dispatch CAD connectivity.
    • Non-emergency admin traffic throttled.
  3. Validation
    • Place test 911 call via alternate carrier.
    • Confirm CAD sync continues via failover path.
  4. Notification
    • NOC escalates to telecom carrier.
    • Supervisor briefed.
  5. Recovery
    • Primary path restored.
    • Failback executed without call drop.
    • SLA incident logged.

3. Incident Response Runbook (Fraudulent or Malicious Call Flood / DDoS)

Objective: Contain denial-of-service or spoofed call attempts threatening PSAP operations.

Step Sequence:

  1. Alert
    • SIEM detects unusual call volume patterns.
    • SBC flags high SIP INVITE rates from untrusted sources.
  2. Containment
    • Activate SIP flood protection rules.
    • Block untrusted ranges via SBC/Firewall.
    • Shift overflow calls to mutual-aid PSAPs.
  3. Eradication
    • SOC validates clean call flow.
    • Update SBC rules/signatures.
  4. Recovery
    • Resume normal load distribution.
    • Validate PSAP throughput with synthetic test calls.
  5. Postmortem
    • Incident filed with regulators.
    • Playbook updated for DDoS response.

4. Disaster Recovery Drill Runbook (PSAP Facility Loss)

Objective: Rehearse full loss of an emergency call center due to disaster (fire, storm, flood).

Step Sequence:

  1. Scenario Trigger
    • Simulate PSAP facility outage.
  2. Failover Activation
    • Calls rerouted automatically to designated backup PSAP (per NENA i3).
    • Dispatch transferred to alternate site with CAD mirroring.
    • Field units routed to alternate PSAP radio channels.
  3. Critical Validation
    • Test 911 call answered at backup PSAP.
    • Confirm CAD incident created and dispatched.
    • Validate recordings remain compliant.
  4. Time-to-Recover Measurement
    • RTO measured from trigger to backup call answer.
    • RPO measured for CAD/AVL data (<5 min).
  5. Debrief
    • Supervisors, compliance, and IT review findings.
    • Adjust DR posture (e.g., add LTE failover kits, diversify carriers).

Roles & Responsibilities

  • NOC: Monitor circuits, SIP trunks, failover health.
  • SOC: Detect SIP floods, spoofed calls, and anomalous access.
  • Dispatch Supervisors: Validate call flows, CAD continuity.
  • Compliance Officers: Ensure CJIS/NENA reporting.
  • Carriers: Restore trunk circuits, SIP services.

KPIs (Public Safety Runbook Metrics)

  • Onboarding: PSAP live <30 days, compliance signed off.
  • Failover: Call continuity ≥99.999%, failover <30 sec.
  • Incident Response (SIP flood): Containment <15 min.
  • DR Drill: RTO ≤2 hours, RPO ≤5 min for CAD data.
  • Compliance: CJIS/NENA/FCC audit pass = 100%.

⚖️ Logos Framing

  • Onboarding = writing a new PSAP “letter” into the grammar of public safety.
  • Failover = synonym substitution (alternate trunks, PSAPs) to keep sentences of emergency calls coherent.
  • Incident Response = correcting corrupted “words” (spoofed/flooded calls).
  • DR Drills = recursive rehearsal ensuring the language of trust survives even under chaos.