Runbooks — Energy & Utilities (Smart Grid & Remote Ops)

1. Onboarding Runbook (New Substation / Remote Facility Integration)

Objective: Safely bring a new substation, generation plant, or remote facility into the secure utility network.

Step Sequence:

  1. Pre-Validation
    • Confirm DIA/MPLS circuits or licensed microwave availability.
    • Validate satellite fallback link.
    • Inventory RTUs (Remote Terminal Units), IEDs (Intelligent Electronic Devices), and SCADA controllers.
  2. Edge Deployment
    • Install SD-WAN appliance with redundant power.
    • Configure VRFs: SCADA control, corporate IT, vendor access.
    • Deploy IoT/edge gateway for sensor telemetry.
  3. Zero-Touch Provisioning (ZTP)
    • Auto-register devices to central orchestrator.
    • Apply latency-sensitive templates for SCADA.
  4. Security & Compliance Enrollment
    • PKI certificates provisioned for IEDs/RTUs.
    • ZTNA roles created for operators, field engineers, vendors.
    • Logs directed to SIEM with NERC-CIP tagging.
  5. Validation Tests
    • Ping SCADA commands (<50 ms).
    • Simulate breaker open/close.
    • Validate historian DB updates.
  6. Handover
    • Facility marked “Production” in CMDB.
    • NOC/SOC thresholds activated.

2. Failover Runbook (Primary Network Loss)

Objective: Maintain telemetry and control when primary link fails.

Step Sequence:

  1. Detection
    • AIOps alarm on SCADA jitter/packet loss.
    • RTU logs delayed control responses.
  2. Automatic Failover
    • SD-WAN reroutes SCADA traffic to licensed microwave or LTE/satellite backup.
    • Telemetry prioritized; non-critical IT traffic throttled.
  3. Validation
    • Synthetic breaker command tested via failover path.
    • Historian DB update verified.
  4. Notification
    • NOC incident raised; utility ops notified.
    • Carrier/vendor escalation opened.
  5. Recovery
    • Primary restored; traffic reverts gracefully.
    • Failover performance logged for audit.

3. Incident Response Runbook (OT Cyber Compromise)

Objective: Contain and remediate malicious activity in OT network.

Step Sequence:

  1. Alert
    • SIEM detects anomalous SCADA commands.
    • NDR flags lateral movement between substations.
  2. Containment
    • Isolate VRF for suspect substation.
    • Block vendor/contractor access via ZTNA.
    • Activate one-way data diode (read-only telemetry).
  3. Eradication
    • Reimage compromised RTUs/IEDs.
    • Patch vulnerabilities.
    • Rotate credentials/keys.
  4. Recovery
    • Reintroduce clean devices.
    • Run SCADA integrity test.
  5. Postmortem
    • Incident documented in CIP compliance system.
    • Controls tightened (e.g., stricter vendor access).

4. Disaster Recovery Drill Runbook (Regional Outage)

Objective: Rehearse recovery of grid services during large-scale outage (storm, fire, cyber).

Step Sequence:

  1. Scenario Trigger
    • Simulate regional fiber cut or substation destruction.
  2. Failover Activation
    • Telemetry rerouted via satellite backup.
    • Critical SCADA workloads moved to alternate control center.
    • DR site historian DB activated.
  3. Critical App Validation
    • Breaker commands issued from DR site.
    • Metering/AMI data delivered.
    • Load-frequency controls maintained.
  4. Time-to-Recover Measurement
    • RTO/RPO logged.
    • Results compared to NERC CIP standards.
  5. Debrief
    • Lessons shared with operations & regulators.
    • Adjust DR plans (e.g., add satellite capacity, microgrid failover).

Roles & Responsibilities

  • NOC: Monitor circuits, failover routing, ticketing.
  • SOC: Cyber detection, incident containment.
  • Utility OT Engineers: SCADA/RTU control, field validation.
  • Compliance Officers: NERC-CIP, regulatory reporting.
  • Vendors/Carriers: Circuit and device support.

KPIs (Energy & Utility Runbook Metrics)

  • Onboarding: Site live <20 days, CIP compliance checklist passed.
  • Failover: Telemetry continuity ≥99.9%, breaker latency <50 ms.
  • Compromise MTTR: <2 hours.
  • DR Drill: RTO ≤4 hours, RPO ≤15 minutes for historian data.
  • Compliance: NERC CIP audit pass = 100%.

⚖️ Logos Framing

  • Onboarding = inscribing a new substation as a “letter” in the energy lexicon.
  • Failover = synonym substitution to keep SCADA sentences intelligible.
  • Incident Response = correcting corrupted meaning in OT grammar.
  • DR Drills = recursive rehearsal so the energy language stays coherent during chaos.