Runbooks — Research HPC & Data Lake (High-Throughput Science)

1. Onboarding Runbook (New Lab / Institute Integration)

Objective: Connect a new research institution, lab, or HPC cluster into the global high-throughput science fabric.

Step Sequence:

  1. Pre-Validation
    • Verify DCI waves or MPLS pseudo-wires to core HPC center.
    • Confirm IX peering with research networks (Internet2, GEANT, APAN).
    • Assess storage: local scratch, long-term data lake, archival tiers.
  2. Edge Deployment
    • Deploy SD-WAN edge for segmentation of flows.
    • Provision high-capacity ports (10/40/100/400G).
    • Configure VLANs/VRFs: instrument/lab data, compute cluster, admin IT.
  3. Zero-Touch Provisioning (ZTP)
    • HPC cluster nodes auto-register with scheduler (Slurm, PBS, Kubernetes for workflows).
    • Apply network templates (QoS for bulk transfers, priority for control-plane traffic).
  4. Security & Compliance Enrollment
    • Certificates provisioned for researchers and cluster nodes.
    • ZTNA roles: PI, researcher, sysadmin, collaborator.
    • SIEM logging activated; GDPR tags enabled if personal data present.
  5. Functional Tests
    • Benchmark dataset transfer (>10 Gbps sustained).
    • Submit test job to scheduler.
    • Validate data replication between primary and data lake.
  6. Handover
    • Lab added to CMDB.
    • NOC thresholds set (throughput, storage capacity, job success rates).

2. Failover Runbook (Link / Wave Failure)

Objective: Maintain research data continuity when primary high-throughput link fails.

Step Sequence:

  1. Detection
    • AIOps flags drop in throughput below 80% of baseline.
    • Job latency metrics spike.
  2. Automatic Failover
    • SD-WAN reroutes transfers via secondary wave/MPLS.
    • If unavailable, DIA + IX peering used with rate-limited backup transfers.
  3. Validation
    • Synthetic dataset transfer resumed.
    • Benchmark throughput measured.
  4. Notification
    • NOC escalates to carrier for wave/optical issue.
    • Researchers notified of rerouting.
  5. Recovery
    • Primary path restored.
    • Queued data flushed to lake.

3. Incident Response Runbook (Data Integrity or Breach Attempt)

Objective: Contain and remediate threats to research data (IP theft, integrity compromise, unauthorized access).

Step Sequence:

  1. Alert
    • SIEM flags anomalous queries on dataset.
    • WAF/API detects unusual access to portal.
  2. Containment
    • Isolate suspect user/device session with ZTNA.
    • Block flows at FWaaS.
    • Quarantine dataset copy for integrity check.
  3. Eradication
    • Endpoint/device patched or reimaged.
    • Rotate API keys and credentials.
    • Apply additional DLP rules.
  4. Recovery
    • Verify dataset checksums/hashes.
    • Restore clean copy from immutable backup.
    • Resume cluster access.
  5. Postmortem
    • Incident logged with compliance board (GDPR, IRB, sponsor).
    • SIEM/DLP rules tuned.

4. Disaster Recovery Drill Runbook (HPC/Data Lake Outage)

Objective: Rehearse loss of an HPC site or data lake region.

Step Sequence:

  1. Scenario Trigger
    • Simulate primary HPC center outage (fire, power failure).
  2. Failover Activation
    • Redirect jobs to alternate HPC cluster (geo-distributed).
    • Replicate datasets to secondary data lake region.
    • Critical research workloads rerouted to cloud HPC burst.
  3. Critical Validation
    • Run benchmark job in DR HPC cluster.
    • Validate access to replicated datasets.
    • Confirm researchers regain access via VPN/ZTNA.
  4. Time-to-Recover Measurement
    • RTO measured for job submission resumption.
    • RPO validated for dataset replication (<15 min data loss).
  5. Debrief
    • Researchers, IT, compliance officers review.
    • Adjust DR playbooks (e.g., add burst contracts with cloud).

Roles & Responsibilities

  • NOC: Monitor optical/DCI throughput, failover.
  • SOC: Detect IP theft attempts, enforce data integrity.
  • Sysadmins: HPC cluster management, scheduler ops.
  • Researchers/PI: Validate access to jobs/datasets.
  • Compliance Officers: GDPR/IRB reporting, sponsor notifications.

KPIs (Research HPC Runbook Metrics)

  • Onboarding: Lab live <30 days with cluster/scheduler integration.
  • Failover: Dataset transfer continuity ≥99.9%; throughput ≥80% of baseline.
  • Breach MTTR: <2 hours.
  • DR Drill: HPC workload recovery ≤4 hours; RPO ≤15 min dataset loss.
  • Compliance: GDPR/IRB audits = 100% pass.

⚖️ Logos Framing

  • Onboarding = inscribing a lab into the grammar of science.
  • Failover = synonym substitution (alternate wave, cloud burst) so the research sentence still reads.
  • Incident Response = correcting corrupted “words” (datasets, results) before meaning is lost.
  • DR Drills = recursive rehearsal — proving that scientific knowledge remains coherent across outages.