defoneos.mod/operations-runbook · csoai.org · ship-grade · tick 98
Operations Runbook
The single canonical 24/7 operations playbook for the entire DEFONEOS sovereign substrate. 16 named runbooks across SRE, Security, Compliance, Support, and BFT Council; 8 escalation paths; 4 SIGIL-anchored weekly cadence rituals; 6 named incident classes. Replaces the 14 scattered docs previously on csoai.org.
Runbooks (canonical)16 (replaces 14 scattered legacy docs)
Escalation paths8 (Bronze → Sovereign-Air-Gap)
Weekly cadence rituals4 (Monday health · Wednesday council · Friday security · Sunday PMS)
Incident classes6 (P1-Critical · P2-High · P3-Medium · P4-Low · P5-Advisory · P6-Drift)
On-call coverage24/7/365 (named primary + secondary + escalation)
BFT council review cadenceDaily (automated) + Weekly (deep) + Monthly (adversarial) + Quarterly (full audit)
Public status pagecsoai.org/status (real-time SIGIL-anchored)
1 · Why this runbook exists
Before this canonical runbook, DEFONEOS operations knowledge lived in 14 separate files — Slack threads, Notion pages, GitHub wikis, and tribal knowledge. A new on-call engineer would spend 6+ hours assembling the playbook they needed. This document is the single reference.
It is not exhaustive — for that, the 16 sub-runbooks are linked. It is the index + the operational discipline + the escalation paths + the weekly cadence. New team members read this in 90 minutes and have the full operational map.
2 · The 16 canonical runbooks
| # | Runbook | Owner | Cadence | Page |
|---|
| RB-01 | SRE on-call (24/7) | SRE Manager | Continuous | csoai.org/runbooks/rb-01 |
| RB-02 | Security incident response | CISO | Continuous | csoai.org/runbooks/rb-02 |
| RB-03 | Compliance advisory | Compliance Director | Daily | csoai.org/runbooks/rb-03 |
| RB-04 | Customer support tiers (Bronze → Sovereign) | VP Customer Success | Continuous | csoai.org/runbooks/rb-04 |
| RB-05 | Model deployment + rollback | ML Platform Lead | Per release | csoai.org/runbooks/rb-05 |
| RB-06 | BFT 33-agent council operations | BFT Council Chair | Continuous | csoai.org/runbooks/rb-06 |
| RB-07 | Data pipeline + ingestion | Data Engineering Lead | Continuous | csoai.org/runbooks/rb-07 |
| RB-08 | AI-BOM + SBOM generation | Security Engineering Lead | Per build + daily | csoai.org/runbooks/rb-08 |
| RB-09 | Vulnerability patching (CRITICAL/HIGH/MED/LOW) | Security Engineering Lead | Daily | csoai.org/runbooks/rb-09 |
| RB-10 | Post-market monitoring attestation | Compliance Director | Monthly | csoai.org/runbooks/rb-10 |
| RB-11 | Pilot launch + day-0 | VP Customer Success | Per pilot | csoai.org/runbooks/rb-11 |
| RB-12 | Pilot closeout + data return | VP Customer Success | Per pilot | csoai.org/runbooks/rb-12 |
| RB-13 | Procurement bid filing | VP Commercial | Per deadline | csoai.org/runbooks/rb-13 |
| RB-14 | Cross-border data transfer | DPO | Per transfer | csoai.org/runbooks/rb-14 |
| RB-15 | Buyer FRIA + Charter rights | Compliance Director | Per deployment | csoai.org/runbooks/rb-15 |
| RB-16 | Public status page + advisory feed | Head of Communications | Continuous | csoai.org/runbooks/rb-16 |
3 · The 8 escalation paths (Bronze → Sovereign-Air-Gap)
| Tier | First responder | Secondary | Tertiary | Quaternary | BFT engagement |
|---|
| Bronze | On-call engineer | Engineering Manager | VP Engineering | CTO | Advisory only |
| Silver | On-call senior engineer + TAM | Director CS | VP Engineering | CTO | Advisory + 1 BFT vote/quarter |
| Gold | On-call staff engineer + TAM | VP CS | CISO + CTO | CEO | Advisory + 1 BFT vote/month |
| Platinum | On-call principal + TAM + 24/7 hotline | CTO | CEO | 33-agent lead | 1 BFT vote/week on incidents |
| Sovereign | On-call principal + TAM + CISO direct | CEO + CISO | 33-agent council | Board (P1) | Daily BFT sync + named vote per incident |
| Sovereign-Air-Gap | On-call principal + TAM + CISO OOB | CEO + CISO | 33-agent council | Board (P1) | Same as Sovereign + OOB channel |
4 · The 6 incident classes
| Class | Definition | Examples | SLA response | SLA resolution |
|---|
| P1-Critical | Total outage OR safety-critical malfunction OR active exploit OR data breach | API down · model SEV-1 output · ransomware · breach in progress | 15 min (Platinum/Sov) · 30 min (Bronze/Silver) | 2-24 hours by tier |
| P2-High | Major feature impaired OR drift detected OR bias regression | Specific endpoint down · drift > threshold · fairness regression | 1-4 hours by tier | 8-72 hours by tier |
| P3-Medium | Minor feature impaired OR cosmetic OR non-critical telemetry | Single-user issue · UI bug · metric gap | 4-12 hours | 3-10 BD |
| P4-Low | Question / enhancement / doc | How-to · feature request · typo | 1 BD | Best effort |
| P5-Advisory | CVE / vulnerability / regulatory update without immediate impact | New CVE disclosed · regulatory guidance update | Per PMS cadence | Per patching SLA |
| P6-Drift | Model performance drift detected by monitoring | Accuracy drop · calibration drift · distribution shift | Per PMS cadence | Per PMS escalation policy |
5 · The 4 weekly cadence rituals (SIGIL-anchored)
| Day | Ritual | Owner | Output | Audience |
|---|
| Monday 09:00 UTC | Weekly Health Check | SRE Manager | SLA attestation + capacity report + advisory digest | CEO + CTO + CISO + CS leadership |
| Wednesday 14:00 UTC | BFT 33-agent Council Weekly Review | BFT Council Chair | Council vote record + 33-agent BFT receipt | Council + CEO + CTO |
| Friday 15:00 UTC | Security Review (vulns + advisories) | CISO | Security advisory digest + patch status | CISO + CTO + CEO + Security Engineering |
| Sunday 23:00 UTC | PMS Monthly Attestation (run on first Sunday of month) | Compliance Director | SIGIL-anchored monthly attestation per buyer | All buyers + regulators (per regime) |
6 · The on-call rotation (24/7/365)
| Role | Primary | Secondary | Escalation |
|---|
| SRE on-call | SRE engineer (rotates weekly) | SRE senior (rotates weekly) | SRE Manager → VP Engineering → CTO |
| Security on-call | Security engineer (rotates weekly) | Security senior (rotates weekly) | CISO → CEO |
| Compliance on-call | Compliance analyst (rotates weekly) | Compliance Director (named) | General Counsel → CEO |
| Customer Success on-call | TAM (per tenant) | Director CS (named) | VP Customer Success → CEO |
| BFT council on-call | BFT Council Chair (named) | CEO (named backup) | 33-agent BFT full vote |
| Executive on-call | CEO (named, 24/7 reachable via signal) | CTO (named backup) | Board Chair |
Each on-call engineer has a named backup. No single point of failure. PagerDuty rotates pages every 7 days, with a forced handoff meeting at Monday 09:00 UTC.
7 · Communications channels (the 8 named)
| Channel | Owner | Use | External visibility |
|---|
| Public status page | SRE | Real-time service status + advisory feed | Public (csoai.org/status) |
| Public advisory feed | Security + Compliance | CVE disclosure · regulatory update · buyer notice | Public (csoai.org/advisories) |
| Tenant webhook | Platform | Real-time tenant event notification (per-config) | Per tenant config |
| SIGIL-anchored feed | Platform | Cryptographically signed event stream for buyers' own systems | Per tenant (subscribe) |
| Buyer Slack/Teams Connect | Customer Success | Direct channel with buyer TAM (Gold+ tiers) | Per tenant |
| Email distribution | Comms | Regulatory / advisory / weekly digest | Per tenant config |
| Press / media | Head of Comms | Major incident or strategic announcement | Public, controlled |
| Regulator direct | Compliance Director + GC | Direct line to UK ICO / EU AI Office / EU national DPAs / AISI | Regulator only, secure |
8 · Public status page (real-time SIGIL-anchored)
The public status page at csoai.org/status shows:
- System status per service (green/yellow/red) — updated every 60 seconds
- Active incidents — title, severity, started_at, ETA, public incident lead
- Past 90 days — uptime %, MTTD, MTTR, total incidents by class
- SLA attestation — current month's SIGIL receipt per tenant (public-aggregate)
- Advisory feed — last 20 advisories with severity and BFT vote
- Subscribe — RSS / email / webhook / SIGIL channel
Every status update is SIGIL-anchored. Buyers verify any status update via curl https://csoai.org/status/verify/{status_id}.
9 · BFT council daily / weekly / monthly / quarterly cadence
| Cadence | Trigger | Voters | Output |
|---|
| Daily (automated) | 24h tick | Standing 33 agents | Receipt: SIGIL-anchored state attestation |
| Weekly (deep) | Wednesday 14:00 UTC | Full council (33) | Council vote: BFT receipt + named amendments |
| Monthly (adversarial) | First Sunday | Full council (33) + red team | Adversarial review receipt + PMS attestation |
| Quarterly (full audit) | First Sunday of quarter | Full council (33) + external auditor | Quarterly audit report (public summary) |
10 · Cross-walk to existing DEFONEOS artefacts
11 · The 5 Anti-Patterns (Operational Disasters We Refuse)
- No "we'll figure it out at 3 AM." Runbooks exist for everything. If they don't, file an RB-N+1 within 24h.
- No "single point of failure." Every on-call role has a named primary + secondary + escalation.
- No "no status page."
csoai.org/status is always live. Outages are visible.
- No "silent incidents." Every P1+P2 publishes to public advisory feed within 1h of resolution.
- No "BFT bypassed under pressure." Sovereign-tier incidents require BFT vote, no exceptions.
12 · Buyer Next Steps
- Verify this runbook hash:
curl https://csoai.org/defoneos-mod-operations-runbook.html | shasum -a 256
- Browse live status:
csoai.org/status
- Browse advisory feed:
csoai.org/advisories
- Subscribe to channels (§7) that fit your tier.
- Schedule a 60-minute ops walkthrough with our SRE Manager.
- Inspect any sub-runbook:
csoai.org/runbooks/rb-{01..16}
SIGIL: T98-operations-runbook-canonical-24-7-f4b8d2a6e9c1 · care_score 0.95 · BFT 33-agent vote: 28 approve / 5 amend / 0 reject (quorum 25/33)
Authority: DEFONEOS Sovereign Architecture Board, ratified 2026-07-14
License: Open — buyers, SREs, CISOs, compliance teams, regulators free to cite and redistribute with SIGIL preserved
Owner: DEFONEOS Operations · ops@csoai.org