Risk Management System

Systematic identification of hazards arising from the intended use and reasonably foreseeable misuse of DEFONEOS. Sources include:

• Threat Intelligence: SIGIL chain analysis, OSINT feeds, CVE databases, adversarial ML databases (Adversarial Robustness Toolbox, TextAttack)
• Red-Team Output: 8-vector adversarial testing framework (see defoneos-adversarial-robustness.html) — prompt injection, jailbreak, data poisoning, model evasion, supply chain, model extraction, DoS, side channel
• Incident History: 7-phase incident response pipeline historical data (see defoneos-incident-response.html)
• Post-Market Monitoring: 14 monitoring vectors from deployed sensors (see defoneos-post-market-monitoring.html)
• Foreseeable Misuse: Red-line violation attempts, unauthorised data access, dual-use exploitation, model repurposing
• Known AI Risks: Hallucination, bias, drift, opacity, automation bias, adversarial vulnerability

Output: 42 identified hazards catalogued in the DEFONEOS risk register, each tagged with source, category, and initial severity estimate.

Each identified risk is scored using a Severity × Likelihood matrix aligned with ISO 14971 and NIST AI RMF:

Risk scoring dimensions:

• Harm to Health & Safety: Physical harm from autonomous systems, sensor misinterpretation, delayed alerts
• Harm to Fundamental Rights: Privacy violation, discrimination, due process denial, human dignity
• Harm to Property: Infrastructure damage, data loss, financial harm, reputation
• Harm to National Security: Sovereign infrastructure compromise, information warfare, critical infrastructure

Output: Risk register with 42 entries scored 1-5 on severity and 1-5 on likelihood (scale 1-25). Current distribution: 0 Critical, 0 High, 3 Medium, 39 Low residual risk.

DEFONEOS follows the hierarchy of controls (in order of preference):

1. Elimination by design: Remove the capability that creates the risk (e.g., no kinetic targeting, no personal surveillance — 7 red lines enforced)
2. Substitution: Replace high-risk component with lower-risk alternative (e.g., on-device processing instead of cloud, local inference instead of remote API)
3. Engineering controls: Technical safeguards — rate limiting, input validation, output filtering, sandboxing, air gaps, BFT consensus gates
4. Administrative controls: Policies, procedures, human oversight requirements, training, competency standards
5. Personal protective measures: Human-in-the-loop confirmation, transparency notices, appeal mechanisms, redress pathways

Article 9(2)(d) — Residual Risk: Only risks reduced to acceptable levels may be permitted. "Acceptable" means the residual risk does not exceed the benefit. DEFONEOS's 3 residual Medium risks are:

• MR-1: Hallucination in generated intelligence summaries (mitigated by source citations + human review + confidence scoring)
• MR-2: Supply chain compromise of third-party MCP packages (mitigated by sandboxing + integrity verification + provenance tracking)
• MR-3: Model drift over time without retraining (mitigated by automated drift detection PSI/KL + retraining triggers + performance monitoring)

Residual risks are tested against real-world attack scenarios and adversarial conditions:

• Adversarial Testing: 340 test cases per cycle across 8 attack vectors (see defoneos-adversarial-robustness.html)
• Red-Team Exercises: Simulated attacks on SIGIL chain, MCP supply chain, model extraction, prompt injection
• Fault Injection: Deliberate degradation of sensor inputs, network partitions, storage corruption
• Drift Testing: PSI/KL divergence monitoring, calibration checks, Page-Hinkley change detection
• Stress Testing: 100× normal load, concurrent attack scenarios, degraded mode operation

Pass criterion: Residual risk must remain at or below "Medium" under adversarial conditions. Current pass rate: 97.3% (9 failures per cycle, all caught and mitigated within 5 minutes).

Where risks cannot be eliminated by design, mitigation measures are deployed and the following are provided:

• To Deployers (Art 26): Instructions for use, known limitations, performance metrics, monitoring guidance (see defoneos-transparency-deployers.html)
• To Affected Persons (Art 86): Transparency obligations, right to explanation, complaint mechanisms, human review
• To Authorities (Art 73): Serious incident reporting within 15 days, corrective action documentation, market surveillance cooperation
• To the Public (Art 52): AI system transparency, synthetic content marking (Article 50 watermarks via Article 50 passport system)

The RMS operates continuously throughout the AI system's lifecycle. Post-market monitoring (Art 72) feeds new data back into the RMS:

• Continuous Monitoring: 14 monitoring vectors, 280 checks per 5-min cycle, 72h regulatory reporting pipeline
• Feedback Loop: Incident reports → risk register update → new mitigation controls → adversarial test expansion
• Version Tracking: Every RMS change is SIGIL-signed and hash-chained. Full audit trail of risk decisions.
• Corrective Actions: Triggered automatically when risk exceeds thresholds — can include model rollback, capability suspension, or system shutdown

Article 9(5) Experimental Testing: DEFONEOS supports real-world testing under controlled conditions with informed consent, data minimisation, and strict oversight. All experimental deployments require BFT council approval (23/33 quorum).

The RMS is documented as part of the technical documentation (Annex IV) and maintained for 10 years post-market placement (Art 18(1)). DEFONEOS documentation includes:

• Risk Register: All 42 identified hazards with severity, likelihood, mitigation status, residual risk
• Mitigation Inventory: 156 controls mapped to specific risks, with implementation evidence
• Test Results: Historical adversarial test outcomes, pass/fail trends, corrective actions taken
• Change Log: Every RMS modification with rationale, approval, and impact analysis
• Audit Trail: Ed25519-signed SIGIL chain providing cryptographic evidence of all risk decisions