How to use this document for your TDD
System context
ORCA sits between the monitoring estate you already run and the ITSM process you already follow. It does not replace either. Raw signals enter from monitoring and observability tools; a single, enriched, auditable incident leaves toward ITSM. Boundary rules that matter for your TDD:- ORCA is deployed in-tenant: all event data, state, and the incident record remain inside your perimeter. No telemetry leaves the environment.
- The CMDB remains the system of record for configuration; ServiceNow remains the system of record for incidents. ORCA is the system of record for correlation evidence — why events were grouped, with the graph paths that justify it.
- All source integrations are read/subscribe; the only writes to external systems are incident creation/update in ITSM and (optionally) drift flags back to the CMDB reconciliation queue.
Component architecture
Every processing stage is an independently scalable Flink job communicating through Kafka topics — there are no direct service-to-service calls in the hot path. This is what makes each stage’s throughput independently observable (see healthy pipeline shape) and each failure independently recoverable.The seven-stage pipeline in detail
Raw signals flow left-to-right; each stage reduces noise and adds context until a single, enriched incident is recorded for audit. For each stage, the tables below give the level of detail a TDD needs: contract, state, scaling unit, and failure semantics.Stage 0 — Sources / Ingestion
Stage 1 — Normalize
Stage 2 — Dedup
Stage 3 — Enrich
Stage 4 — Suppress
Stage 5 — Correlate
Stage 6 — Incident record
Event lifecycle (sequence)
Healthy pipeline shape
Volume must decrease monotonically left-to-right: dedup removes duplicates, suppression removes maintenance noise, correlation collapses groups.Streaming backbone
- Kafka (KRaft mode) — the only transport in the hot path. Topics per stage boundary (
events.raw.*,events.normalized,events.deduped,events.enriched,events.actionable,incidents); partitioned so that per-CI event ordering is preserved end-to-end. Greenfield deployments start on Kafka 4.0+, whose consumer-rebalance protocol materially shortens broker-failure recovery. - Flink — one job per stage; RocksDB state backend; periodic checkpoints to MinIO. Checkpointing is the recovery mechanism proven in chaos experiment 4.
- Schema Registry — every topic schema versioned with enforced compatibility mode; producers cannot publish breaking changes. Local cache provides 30 minutes of registry-outage tolerance.
- Debezium CDC — streams CMDB/configuration changes into the graph-sync path so topology used for correlation tracks reality, not last quarter’s import.
- Dead-letter topics — every stage dead-letters poison events with a failure class instead of dropping them;
orca-dlq-replayprovides inspection and selective, idempotent replay (replayed events pass back through dedup, so replay cannot create duplicate incidents). - MinIO — original payload archive (forensics), Flink checkpoints, and evidence-bundle storage.
Control tower — self-monitoring
An observability platform that cannot observe itself is a liability at 2 a.m. The control tower monitors every ORCA component so a degraded component is detected before it corrupts incident accuracy. The four Grafana panels are required at go-live and ship as JSON in the release package. External integrations sit behind circuit breakers surfaced on panel 3. SLO alerting uses multi-burn-rate windows (5 m / 1 h / 6 h / 30 d — the Google SRE standard).Human-in-the-loop remediation
Approval thresholds are configurable per tier and service: auto-approval is earned per runbook through the feedback loop, never default. Every execution is recorded with approver identity in the audit trail. ORCA recommends; you decide.Deployment topology
- Reference layout: three zones (or three failure domains in a single DC), replication factor 3 for Kafka, Neo4j causal cluster, ClickHouse replicated tables, Redis Cluster with replicas crossing zones. Anti-affinity rules ship in the Helm charts.
- Profiles: the same charts deploy the sovereign profile (in-region, data-residency pinned) and the air-gapped profile (mirrored registry, offline updates) — see Air-Gapped Install.
- Node pools: stateful stores on storage-optimized nodes; Flink on compute-optimized; services on general purpose. Exact counts per tier are in the Sizing Guide.
High availability and failure semantics
Every figure in this table is a measured result from the mandatory chaos suite run against production-sized state in your environment — never a vendor claim. The chaos run is go-live Gate 6; its outputs regenerate the on-call runbook.
Data flow, retention, and lineage
End-to-end lineage: any incident can be traced to every raw event that contributed to it, and any raw event to the incident (or suppression rule, or DLQ entry) it ended in. Nothing exits the pipeline without a recorded disposition — the property that makes the record audit-ready.