Orchestrated Saga Pattern
Problem
Some provisioning workflows in CIVITAS/CORE require sequential, cross-adapter operations where each step depends on outputs from previous steps. If a later step fails, all previously completed steps must be rolled back to maintain consistency.
Example: Dataset / Dataspace Provisioning
| Step | System | Action | Output | Depends On |
|---|---|---|---|---|
| 1 | FROST Server | Create project | projectId, internal URL | -- |
| 2 | APISIX Gateway | Create read-only route | External URL | Step 1: internal URL |
| 3 | Redpanda Connect | Deploy pipelines | Pipeline ID | Step 1 + Step 2 |
If step 3 fails after steps 1 and 2 succeeded, the APISIX route and FROST project must be removed.
The existing Config Adapter framework handles independent, stateless operations well, but cannot manage these cross-adapter dependencies.
Why Orchestration over Choreography
A choreography-based approach was evaluated and rejected (ADR 031). With three or more sequential steps, choreography introduces:
- Implicit workflow coupling -- each adapter must know its predecessor and successor
- Cascading compensation -- rollback propagates backwards; if any compensation fails, subsequent compensations never trigger
- Distributed routing logic -- adapters accumulate responsibilities beyond their domain
- Bloated event payloads -- the initial event must carry configuration for all adapters
Solution: Dedicated Orchestrator
A dedicated config-adapter-orchestrator module centrally manages multi-step provisioning workflows. Adapters remain simple command handlers with no workflow knowledge.
Architecture Overview
Responsibility Split
| Component | Responsibilities |
|---|---|
| Orchestrator | Workflow execution, saga state management, data mapping between steps, compensation on failure, status reporting |
| Adapters | Pure command handlers -- receive a command, execute it, return a result. No workflow knowledge, no awareness of sagas or compensations |
An adapter does not know whether a delete command is a regular operation or a compensation rollback. It executes the same logic in both cases.
Success Flow
Failure Flow with Compensation
When a step fails, the orchestrator triggers parallel compensation for all previously successful steps:
Saga State Machine
| State | Description |
|---|---|
PENDING | Saga created, not yet started |
EXECUTING | Steps are being executed sequentially |
COMPLETED | All steps finished successfully |
COMPENSATING | A step failed; compensating previous steps |
COMPENSATED | All compensations completed successfully |
COMPENSATION_FAILED | A compensation failed; requires manual intervention |
Workflow Definition
Workflows are defined declaratively. Adding a new backend (e.g., Stellio/NGSI-LD) means adding a new workflow definition -- no existing adapter code changes.
workflows:
dataspace-create-frost:
trigger: "dataspace.create.requested"
condition: "payload.backendType == 'FROST'"
steps:
- name: create-frost-project
command: "frost.project.create"
input:
projectName: "{{payload.datasetName}}"
output:
projectId: "{{result.resultData.projectId}}"
baseUrl: "{{result.resultData.baseUrl}}"
compensate:
command: "frost.project.delete"
input:
projectId: "{{steps.create-frost-project.output.projectId}}"
- name: create-apisix-route
command: "apisix.route.create"
input:
uri: "/api/dataspace/{{payload.datasetId}}/*"
upstreamUrl: "{{steps.create-frost-project.output.baseUrl}}"
methods: ["GET"]
output:
routeId: "{{result.resultData.routeId}}"
compensate:
command: "apisix.route.delete"
input:
routeId: "{{steps.create-apisix-route.output.routeId}}"
- name: deploy-pipelines
command: "redpanda.pipeline.deploy"
input:
pipelineJson: "{{payload.pipelineJson}}"
targetUrl: "{{steps.create-frost-project.output.baseUrl}}"
compensate:
command: "redpanda.pipeline.delete"
input:
pipelineId: "{{steps.deploy-pipelines.output.pipelineId}}"
Saga State Persistence
The orchestrator requires persistent storage for saga state. PostgreSQL is recommended, as the platform stack already includes it.
| Option | Pros | Cons |
|---|---|---|
| PostgreSQL (shared) | Proven, transactional, queryable | Shared DB dependency |
| PostgreSQL (dedicated) | Full isolation | Additional infrastructure |
| Kafka Streams state store | No external DB | Harder to query, operational complexity |
API Extension
The ConfigResultEvent includes an additional field to carry structured result data from adapters:
// Backwards-compatible, nullable
private Map<String, String> resultData;
Existing adapters that do not set resultData continue to work via the existing resourceId field.
Trade-offs
| Benefit | Trade-off |
|---|---|
| Adapters stay simple (no workflow logic) | Additional deployment unit (orchestrator) |
| Central workflow visibility and queryable state | Requires a database for saga state |
| Parallel compensation (not cascading) | Orchestrator must be highly available |
| Scalable to new backends via workflow definitions | Orchestrator framework must be built |
| Existing adapters remain unchanged | Single point of coordination |