ADR 031: Orchestrated Saga for Multi-Adapter Provisioning

Date: 2026-02-11

Status: Reviewed

Decision Makers: Architecture Board

Context

The Config-Adapter Framework currently handles independent, stateless operations: each adapter receives a command event, configures its target system, and publishes a result event. This works well for isolated tasks like creating a Keycloak realm or an APISIX route.

However, Datasset / Dataspace provisioning introduces sequential, cross-adapter dependencies:

Step	System	Action	Output	Depends On
1	FROST Server	Create project	projectId: "proj-123" → internal URL	None
2	APISIX Gateway	Create read-only route	External URL → internal FROST URL	projectId from step 1
3	Redpanda Connect	Deploy pipelines	Pipeline writing to FROST project	URL from step 1, route from step 2

If step 3 fails after steps 1 and 2 succeeded, all previous changes must be rolled back to maintain consistency.

Key Constraints

• Adapters must remain simple command handlers without workflow knowledge
• Rollback must guarantee eventual consistency across all involved systems
• The solution must scale to additional backends (NGSI-LD/Stellio, MinIO, ...) without modifying existing adapters
• Saga state must survive process restarts
• The existing resultTopic mechanism in ConfigEvent must be preserved

Example Scenario

Dataset "Zählerdaten Stadtwerke" (Dataspace-ID: ds-stadtwerke-zaehler)

Contains two pipelines:

1. DB-Pipeline: Reads meter master data from a database, transforms to OGC STA entities, writes to FROST Server
2. MQTT-Pipeline: Receives meter readings via MQTT, transforms to OGC STA Observations, writes to FROST Server

Provisioning requires FROST project creation → APISIX route creation → Redpanda pipeline deployment, in strict order.

Why Not Choreography?

An initial Choreography-Based Saga proposal was evaluated. With ≥3 sequential steps, choreography leads to:

1. Implicit workflow coupling: Each adapter must know its predecessor and successor in the chain
2. Cascading compensation: Rollback propagates backwards through the entire chain; if any compensation fails, subsequent compensations never trigger
3. Distributed routing logic: The APISIX adapter must distinguish FROST vs. Stellio predecessors and route compensations accordingly — violating Single Responsibility
4. Bloated event payloads: The initial event carries configuration for all adapters, violating Information Hiding

A detailed analysis (last comment) documents these problems with sequence diagrams and code examples.

Checked Architecture Principles

• [full] Model-centric data flow
• [full] Distributed architecture with unified user experience
• [full] Modular design — Orchestrator is a separate module; adapters remain independent
• [full] Integration capability through defined interfaces — Uses existing ConfigEvent/ConfigResultEvent contracts
• [full] Open source as the default — Built on Apache Kafka
• [full] Cloud-native architecture — Stateless adapters, persistent saga state
• [full] Prefer standard solutions over custom development — Saga Orchestration is a well-established pattern
• [partial] Self-contained deployment — Adds one additional deployment unit (orchestrator module)
• [full] Technological consistency to ensure maintainability — Same Java 21, Maven, Kafka stack
• [full] Multi-tenancy — Saga context carries dataspace/tenant identifiers
• [full] Security by design — No new attack surface; uses existing Kafka ACLs

Decision

Introduce a dedicated config-adapter-orchestrator module that centrally manages multi-step provisioning workflows using the Orchestrated Saga pattern.

Architecture

                    Portal Backend
                         │
                         │ dataspace.create.requested
                         ▼
              ┌───────────────────────┐
              │      Orchestrator     │
              │                       │
              │    Saga State (DB)    │
              │    Workflow Engine    │
              │    Compensation Mgr   │
              └──┬───────┬────────┬───┘
                 │       │        │
   frost.project.│   apisix.route.│  redpanda.pipeline.
   create/delete │   create/delete│  deploy/delete
                 │       │        │
                 ▼       ▼        ▼
              ┌──────┐┌──────┐┌────────┐
              │FROST ││APISIX││Redpanda│
              │Adapt.││Adapt.││Adapt.  │
              └──────┘└──────┘└────────┘
                 │        │        │
                 │  result events  │
                 └────────┴────────┘
                          │
                          ▼
               ┌─────────────────────┐
               │    Orchestrator     │
               │  (updates saga)     │
               └─────────────────────┘
                          │
                          │ dataspace.create.completed / .failed
                          ▼
                     Portal Backend

Orchestrator Responsibilities

The orchestrator is the only component with workflow knowledge:

Responsibility	Description
Workflow execution	Executes steps sequentially, passing outputs from one step as inputs to the next
Saga state management	Persists step results (resource IDs, URLs) to survive restarts
Data mapping	Builds adapter-specific commands from previous step outputs (e.g., projectId → APISIX upstream URL)
Compensation	On failure, triggers DELETE commands for all successfully completed steps — in parallel
Status reporting	Reports final success/failure to the Portal Backend via a single result event

Adapter Responsibilities

Adapters remain pure command handlers. They do not know whether a command is part of a saga, a manual operation, or a compensation:

Adapter receives: frost.project.create {name: "my-dataset"}
Adapter returns:  frost.project.result  {status: SUCCESS, resourceId: "proj-123",
                                         resultData: {"projectId": "proj-123",
                                                      "baseUrl": "http://frost/v1.1/projects/proj-123"}}

Adapter receives: frost.project.delete {projectId: "proj-123"}
Adapter returns:  frost.project.result  {status: SUCCESS}

The adapter does not know the delete is a rollback. It is a regular delete operation.

Success Flow

1. Backend → Kafka:  dataspace.create.requested {backendType: "FROST", ...}
2. Orchestrator creates saga, persists state
3. Orchestrator → FROST Adapter:   frost.project.create
4. FROST Adapter → Orchestrator:   frost.project.result {projectId, baseUrl}
5. Orchestrator persists projectId, builds APISIX command
6. Orchestrator → APISIX Adapter:  apisix.route.create {upstream: baseUrl, uri: ...}
7. APISIX Adapter → Orchestrator:  apisix.route.result {routeId}
8. Orchestrator persists routeId, builds Redpanda command
9. Orchestrator → Redpanda Adapter: redpanda.pipeline.deploy {targetUrl, pipelineJson}
10. Redpanda Adapter → Orchestrator: redpanda.pipeline.result {pipelineId}
11. Orchestrator → Backend:         dataspace.create.completed

Failure Flow (Redpanda Fails)

Saga State after step 7:
  FROST:    SUCCESS  (proj-123)
  APISIX:   SUCCESS  (route-456)
  Redpanda: FAILED   (connection refused)

Orchestrator triggers parallel compensation:
  → apisix.route.delete  {routeId: "route-456"}
  → frost.project.delete {projectId: "proj-123"}

Both compensations complete independently.
Orchestrator → Backend: dataspace.create.failed {reason, compensated: true}

Workflow Definition (Conceptual)

workflows:
  dataspace-create-frost:
    trigger: "dataspace.create.requested"
    condition: "payload.backendType == 'FROST'"
    steps:
      - name: create-frost-project
        command: "frost.project.create"
        input:
          projectName: "{{payload.dataspaceName}}"
        output:
          projectId: "{{result.resultData.projectId}}"
          baseUrl: "{{result.resultData.baseUrl}}"
        compensate:
          command: "frost.project.delete"
          input:
            projectId: "{{steps.create-frost-project.output.projectId}}"

      - name: create-apisix-route
        command: "apisix.route.create"
        input:
          uri: "/api/dataspace/{{payload.dataspaceId}}/*"
          upstreamUrl: "{{steps.create-frost-project.output.baseUrl}}"
          methods: ["GET"]
        output:
          routeId: "{{result.resultData.routeId}}"
        compensate:
          command: "apisix.route.delete"
          input:
            routeId: "{{steps.create-apisix-route.output.routeId}}"

      - name: deploy-pipelines
        command: "redpanda.pipeline.deploy"
        input:
          pipelineJson: "{{payload.pipelineJson}}"
          targetUrl: "{{steps.create-frost-project.output.baseUrl}}"
        compensate:
          command: "redpanda.pipeline.delete"
          input:
            pipelineId: "{{steps.deploy-pipelines.output.pipelineId}}"

Adding a new backend (e.g., Stellio) means adding a new workflow definition — no existing adapter code changes.

Required API Extension

The ConfigResultEvent needs one additional field to carry structured result data:

// New field (backwards-compatible, nullable)
private Map<String, String> resultData;

Existing adapters that do not set resultData continue to work via the existing resourceId field.

Topic Structure

Topic	Publisher	Subscriber	Purpose
core.civitas.dataspace.create.requested	Portal Backend	Orchestrator	Trigger provisioning workflow
core.civitas.frost.project.create	Orchestrator	FROST Adapter	Create FROST project
core.civitas.frost.project.delete	Orchestrator	FROST Adapter	Delete FROST project (or compensate)
core.civitas.frost.project.result	FROST Adapter	Orchestrator	Step result
core.civitas.apisix.route.create	Orchestrator	APISIX Adapter	Create APISIX route
core.civitas.apisix.route.delete	Orchestrator	APISIX Adapter	Delete APISIX route (or compensate)
core.civitas.apisix.route.result	APISIX Adapter	Orchestrator	Step result
core.civitas.redpanda.pipeline.deploy	Orchestrator	Redpanda Adapter	Deploy pipeline
core.civitas.redpanda.pipeline.delete	Orchestrator	Redpanda Adapter	Delete pipeline (or compensate)
core.civitas.redpanda.pipeline.result	Redpanda Adapter	Orchestrator	Step result
core.civitas.dataspace.create.completed	Orchestrator	Portal Backend	Saga succeeded
core.civitas.dataspace.create.failed	Orchestrator	Portal Backend	Saga failed (with compensation status)

Saga State Machine

           ┌──────────┐
           │ PENDING  │
           └────┬─────┘
                │ first step started
                ▼
           ┌──────────┐
     ┌─────│EXECUTING │─────┐
     │     └──────────┘     │
     │ all steps done       │ step failed
     ▼                      ▼
┌─────────┐          ┌──────────────┐
│COMPLETED│          │ COMPENSATING │
└─────────┘          └──────┬───────┘
                            │
                ┌───────────┴───────────┐
                │                       │
                ▼                       ▼
         ┌────────────┐      ┌─────────────────────┐
         │COMPENSATED │      │ COMPENSATION_FAILED │
         │  (clean)   │      │(manual intervention)│
         └────────────┘      └─────────────────────┘

Limitation: Saga State Persistence

The orchestrator requires persistent storage for saga state. Options:

Storage	Pros	Cons
Kafka Streams state store	No external DB needed	Harder to query, operational complexity
PostgreSQL (shared with backend)	Proven, transactional, queryable	Shared DB dependency
Dedicated DB	Full isolation	Additional infrastructure

Recommendation: Use PostgreSQL (shared or dedicated), as the stack already includes it.

Consequences

Benefits

• Adapters stay simple: No workflow knowledge, no compensation chains, no predecessor/successor awareness
• Central workflow visibility: Saga state is queryable in one place — enables dashboards, debugging, and manual intervention
• Parallel compensation: On failure, all successful steps are compensated simultaneously (not cascading)
• Scalable to new backends: Adding Stellio/NGSI-LD = new workflow definition, zero adapter code changes
• Existing adapters unchanged: APISIX and Keycloak adapters continue to work as-is
• Idempotent by design: Orchestrator can safely retry failed steps or compensations using existing adapter idempotency

Trade-offs

• Additional deployment unit: One new module (config-adapter-orchestrator) to build, deploy, and monitor
• Saga state storage: Requires a database for persistence (PostgreSQL recommended)
• Development effort: Orchestrator framework must be built (workflow engine, state management, compensation logic)
• Single point of coordination: Orchestrator must be highly available; if it is down, no new sagas start (in-flight sagas resume on restart)

Affected Components

Component	Change	Impact
config-adapter-api	Add resultData: Map<String, String> to ConfigResultEvent	Backwards-compatible, minimal
config-adapter-orchestrator	New module	New development
config-adapter-apisix	None	Unchanged
config-adapter-keycloak	None	Unchanged
config-adapter-application	Wire orchestrator module	Minimal
Portal Backend	Send dataspace.create.requested, receive completed/failed	Minor
FROST Adapter (new)	Implement as simple command handler	New development
Redpanda Adapter (new)	Implement as simple command handler	New development

Alternatives Considered

Alternative	Decision	Rationale
Choreography-Based Saga	Rejected	Adapters accumulate workflow routing and compensation logic; cascading rollback breaks on partial failure; every new backend requires changes to existing adapters. See detailed analysis (TODO: add link).
Portal Backend as Orchestrator (Option A)	Deferred	Viable long-term, but requires significant backend changes for async Kafka processing and saga state management. The dedicated module can be migrated into the backend later. A standalone orchestrator has the benefit of decoupled extensibility and maintenance
Process engine (Redpanda)		Introduces heavyweight infrastructure dependency; overkill for the current number of workflows. Can be reconsidered if workflow complexity grows significantly.

Migration Path

1. Phase 1: Build config-adapter-orchestrator with saga state management and the FROST dataspace workflow
2. Phase 2: Refactor FROST adapters as simple command handlers. Create Redpanda adapter as simple command handler
3. Phase 3: Connect Portal Backend to orchestrator topics
4. Phase 4 (optional): Add Stellio/NGSI-LD workflow definition — existing adapters remain unchanged
5. Long-term (optional): Migrate orchestrator logic into the Portal Backend if architecturally desired

References

• Saga Pattern — Microsoft Architecture Center
• Choreography vs. Orchestration — microservices.io
• When Not to Use Choreography — Bernd Rücker
• CIVITAS Architecture Principles
• Issue #832: Proposal SAGA Pattern for config-adapters