ADR 042: Kafka Authorization via OPA with Ternary Grant Model

Date: 2026-03-27

Status: To Be Reviewed

Decision Makers: Architecture Board

Context

The CIVITAS/CORE V2 data platform uses Apache KafkaApache KafkaA distributed event streaming platform. In CIVITAS/CORE it is used as the message bus to transport events, models and data in data flows. as its central message bus. With authentication established via SASLSASL (Simple Authentication and Security Layer)A framework defined in RFC 4422 that decouples authentication from application protocols./SCRAMSCRAM (Salted Challenge Response Authentication Mechanism)A challenge-response authentication protocol defined in RFC 5802.-SHA-512 (ADR 042), an authorization model is needed to control what each authenticated principal may do on which topics.

The platform already uses Open Policy AgentOpen Policy AgentThe platform's authorization service: it evaluates 'who may do what' against the CIVITAS/CORE authorization model. OPA is the platform's central Policy Decision Point (PDP) for API authorization. (OPAOpen Policy AgentThe platform's authorization service: it evaluates 'who may do what' against the CIVITAS/CORE authorization model. OPA is the platform's central Policy Decision Point (PDP) for API authorization.) as its central Policy Decision PointPolicy Decision PointThe component that decides whether a request is authorized. CIVITAS/CORE uses Open Policy Agent (OPA) as its PDP. (PDPPolicy Decision PointThe component that decides whether a request is authorized. CIVITAS/CORE uses Open Policy Agent (OPA) as its PDP.) for API authorization (APISIXApache APISIXAn open-source API gateway for traffic management, security and observability. In CIVITAS/CORE it is used as the centralized entrypoint to route and protect externally exposed APIs. → OPAOpen Policy AgentThe platform's authorization service: it evaluates 'who may do what' against the CIVITAS/CORE authorization model. OPA is the platform's central Policy Decision Point (PDP) for API authorization. → AuthZ Repository → PostgreSQL). Extending OPAOpen Policy AgentThe platform's authorization service: it evaluates 'who may do what' against the CIVITAS/CORE authorization model. OPA is the platform's central Policy Decision Point (PDP) for API authorization. to cover KafkaApache KafkaA distributed event streaming platform. In CIVITAS/CORE it is used as the message bus to transport events, models and data in data flows. authorization unifiesApache NiFiA stream processing and connector framework. In CIVITAS/CORE it is the pipeline engine for data integration and transformation (see ADR 047) and implements dataset-defined data flows. policy management across the platform.

KafkaApache KafkaA distributed event streaming platform. In CIVITAS/CORE it is used as the message bus to transport events, models and data in data flows.'s native ACL mechanism is limited to static allow/deny rules without support for dynamic, data-driven policies or integration with the platform's existing authorization database.

The authorization model must support the platform's two message categories:

• Configuration events (de.civitascore.*) -- static topic grants defined at installation time
• Data (de.civitascore.payload.<dataset>.*) -- dynamic topic grants that change when datasetsDatasetA data-related element that contains processed data and makes it available for consumption. A Dataset is populated via Pipelines and carries Metadata and access permissions. are published or removed

A key requirement is the ternary authorization model: two principals with the same role (e.g., data-producer) must be able to have independent, potentially disjoint sets of topic grants. Role assignment alone does not imply topic access.

The full concept is documented in the Authorization Concept.

Checked Architecture Principles

• [full] Model-centric data flow
• [full] Distributed architecture with unified user experience
• [full] Modular design -- OPAOpen Policy AgentThe platform's authorization service: it evaluates 'who may do what' against the CIVITAS/CORE authorization model. OPA is the platform's central Policy Decision Point (PDP) for API authorization. policies are modular and independently testable
• [full] Integration capability through defined interfaces -- OPAOpen Policy AgentThe platform's authorization service: it evaluates 'who may do what' against the CIVITAS/CORE authorization model. OPA is the platform's central Policy Decision Point (PDP) for API authorization. HTTP API, PostgreSQL as shared data store
• [full] Open source as the default -- OPAOpen Policy AgentThe platform's authorization service: it evaluates 'who may do what' against the CIVITAS/CORE authorization model. OPA is the platform's central Policy Decision Point (PDP) for API authorization. is CNCF-graduated open source
• [full] Cloud-native architecture
• [full] Prefer standard solutions over custom development -- reuses existing OPAOpen Policy AgentThe platform's authorization service: it evaluates 'who may do what' against the CIVITAS/CORE authorization model. OPA is the platform's central Policy Decision Point (PDP) for API authorization. infrastructure
• [full] Self-contained deployment
• [full] Technological consistency to ensure maintainability -- single PDPPolicy Decision PointThe component that decides whether a request is authorized. CIVITAS/CORE uses Open Policy Agent (OPA) as its PDP. for API and KafkaApache KafkaA distributed event streaming platform. In CIVITAS/CORE it is used as the message bus to transport events, models and data in data flows.
• [full] Security by design -- fail-secure defaults, least privilege, audit logging

Decision

Open Policy AgentOpen Policy AgentThe platform's authorization service: it evaluates 'who may do what' against the CIVITAS/CORE authorization model. OPA is the platform's central Policy Decision Point (PDP) for API authorization. (OPAOpen Policy AgentThe platform's authorization service: it evaluates 'who may do what' against the CIVITAS/CORE authorization model. OPA is the platform's central Policy Decision Point (PDP) for API authorization.) is adopted as the authorization engine for KafkaApache KafkaA distributed event streaming platform. In CIVITAS/CORE it is used as the message bus to transport events, models and data in data flows., replacing KafkaApache KafkaA distributed event streaming platform. In CIVITAS/CORE it is used as the message bus to transport events, models and data in data flows.'s native ACL mechanism. Authorization decisions follow a ternary model (principal × role × topic grants) and use the existing PostgreSQL authorization database as the single source of truth.

Key Design Decisions

1. Custom KafkaApache KafkaA distributed event streaming platform. In CIVITAS/CORE it is used as the message bus to transport events, models and data in data flows. Authorizer delegating to OPAOpen Policy AgentThe platform's authorization service: it evaluates 'who may do what' against the CIVITAS/CORE authorization model. OPA is the platform's central Policy Decision Point (PDP) for API authorization. -- A Java plugin implementing org.apache.kafka.server.authorizer.Authorizer sends authorization requests to OPAOpen Policy AgentThe platform's authorization service: it evaluates 'who may do what' against the CIVITAS/CORE authorization model. OPA is the platform's central Policy Decision Point (PDP) for API authorization. via HTTP. Fail-secure: all requests are denied if OPAOpen Policy AgentThe platform's authorization service: it evaluates 'who may do what' against the CIVITAS/CORE authorization model. OPA is the platform's central Policy Decision Point (PDP) for API authorization. is unreachable.

2. Ternary authorization model -- Each principal is independently assigned roles (defining the type of access) and topic grants (defining which topics). Two principals with the same role can have different topic grant sets.

   Principal	Role	Topic Grants
   config-outbox-relay-producer	config-producer	de.civitascore.config.*
   config-pipeline-umwelt-consumer	config-consumer	de.civitascore.config.*
   dataset-luftqualitaet-producer	data-producer	de.civitascore.payload.luftqualitaet.*
   dataset-wasserqualitaet-producer	data-producer	de.civitascore.payload.wasserqualitaet.*
   dataset-luftqualitaet-consumer	data-consumer	de.civitascore.payload.luftqualitaet.*

   The table illustrates the key property: dataset-luftqualitaet-producer and dataset-wasserqualitaet-producer share the same role but hold disjoint topic grants. Role assignment alone does not imply topic access.

3. PostgreSQL as single source of truth -- Topic grants are stored in the existing authorization database, shared with the API authorization layer. The required data model introduces the concepts of kafka_principals, kafka_principal_roles, and kafka_topic_grants. Whether these are implemented as separate tables, as an extension of the existing user/role model, or in a dedicated schema is left to the detailed design (Feinkonzept). External platform users and internal KafkaApache KafkaA distributed event streaming platform. In CIVITAS/CORE it is used as the message bus to transport events, models and data in data flows. principals are structurally different, but unification is possible if the data model allows it.

4. Event-driven policy refresh instead of periodic polling -- Configuration event topic grants are loaded at OPAOpen Policy AgentThe platform's authorization service: it evaluates 'who may do what' against the CIVITAS/CORE authorization model. OPA is the platform's central Policy Decision Point (PDP) for API authorization. startup (static). Data topic grants are refreshed event-driven when the Management PortalManagement PortalThe central user interface of the Platform that provides access to all functionalities for managing data, configurations, users, and access. It serves as the main entry point for working with data-related elements. triggers a datasetDatasetA data-related element that contains processed data and makes it available for consumption. A Dataset is populated via Pipelines and carries Metadata and access permissions. publish/update. No periodic database polling required.

5. Dedicated OPAOpen Policy AgentThe platform's authorization service: it evaluates 'who may do what' against the CIVITAS/CORE authorization model. OPA is the platform's central Policy Decision Point (PDP) for API authorization. policy package -- KafkaApache KafkaA distributed event streaming platform. In CIVITAS/CORE it is used as the message bus to transport events, models and data in data flows. authorization uses civitas.kafka.authz, separate from the existing API authorization policies (civitas.authz). Both packages run in the same OPAOpen Policy AgentThe platform's authorization service: it evaluates 'who may do what' against the CIVITAS/CORE authorization model. OPA is the platform's central Policy Decision Point (PDP) for API authorization. instance but are independently maintainable.

For the full RegoRegoOPA's declarative policy language, used to express authorization rules such as checking role assignments at the most specific scope first. policy, PostgreSQL schema, caching strategy, and implementation details, see the Authorization Concept.

Consequences

• KafkaApache KafkaA distributed event streaming platform. In CIVITAS/CORE it is used as the message bus to transport events, models and data in data flows.'s native ACL mechanism is not used. All authorization goes through the Custom Authorizer → OPAOpen Policy AgentThe platform's authorization service: it evaluates 'who may do what' against the CIVITAS/CORE authorization model. OPA is the platform's central Policy Decision Point (PDP) for API authorization. path.
• A custom KafkaApache KafkaA distributed event streaming platform. In CIVITAS/CORE it is used as the message bus to transport events, models and data in data flows. Authorizer plugin (Java) must be implemented and maintained.
• The existing PostgreSQL authorization database gains three new tables for KafkaApache KafkaA distributed event streaming platform. In CIVITAS/CORE it is used as the message bus to transport events, models and data in data flows. principal and grant management.
• A Bundle Service is needed to export grant data from PostgreSQL to OPAOpen Policy AgentThe platform's authorization service: it evaluates 'who may do what' against the CIVITAS/CORE authorization model. OPA is the platform's central Policy Decision Point (PDP) for API authorization. (can be part of existing event handling, an AuthZ Repository extension, or a standalone service).
• OPAOpen Policy AgentThe platform's authorization service: it evaluates 'who may do what' against the CIVITAS/CORE authorization model. OPA is the platform's central Policy Decision Point (PDP) for API authorization. decision logs provide built-in audit logging for all KafkaApache KafkaA distributed event streaming platform. In CIVITAS/CORE it is used as the message bus to transport events, models and data in data flows. authorization decisions.
• The Authorizer adds a network hop (broker → OPAOpen Policy AgentThe platform's authorization service: it evaluates 'who may do what' against the CIVITAS/CORE authorization model. OPA is the platform's central Policy Decision Point (PDP) for API authorization.) per authorization decision; a local decision cache (configurable TTL) mitigates latency impact.
• The SASLSASL (Simple Authentication and Security Layer)A framework defined in RFC 4422 that decouples authentication from application protocols. username from authentication (ADR 042) must match kafka_principals.principal_name in the authorization database.

Alternatives

• KafkaApache KafkaA distributed event streaming platform. In CIVITAS/CORE it is used as the message bus to transport events, models and data in data flows. native ACLs: Rejected because they are static, lack integration with the platform's existing authorization database, and do not support the ternary model (role + independent topic grants per principal).
• KafkaApache KafkaA distributed event streaming platform. In CIVITAS/CORE it is used as the message bus to transport events, models and data in data flows. native ACLs with external sync: Rejected because ACL sync tools add operational complexity without the policy flexibility of OPAOpen Policy AgentThe platform's authorization service: it evaluates 'who may do what' against the CIVITAS/CORE authorization model. OPA is the platform's central Policy Decision Point (PDP) for API authorization.. Does not support event-driven refresh.
• OPAOpen Policy AgentThe platform's authorization service: it evaluates 'who may do what' against the CIVITAS/CORE authorization model. OPA is the platform's central Policy Decision Point (PDP) for API authorization. with periodic polling instead of event-driven refresh: Rejected because configuration event grants are static (no polling needed) and data grants change at known, discrete events (datasetDatasetA data-related element that contains processed data and makes it available for consumption. A Dataset is populated via Pipelines and carries Metadata and access permissions. publish). Periodic polling would add unnecessary database load.

See also

• Authorization Concept: Full concept with RegoRegoOPA's declarative policy language, used to express authorization rules such as checking role assignments at the most specific scope first. policies, PostgreSQL schema, caching strategy, and implementation steps
• ADR 042: KafkaApache KafkaA distributed event streaming platform. In CIVITAS/CORE it is used as the message bus to transport events, models and data in data flows. Authentication via SASLSASL (Simple Authentication and Security Layer)A framework defined in RFC 4422 that decouples authentication from application protocols./SCRAMSCRAM (Salted Challenge Response Authentication Mechanism)A challenge-response authentication protocol defined in RFC 5802.-SHA-512
• ADR 026: Select Message Bus
• ADR 041: Revised Topic Naming Convention for Configuration Events