Version: V2-Next

ADR 042: Kafka Authorization via OPA with Ternary Grant Model

Date: 2026-03-27

Status: To Be Reviewed

Decision Makers: Architecture Board

Context

The CIVITAS/CORE V2 data platform uses Apache Kafka as its central message bus. With authentication established via SASL/SCRAM-SHA-512 (ADR 042), an authorization model is needed to control what each authenticated principal may do on which topics.

The platform already uses Open Policy Agent (OPA) as its central Policy Decision Point (PDP) for API authorization (APISIX → OPA → AuthZ Repository → PostgreSQL). Extending OPA to cover Kafka authorization unifies policy management across the platform.

Kafka's native ACL mechanism is limited to static allow/deny rules without support for dynamic, data-driven policies or integration with the platform's existing authorization database.

The authorization model must support the platform's two message categories:

Configuration events (de.civitascore.*) -- static topic grants defined at installation time
Data (de.civitascore.payload.<dataset>.*) -- dynamic topic grants that change when datasets are published or removed

A key requirement is the ternary authorization model: two principals with the same role (e.g., data-producer) must be able to have independent, potentially disjoint sets of topic grants. Role assignment alone does not imply topic access.

The full concept is documented in the Authorization Concept.

Checked Architecture Principles

[full] Model-centric data flow
[full] Distributed architecture with unified user experience
[full] Modular design -- OPA policies are modular and independently testable
[full] Integration capability through defined interfaces -- OPA HTTP API, PostgreSQL as shared data store
[full] Open source as the default -- OPA is CNCF-graduated open source
[full] Cloud-native architecture
[full] Prefer standard solutions over custom development -- reuses existing OPA infrastructure
[full] Self-contained deployment
[full] Technological consistency to ensure maintainability -- single PDP for API and Kafka
[full] Security by design -- fail-secure defaults, least privilege, audit logging

Decision

Open Policy Agent (OPA) is adopted as the authorization engine for Kafka, replacing Kafka's native ACL mechanism. Authorization decisions follow a ternary model (principal × role × topic grants) and use the existing PostgreSQL authorization database as the single source of truth.

Key Design Decisions

Custom Kafka Authorizer delegating to OPA -- A Java plugin implementing org.apache.kafka.server.authorizer.Authorizer sends authorization requests to OPA via HTTP. Fail-secure: all requests are denied if OPA is unreachable.

Ternary authorization model -- Each principal is independently assigned roles (defining the type of access) and topic grants (defining which topics). Two principals with the same role can have different topic grant sets.

Principal	Role	Topic Grants
`config-outbox-relay-producer`	`config-producer`	`de.civitascore.config.*`
`config-pipeline-umwelt-consumer`	`config-consumer`	`de.civitascore.config.*`
`dataset-luftqualitaet-producer`	`data-producer`	`de.civitascore.payload.luftqualitaet.*`
`dataset-wasserqualitaet-producer`	`data-producer`	`de.civitascore.payload.wasserqualitaet.*`
`dataset-luftqualitaet-consumer`	`data-consumer`	`de.civitascore.payload.luftqualitaet.*`

The table illustrates the key property: dataset-luftqualitaet-producer and dataset-wasserqualitaet-producer share the same role but hold disjoint topic grants. Role assignment alone does not imply topic access.

PostgreSQL as single source of truth -- Topic grants are stored in the existing authorization database, shared with the API authorization layer. The required data model introduces the concepts of kafka_principals, kafka_principal_roles, and kafka_topic_grants. Whether these are implemented as separate tables, as an extension of the existing user/role model, or in a dedicated schema is left to the detailed design (Feinkonzept). External platform users and internal Kafka principals are structurally different, but unification is possible if the data model allows it.
Event-driven policy refresh instead of periodic polling -- Configuration event topic grants are loaded at OPA startup (static). Data topic grants are refreshed event-driven when the Management Portal triggers a dataset publish/update. No periodic database polling required.
Dedicated OPA policy package -- Kafka authorization uses civitas.kafka.authz, separate from the existing API authorization policies (civitas.authz). Both packages run in the same OPA instance but are independently maintainable.

For the full Rego policy, PostgreSQL schema, caching strategy, and implementation details, see the Authorization Concept.

Consequences

Kafka's native ACL mechanism is not used. All authorization goes through the Custom Authorizer → OPA path.
A custom Kafka Authorizer plugin (Java) must be implemented and maintained.
The existing PostgreSQL authorization database gains three new tables for Kafka principal and grant management.
A Bundle Service is needed to export grant data from PostgreSQL to OPA (can be part of existing event handling, an AuthZ Repository extension, or a standalone service).
OPA decision logs provide built-in audit logging for all Kafka authorization decisions.
The Authorizer adds a network hop (broker → OPA) per authorization decision; a local decision cache (configurable TTL) mitigates latency impact.
The SASL username from authentication (ADR 042) must match kafka_principals.principal_name in the authorization database.

Alternatives

Kafka native ACLs: Rejected because they are static, lack integration with the platform's existing authorization database, and do not support the ternary model (role + independent topic grants per principal).
Kafka native ACLs with external sync: Rejected because ACL sync tools add operational complexity without the policy flexibility of OPA. Does not support event-driven refresh.
OPA with periodic polling instead of event-driven refresh: Rejected because configuration event grants are static (no polling needed) and data grants change at known, discrete events (dataset publish). Periodic polling would add unnecessary database load.

Context​

Checked Architecture Principles​

Decision​

Key Design Decisions​

Consequences​

Alternatives​

See also​