Authorization Concept

Status: Concept -- Draft

This document describes a planned authorization model for the Kafka message bus. It has not been started to be implemented and is subject to review by the Architecture Board.

Objective

The concept defines a Kafka authorization (AuthZ) model for the CIVITAS/CORE V2 data platform that enforces the least privilege principle: every component receives exactly the permissions required for its purpose on working with the central message bus.

Authorization decisions are delegated to Open Policy Agent (OPA) via a custom Kafka Authorizer plugin. OPA is already the platform's central Policy Decision Point (PDP) for API authorization (APISIX → OPA → AuthZ Repository) and is extended here to cover Kafka topic access. The existing PostgreSQL-based authorization database serves as the single source of truth for both API and Kafka permissions.

The concept covers both message categories handled by the bus:

• Configuration messages -- distributed and consumed via dedicated Configuration Adapters. The Outbox and Saga patterns are currently planned and partly implemented for this category.
• Payload data -- distributed via dataset pipelines with dataset-specific topic structures. No Outbox or Saga usage is planned for payload data at this time.

Scope and Boundaries

Authentication vs. Authorization

Aspect	Scope
Authentication (AuthN)	Identity of a producer/consumer via SASL/SCRAM-SHA-512 -- out of scope for this document, see Authentication Concept
Authorization (AuthZ)	What an authenticated identity may do on which topic -- in scope

Relation to Existing OPA Infrastructure

The platform already operates an OPA-based authorization chain for API access:

APISIX → OPA (Rego policies) → AuthZ Repository → PostgreSQL

This concept extends OPA to cover Kafka:

Kafka Broker → Custom Authorizer → OPA (Rego policies, in-memory bundle)

Both paths share the same PostgreSQL database as the single source of truth. The key difference is how OPA accesses policy data at decision time — see OPA Decision-Time Data Flow below.

Both paths share the same PostgreSQL database for permission data, ensuring a single source of truth across API and message bus authorization.

Recommended: Separate OPA Instances for API and Kafka

Recommendation: deploy separate OPA instances for API authorization and Kafka authorization.

	Shared Instance	Separate Instances (recommended)
Failure domain	OPA down → API + bus both unavailable	Independent — Kafka-OPA down affects bus only
Exposure profile	Mixed (external API + internal broker)	Isolated per concern
Operational effort	Lower	Higher (two deployments to operate)
Data flow pattern	Conflict: API uses runtime-fetch, Kafka uses in-memory bundle	Each instance can be optimised independently

Two factors make separation the preferred direction: first, the threat model — an OPA failure with fail-secure behaviour would simultaneously block the API and the entire message bus, which is a disproportionate blast radius. Second, the data flow patterns diverge fundamentally: the existing APISIX–OPA integration fetches assignments from the backend DB at decision time, while Kafka authorization uses an in-memory bundle refreshed event-driven. Running both patterns in the same OPA instance creates operational and reasoning complexity.

The "single PDP" architecture principle (ADR041) refers to the technology choice (OPA), not the deployment topology, and is satisfied by both options.

Dataset Definition

A dataset is the central organizing unit for payload data on the message bus. Each dataset may involve one or more pipelines and requires one or more Kafka topics to support them (e.g., raw ingestion, enrichment, error handling). All topics of a dataset share a common namespace prefix:

de.civitascore.data.<dataset>.*

Each dataset has:

• A dedicated namespace within the payload prefix (e.g., de.civitascore.data.luftqualitaet.*)
• One or more topics within that namespace, one per pipeline stage or concern
• Dataset-specific principals with dedicated topic grants

Relation to Platform Authorization Model

This concept deliberately uses Kafka-native terminology throughout. The implementation team works directly with Strimzi, OPA, and Kafka documentation — all of which use the same terms. Translating to platform vocabulary inside this document would create a mismatch with every external reference and make implementation harder, not easier.

The translation happens at the boundary: this concept is the bridge between the platform domain model and the Kafka tool domain. The table below provides that translation for readers who need to reason across both models:

Platform AuthZ Model	Kafka AuthZ Concept	Notes
Permission (atomic right: READ, CREATE, ...)	Role (config-consumer, data-producer)	Both describe the type of allowed operation.
Role (bundle of permissions, e.g. "Data Architect")	(no equivalent)	Kafka principals are assigned operation types directly — there is no bundling layer.
Scope (DataSet, DataPool, Platform)	Topic Grant (topic pattern)	Both describe where a right applies.
Assignment (Group × Role × Scope)	Principal × Role × Topic Grant	Structurally identical. A Kafka principal is the bearer of the assignment.

Outside this concept — in platform architecture documents, ADRs, and cross-cutting discussions — the platform vocabulary applies.

Principal Model

Each Kafka client instance has a unique principal -- Kafka's native term for a technical user identity. Clients authenticate via SASL/SCRAM-SHA-512 (see Authentication Concept); the SASL username becomes the Kafka principal. Principals are assigned to one or more roles. Roles define the type of access (produce, consume, admin), while topic grants define which specific topics a principal may access.

Ternary Authorization Model

Authorization decisions are always ternary -- they are evaluated along three dimensions:

Key property: Two principal instances can hold the same role but receive different, independent sets of topic grants. Topic grant sets may be disjoint, identical, or overlapping. This enables fine-grained, per-instance access control without role proliferation.

Example:

Principal	Role	Topic Grants
config-frost-adapter-consumer	config-consumer	de.civitascore.config.frost.*
config-apisix-adapter-consumer	config-consumer	de.civitascore.config.apisix.*
dataset-luftqualitaet-producer	data-producer	de.civitascore.data.luftqualitaet.*
dataset-zaehlstellen-producer	data-producer	de.civitascore.data.zaehlstellen.*

Both config-consumers share the role but are scoped to their own adapter's config namespace. Both data-producers share the role but have disjoint topic namespaces — the defining property of the ternary model.

Principal Types

Principal Type	Description	Example
config-producer	Instance that produces configuration events	config-outbox-relay-producer, config-saga-orchestrator-producer
config-consumer	Instance that consumes configuration events	config-pipeline-a-consumer, config-saga-orchestrator-consumer
dataset-producer	Producer for a dataset's pipelines	dataset-luftqualitaet-producer
dataset-consumer	Consumer for a dataset's pipelines	dataset-luftqualitaet-consumer
platform-admin	Platform operations (restricted, audited)	admin-mmustermann

Principal Separation Rules

• Each component receives its own principal -- no credential sharing
• Producer and consumer roles are assigned separately, even if a component acts as both
• Infrastructure components (e.g., outbox relay, saga orchestrator) receive dedicated principals, separate from application principals
• Each principal receives its own set of topic grants -- role assignment alone does not imply topic access

Compatibility with Existing Patterns

The authorization model must be compatible with the platform's established messaging patterns. Both of the following patterns are currently scoped to configuration events only and operate within the de.civitascore.config.* namespace.

Saga Pattern (details): The Saga orchestrator sends commands to and receives results from multiple configuration adapter topics. Its Kafka principal requires both produce and consume access on de.civitascore.config.*. This is achieved by assigning both config-producer and config-consumer roles to the orchestrator's dedicated principal, with topic grants covering the full de.civitascore.config.* namespace.

Outbox Pattern (details): Application components (esp. the management portal backend) write configuration events to a local outbox table. A separate relay process publishes these events to Kafka. The relay process receives a dedicated principal with the config-producer role and corresponding topic grants; the application component itself has no direct Kafka principal.

High-Value Targets

The Saga Orchestrator and Outbox Relay are high-value targets from a security perspective:

• The Saga Orchestrator holds READ + WRITE access across the entire de.civitascore.config.* namespace, has access to saga state (internal URLs, project IDs, route IDs), and can trigger compensation cascades (deletions).
• The Outbox Relay can inject arbitrary configuration events into the bus.

A compromise of either component would have platform-wide impact. The following hardening measures should be evaluated by the implementation team:

Measure	Target	Rationale
Dedicated DB instance for saga state	Saga Orchestrator	Separates saga state from portal and authz data; limits blast radius of a portal-level SQL injection
HMAC/checksum on saga state entries	Saga Orchestrator	Detects DB-level tampering without additional infrastructure cost
Compensation rate limiting	Saga Orchestrator	Limits impact of a compromised orchestrator triggering deletion cascades
Anomaly alerting on compensation events	Saga Orchestrator	Detects unusual compensation patterns that may indicate compromise
Dedicated audit log for compensation events	Saga Orchestrator	Enables forensic analysis separate from the general OPA decision log

Topic Namespace Structure

A consistent namespace structure is a prerequisite for efficient, rule-based access control. The platform uses two distinct namespace prefixes that enable clear separation of configuration and payload traffic.

Configuration Events

Configuration events use the fixed platform prefix de.civitascore.config, structured by adapter, entity, and event type. See Topic Configuration for the full event catalog.

de.civitascore.config.<adapter>.<entity>.<event>

Examples:

• de.civitascore.config.frost.project.created
• de.civitascore.config.apisix.route.deleted
• de.civitascore.config.idm.user.created

Saga topics are part of this namespace. SAGA commands and results (e.g. de.civitascore.config.frost.project.create) are routed through de.civitascore.config.* and are therefore fully covered by the config event grant model. There is no separate topic family for saga traffic.

Payload Data

Payload data topics use the fixed platform prefix de.civitascore.data, structured by dataset and topic name. Each dataset may require multiple topics for its pipelines (e.g., raw ingestion, enriched output, error handling).

de.civitascore.data.<dataset>.<topic>

Examples (dataset luftqualitaet with three topics):

• de.civitascore.data.luftqualitaet.raw
• de.civitascore.data.luftqualitaet.enriched
• de.civitascore.data.luftqualitaet.errors

Examples (dataset zaehlstellen with two topics):

• de.civitascore.data.zaehlstellen.raw
• de.civitascore.data.zaehlstellen.enriched

Access is granted per dataset using a wildcard pattern: de.civitascore.data.<dataset>.*

Permission Matrix

Configuration Event Permissions

Principal Type	Topic Pattern	Produce	Consume	Description
config-producer (outbox relay)	de.civitascore.config.<adapter>.* per known adapter	yes	--	Explicitly scoped per adapter namespace — no wildcard. New adapters require an explicit grant addition.
config-producer (saga orchestrator)	de.civitascore.config.*	yes	--	Needs full namespace access to send commands to all adapters
config-consumer	de.civitascore.config.<adapter>.*	--	yes	Scoped to own adapter's config namespace (e.g. de.civitascore.config.frost.*)
config-consumer (saga)	de.civitascore.config.*	--	yes	Saga orchestrator consumes results across all domains

A config adapter must not read configuration events intended for other domains. Cross-component configuration is routed exclusively through the central config producer.

Payload Data

Principal Type	Topic Pattern	Produce	Consume	Description
data-producer (Dataset X)	de.civitascore.data.<dataset-x>.*	yes	--	Dataset producer writes only own topics
data-consumer (Dataset X)	de.civitascore.data.<dataset-x>.*	--	yes	Dataset consumer reads only own topics

A principal that needs read access to another dataset's topics simply receives an additional topic grant for that dataset's namespace. This is a regular topic grant entry in PostgreSQL -- no separate approval mechanism is required beyond the standard grant process.

Access Control Implementation -- OPA-based Kafka Authorizer

Architecture Overview

Authorization is enforced through a custom Kafka Authorizer that delegates decisions to OPA. This replaces Kafka's native ACL mechanism with the platform's central policy engine.

Request Flow

1. A Kafka client (authenticated via SASL/SCRAM-SHA-512, see Authentication Concept) issues a produce or consume request
2. The Kafka broker invokes the custom Authorizer with the request context:
   • principal -- the authenticated Kafka identity
   • operation -- WRITE, READ, CREATE, DESCRIBE, DELETE, etc.
   • resource_type -- TOPIC, GROUP, CLUSTER, TRANSACTIONAL_ID
   • resource_name -- the specific topic name or consumer group ID
3. The Authorizer sends an HTTP POST to OPA with the structured request
4. OPA evaluates the request against its Rego policy using pre-loaded policy data from PostgreSQL
5. OPA returns { "result": { "allow": true/false, "reason": "..." } }
6. The Authorizer maps the decision to Kafka's AuthorizationResult

Kafka Authorizer Plugin

The platform runs on Strimzi, which has an existing community OPA Authorizer plugin that already handles TOPIC, GROUP, CLUSTER, and TRANSACTIONAL_ID resources and delegates decisions to OPA over HTTP. This is the preferred approach: adopt the Strimzi OPA Authorizer and focus platform effort on the Rego policy rather than Java plumbing.

Configuration (Strimzi Kafka CR):

spec:
  kafka:
    authorization:
      type: custom
      authorizerClass: org.openpolicyagent.kafka.OpaAuthorizer
      superUsers:
        - User:admin
      configuration:
        - name: opa.authorizer.url
          value: http://opa-kafka:8181/v1/data/civitas/kafka/authz/allow
        - name: opa.authorizer.allow.on.error
          value: "false"
        - name: opa.authorizer.cache.expire.after.seconds
          value: "30"

Fail-secure: If OPA is unreachable, the Authorizer denies all requests (allow.on.error=false). A short-lived local cache reduces latency for repeated decisions without compromising security.

Fallback: Custom Authorizer

If the implementation team identifies blockers with the Strimzi OPA Authorizer (e.g. incompatibility with the platform's Strimzi version, missing features, or policy data model constraints), a custom Java plugin implementing org.apache.kafka.server.authorizer.Authorizer can be developed as a fallback. The Rego policy and PostgreSQL schema remain identical in both cases.

OPA Policy for Kafka Authorization

OPA evaluates Kafka requests using a dedicated Rego policy package (civitas.kafka.authz) separate from the existing API authorization policies (civitas.authz).

Policy Data Model

OPA receives policy data from PostgreSQL via the Bundle Service (see OPA Bundle Sync from PostgreSQL). The data is structured as follows:

{
  "kafka_principals": {
    "config-frost-adapter-consumer": {
      "roles": ["config-consumer"],
      "topic_grants": [
        {
          "topic_pattern": "de.civitascore.config.frost.*",
          "operations": ["READ"]
        }
      ]
    },
    "config-outbox-relay-producer": {
      "roles": ["config-producer"],
      "topic_grants": [
        {
          "topic_pattern": "de.civitascore.config.frost.*",
          "operations": ["WRITE", "DESCRIBE"]
        },
        {
          "topic_pattern": "de.civitascore.config.apisix.*",
          "operations": ["WRITE", "DESCRIBE"]
        }
      ]
    },
    "dataset-luftqualitaet-producer": {
      "roles": ["data-producer"],
      "topic_grants": [
        {
          "topic_pattern": "de.civitascore.data.luftqualitaet.*",
          "operations": ["WRITE", "DESCRIBE"]
        }
      ]
    }
  }
}

Rego Policy

# CIVITAS CORE Kafka AuthZ Policy
# Evaluates Kafka broker authorization requests against topic grants.
#
# Ternary model: principal × role × topic_grants
# Two principals with the same role can have independent topic grant sets.

package civitas.kafka.authz

import rego.v1

# Default deny -- fail secure
default allow := false
default decision := {"allow": false, "reason": "default_deny"}

# Main decision rule
decision := result if {
    result := evaluate_request
}

allow if {
    decision.allow
}

# --- Principal lookup ---

principal_entry := data.kafka_principals[input.principal]

# --- Grant evaluation ---

# Allow if at least one topic_grant matches both topic and operation
evaluate_request := {"allow": true, "reason": "topic_grant_matched"} if {
    principal_entry
    some grant in principal_entry.topic_grants
    topic_matches(grant.topic_pattern, input.resource_name)
    input.operation in grant.operations
}

# Known principal but no matching grant
evaluate_request := {"allow": false, "reason": "no_matching_grant"} if {
    principal_entry
    not has_matching_grant
}

# Unknown principal
evaluate_request := {"allow": false, "reason": "unknown_principal"} if {
    not principal_entry
}

has_matching_grant if {
    some grant in principal_entry.topic_grants
    topic_matches(grant.topic_pattern, input.resource_name)
    input.operation in grant.operations
}

# --- Topic pattern matching ---
# Supports exact match and wildcard suffix (e.g., "de.civitascore.config.*")

topic_matches(pattern, topic) if {
    not endswith(pattern, ".*")
    pattern == topic
}

topic_matches(pattern, topic) if {
    endswith(pattern, ".*")
    prefix := trim_suffix(pattern, "*")
    startswith(topic, prefix)
}

# --- Consumer Group access ---
# Consumer groups follow the naming convention: cg-<principal-name>
# e.g. cg-dataset-luftqualitaet-consumer
# A principal may only use consumer groups that match its own name.

evaluate_request := {"allow": true, "reason": "consumer_group_matched"} if {
    input.resource_type == "GROUP"
    principal_entry
    input.resource_name == concat("", ["cg-", input.principal])
}

evaluate_request := {"allow": false, "reason": "consumer_group_not_allowed"} if {
    input.resource_type == "GROUP"
    principal_entry
    input.resource_name != concat("", ["cg-", input.principal])
}

Platform-Admin Override

The platform-admin role receives unrestricted access. This is handled as a dedicated rule:

# Platform admin bypass -- all operations allowed (audited separately)
evaluate_request := {"allow": true, "reason": "platform_admin"} if {
    principal_entry
    "platform-admin" in principal_entry.roles
}

PostgreSQL Schema Extension

The existing authorization database is extended with tables for Kafka principal management. These tables are part of the same database used by the AuthZ Repository for API permissions.

Open Design Decision: Database Separation

The platform operates on a shared DB cluster; the level of control available is at the database level, not the server or cluster level. The meaningful separation options are therefore:

Option	Blast Radius	Notes
Single database, shared schemas	Portal compromise → access to authz + saga state	Simplest, least isolation
Separate databases per concern	Isolated per DB — portal compromise does not automatically reach saga or authz DB	Requires separate connection strings and DB users per component
Single database, schema-level role separation	Portal DB user has access only to its own schema	Pragmatic middle ground; relies on correct role configuration

Note: full isolation is structurally incomplete regardless of approach — the portal backend must write to authz tables (grant provisioning). The most pragmatic mitigation is schema-level role separation: each component's DB user is granted only the minimum required permissions per schema, preventing lateral movement from a portal-level compromise into saga state or unrelated authz data. The implementation team should decide based on the platform's threat model and the DB provisioning model in use.

-- Kafka principal identities
CREATE TABLE kafka_principals (
    id              UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    principal_name  VARCHAR(255) NOT NULL UNIQUE,
    description     TEXT,
    is_active       BOOLEAN NOT NULL DEFAULT true,
    created_at      TIMESTAMP NOT NULL DEFAULT NOW(),
    updated_at      TIMESTAMP NOT NULL DEFAULT NOW()
);

-- Reuse existing roles table (role_type = 'KAFKA')
-- INSERT INTO roles (name, description, role_type)
-- VALUES ('config-producer', '...', 'KAFKA');

-- Principal-to-role assignment
CREATE TABLE kafka_principal_roles (
    id              UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    principal_id    UUID NOT NULL REFERENCES kafka_principals(id),
    role_id         UUID NOT NULL REFERENCES roles(id),
    created_at      TIMESTAMP NOT NULL DEFAULT NOW(),
    UNIQUE (principal_id, role_id)
);

-- Topic grants (the per-principal topic access sets)
CREATE TABLE kafka_topic_grants (
    id              UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    principal_id    UUID NOT NULL REFERENCES kafka_principals(id),
    topic_pattern   VARCHAR(512) NOT NULL,
    operations      VARCHAR(255)[] NOT NULL,  -- e.g., '{READ}', '{WRITE,DESCRIBE}'
    valid_from      TIMESTAMP NOT NULL DEFAULT NOW(),
    valid_until     TIMESTAMP,                -- NULL = no expiry
    granted_by      VARCHAR(255),             -- who approved the grant
    approval_ref    VARCHAR(512),             -- reference to approval (e.g., ticket ID)
    created_at      TIMESTAMP NOT NULL DEFAULT NOW(),
    UNIQUE (principal_id, topic_pattern)
);

-- Index for OPA bundle export query
CREATE INDEX idx_kafka_topic_grants_principal
    ON kafka_topic_grants(principal_id)
    WHERE valid_until IS NULL OR valid_until > NOW();

OPA Decision-Time Data Flow

Kafka AuthZ uses an in-memory bundle — not runtime database queries.

At decision time, OPA evaluates Kafka authorization requests entirely against data held in memory. There is no database query per authorization request. This is a deliberate divergence from the existing APISIX–OPA integration, where OPA fetches assignments from the AuthZ Repository at decision time.

	APISIX–OPA (existing)	Kafka–OPA (this concept)
Data access at decision time	Runtime fetch from AuthZ Repository	In-memory bundle
Latency per decision	DB round-trip	Memory lookup
Grant freshness	Always current	Eventually consistent (refreshed event-driven)
Suitable for	API requests (low frequency, complex queries)	Kafka broker (high frequency, simple lookup)

The in-memory bundle is the correct pattern for Kafka: every produce and consume request triggers an authorization check, so a database round-trip per decision would have unacceptable throughput impact. Grants change only at known, discrete events (dataset publish/delete), making event-driven refresh sufficient.

This divergence in data flow pattern is an additional reason to run separate OPA instances for API and Kafka authorization (see Recommended: Separate OPA Instances).

Dataset publishing as a Saga with compensation:

Dataset publishing is modelled as a Saga. If any step fails (grant write, OPA refresh, principal creation, or credential distribution), the Saga compensates all completed steps in reverse order — removing the principal from Kafka, removing the grants from PostgreSQL, and triggering an OPA bundle refresh. This ensures no dangling principals with grants exist after a failed publish.

OPA Bundle Sync from PostgreSQL

OPA receives policy data as a bundle that is assembled from PostgreSQL by a lightweight Bundle Service. Data is loaded once at OPA startup and then refreshed event-driven when the Bundle Service is triggered (see Caching and Refresh Strategy).

Bundle export query:

SELECT
    kp.principal_name,
    array_agg(DISTINCT r.name) AS roles,
    json_agg(json_build_object(
        'topic_pattern', ktg.topic_pattern,
        'operations', ktg.operations
    )) AS topic_grants
FROM kafka_principals kp
LEFT JOIN kafka_principal_roles kpr ON kpr.principal_id = kp.id
LEFT JOIN roles r ON r.id = kpr.role_id
LEFT JOIN kafka_topic_grants ktg ON ktg.principal_id = kp.id
    AND (ktg.valid_until IS NULL OR ktg.valid_until > NOW())
WHERE kp.is_active = true
GROUP BY kp.principal_name;

The bundle service can be implemented as:

• As part of the existing eventhandling for configuration events
• An extension of the existing AuthZ Repository
• An OPA plugin using the opa-envoy-plugin or custom Go plugin for direct PostgreSQL queries

Caching and Refresh Strategy

The two topic categories have fundamentally different change characteristics, which the caching strategy reflects:

Configuration event topics (de.civitascore.config.*) are defined at installation time and remain static during platform runtime. Changes only occur through platform updates/upgrades. Their topic grants can be loaded at OPA startup and do not require periodic refresh.

Payload data topics (de.civitascore.data.*) change when datasets are published or updated through the Management Portal. These changes are discrete, known events -- not continuous background changes.

Event-Driven Cache Invalidation

Instead of periodic polling, the Bundle Service supports targeted, event-driven refresh triggered by the Management Portal backend:

This approach avoids unnecessary periodic database queries. The Management Portal already knows when topic grants change (dataset publication, principal provisioning) and can trigger a targeted refresh for the affected dataset.

Grant Provisioning

Topic grants for dataset principals are written to kafka_topic_grants by the Management Portal backend as part of the dataset publish process — not by the Config Adapter. This is an explicit design decision to ensure race-condition-free principal provisioning:

1. Backend writes topic grants to kafka_topic_grants (PostgreSQL)
2. Backend triggers Bundle Service refresh → OPA receives updated grants
3. Backend sends configuration event → Config Adapter creates principal in Kafka
4. Config Adapter reports back to SAGA Orchestrator → credentials stored in Secrets Management

This ordering guarantees that OPA already holds the correct grants before the principal exists in Kafka. An authorization failure immediately after principal creation is therefore not possible.

Cache Layers

Layer	Cache	Refresh	Purpose
Kafka Authorizer	Local decision cache (principal + operation + resource → result)	30s TTL (configurable)	Reduce HTTP round-trips for repeated operations
OPA	In-process policy data	Event-driven (push from Bundle Service)	Fresh policy data without polling
Bundle Service	Query cache per dataset	Invalidated on refresh trigger	Reduce database load

Refresh Triggers

Event	Trigger Source	Scope
Dataset published / updated	Management Portal	Affected dataset only
New principal provisioned	Platform Operations	Affected principal's dataset(s)
Platform update / upgrade	Deployment pipeline	Full reload (all datasets)
Principal compromise	Platform Operations (emergency)	Immediate full reload + Authorizer cache flush

Emergency Invalidation

For urgent changes (e.g., principal compromise), the standard event-driven flow is supplemented by direct intervention:

1. Setting is_active = false on the principal in PostgreSQL
2. Triggering an immediate full OPA bundle reload (Bundle Service API or OPA API: POST /v1/data)
3. Flushing the Kafka Authorizer cache via JMX or Kafka admin API

Audit Logging

OPA provides built-in decision logging that captures every authorization decision with full context:

{
  "decision_id": "...",
  "input": {
    "principal": "config-frost-adapter-consumer",
    "operation": "READ",
    "resource_type": "TOPIC",
    "resource_name": "de.civitascore.config.frost.project.created"
  },
  "result": {
    "allow": true,
    "reason": "topic_grant_matched"
  },
  "timestamp": "2026-03-27T10:15:30Z"
}

• All decisions (allow and deny) are logged by OPA decision logs
• Decision logs can be forwarded to the platform's central logging infrastructure
• For payload data topics, log volume can be managed via OPA's decision log sampling configuration
• The existing OPA decision log masking policy (mask.rego) can be extended for Kafka-specific fields

Periodic Review

• Quarterly review of all topic grants (check approval_ref and valid_until in kafka_topic_grants)
• Annual review of all principals and topic grants for currency
• Expired grants (valid_until < NOW()) are automatically excluded from OPA bundle sync

Permission Registry

The permission registry is implemented as the PostgreSQL tables described in PostgreSQL Schema Extension. It replaces a static document artifact with a queryable, machine-enforced data store.

For reporting and review purposes, the following view provides a human-readable registry:

CREATE VIEW kafka_permission_registry AS
SELECT
    kp.principal_name,
    kp.is_active,
    array_agg(DISTINCT r.name) AS roles,
    ktg.topic_pattern,
    ktg.operations,
    ktg.valid_until,
    ktg.granted_by,
    ktg.approval_ref
FROM kafka_principals kp
LEFT JOIN kafka_principal_roles kpr ON kpr.principal_id = kp.id
LEFT JOIN roles r ON r.id = kpr.role_id
LEFT JOIN kafka_topic_grants ktg ON ktg.principal_id = kp.id
GROUP BY kp.principal_name, kp.is_active, ktg.topic_pattern,
         ktg.operations, ktg.valid_until, ktg.granted_by, ktg.approval_ref
ORDER BY kp.principal_name, ktg.topic_pattern;

Implementation Steps Overview

#	Task
1	Implement custom Kafka Authorizer plugin (Java)
2	Extend OPA Rego policies for Kafka (civitas.kafka.authz package)
3	Implement or extend bundle service for PostgreSQL → OPA sync
4	Create PostgreSQL schema migration for Kafka AuthZ tables
5	Populate initial topic grants for existing components
6	Define audit logging pipeline for OPA Kafka decision logs
7	Formalize topic grant approval process
8	Load testing: Measure Authorizer + OPA latency impact on broker throughput
9	Evaluate Authorizer decision cache TTL vs. refresh latency trade-offs

Glossary

Term	Definition
AuthN	Authentication -- identity verification
AuthZ	Authorization -- access control
Bundle Service	Lightweight service that exports PostgreSQL-based policy data as OPA bundles
Dataset	A published data product on the platform; the central organizing unit for payload topics. A dataset may involve one or more pipelines and one or more Kafka topics within its namespace (de.civitascore.data.<dataset>.*)
Custom Authorizer	Kafka plugin implementing org.apache.kafka.server.authorizer.Authorizer that delegates to OPA
OPA	Open Policy Agent -- the platform's central Policy Decision Point for authorization
Outbox Pattern	Transactionally safe event publishing via an intermediate database table
PDP	Policy Decision Point -- component that evaluates authorization requests (OPA)
Principal	Kafka's native term for a technical user identity -- the authenticated identity of a Kafka client. In this platform, the SASL/SCRAM username serves as the principal. Corresponds to java.security.Principal
Rego	OPA's declarative policy language
Saga Pattern	Distributed transaction pattern across multiple services/topics
Topic Grant	A mapping of a principal to a topic pattern with allowed operations -- the per-instance access control unit

Related Documentation

• Authentication Concept
• Message Bus Overview
• Topic Configuration -- Current State
• Outbox Pattern
• Saga Pattern