ADR 043: Kafka Authentication via SASL/SCRAM-SHA-512

Date: 2026-03-27

Status: To Be Reviewed

Decision Makers: Architecture Board

Context

The CIVITAS/CORE V2 data platform uses Apache KafkaApache KafkaA distributed event streaming platform. In CIVITAS/CORE it is used as the message bus to transport events, models and data in data flows. as its central message bus for both configuration events and payload data. Until now, no formal authentication mechanism has been defined for KafkaApache KafkaA distributed event streaming platform. In CIVITAS/CORE it is used as the message bus to transport events, models and data in data flows. clients.

The platform must support two deployment scenarios:

• With service mesh -- Kubernetes environments where mTLS is already provided at the infrastructure level (e.g., Istio, Linkerd)
• Without service mesh -- Environments where KafkaApache KafkaA distributed event streaming platform. In CIVITAS/CORE it is used as the message bus to transport events, models and data in data flows. must handle authentication independently

Clients are predominantly technical users (platform components and datasetDatasetA data-related element that contains processed data and makes it available for consumption. A Dataset is populated via Pipelines and carries Metadata and access permissions. pipeline processes). Human users access KafkaApache KafkaA distributed event streaming platform. In CIVITAS/CORE it is used as the message bus to transport events, models and data in data flows. only for administration and troubleshooting. The authentication mechanism must support automated credential provisioning for technical users and integrate with the platform's existing secrets management.

Technical users further divide into two categories with different lifecycle requirements:

• Component principals -- long-lived platform infrastructure (outbox relay, saga orchestrator, config adapters), provisioned at deployment time via Kubernetes Secrets
• Pipeline principals -- datasetDatasetA data-related element that contains processed data and makes it available for consumption. A Dataset is populated via Pipelines and carries Metadata and access permissions.-specific producers and consumers, dynamically created and deleted by Config Adapters via the platform Secrets Management

The full concept is documented in the Authentication Concept.

Checked Architecture Principles

• [full] Distributed architecture with unified user experience
• [full] Modular design
• [full] Integration capability through defined interfaces
• [full] Cloud-native architecture -- works with and without service mesh
• [full] Prefer standard solutions over custom development -- SASLSASL (Simple Authentication and Security Layer)A framework defined in RFC 4422 that decouples authentication from application protocols./SCRAMSCRAM (Salted Challenge Response Authentication Mechanism)A challenge-response authentication protocol defined in RFC 5802. is a KafkaApache KafkaA distributed event streaming platform. In CIVITAS/CORE it is used as the message bus to transport events, models and data in data flows.-native mechanism
• [full] Self-contained deployment -- no external PKI dependency
• [full] Technological consistency to ensure maintainability
• [full] Security by design -- challenge-response protocol, no cleartext passwords, salted hashes

Decision

SASLSASL (Simple Authentication and Security Layer)A framework defined in RFC 4422 that decouples authentication from application protocols./SCRAMSCRAM (Salted Challenge Response Authentication Mechanism)A challenge-response authentication protocol defined in RFC 5802.-SHA-512 is adopted as the primary authentication mechanism for all KafkaApache KafkaA distributed event streaming platform. In CIVITAS/CORE it is used as the message bus to transport events, models and data in data flows. clients in both deployment scenarios.

Key Design Decisions

1. SASLSASL (Simple Authentication and Security Layer)A framework defined in RFC 4422 that decouples authentication from application protocols./SCRAMSCRAM (Salted Challenge Response Authentication Mechanism)A challenge-response authentication protocol defined in RFC 5802.-SHA-512 over mTLS for KafkaApache KafkaA distributed event streaming platform. In CIVITAS/CORE it is used as the message bus to transport events, models and data in data flows. authentication -- Works identically with and without service mesh. No dependency on certificate infrastructure. Avoids double TLS termination conflicts with mesh sidecars. Simpler credential rotation (password change vs. certificate reissue).

2. Dual listener configuration -- SASL_SSL (port 9093) for environments without service mesh (KafkaApache KafkaA distributed event streaming platform. In CIVITAS/CORE it is used as the message bus to transport events, models and data in data flows. terminates TLS); SASL_PLAINTEXT (port 9092) for environments with service mesh (sidecar handles TLS).

3. Two-tier credential provisioning -- Component principals are provisioned by the deployment pipeline and distributed via Kubernetes Secrets. Pipeline principals are dynamically managed by Config Adapters and distributed via platform Secrets Management.

4. Config Adapter manages pipeline principal lifecycle -- Config Adapters create SCRAMSCRAM (Salted Challenge Response Authentication Mechanism)A challenge-response authentication protocol defined in RFC 5802. credentials when datasetsDatasetA data-related element that contains processed data and makes it available for consumption. A Dataset is populated via Pipelines and carries Metadata and access permissions. are published and delete them when datasetsDatasetA data-related element that contains processed data and makes it available for consumption. A Dataset is populated via Pipelines and carries Metadata and access permissions. are removed, ensuring credentials do not outlive the resources they protect.

For implementation details, listener configuration, credential rotation procedures, and naming conventions, see the Authentication Concept.

Consequences

• All KafkaApache KafkaA distributed event streaming platform. In CIVITAS/CORE it is used as the message bus to transport events, models and data in data flows. clients must authenticate via SASLSASL (Simple Authentication and Security Layer)A framework defined in RFC 4422 that decouples authentication from application protocols./SCRAMSCRAM (Salted Challenge Response Authentication Mechanism)A challenge-response authentication protocol defined in RFC 5802.-SHA-512. Unauthenticated connections are rejected.
• The deployment pipeline must be extended to create SCRAMSCRAM (Salted Challenge Response Authentication Mechanism)A challenge-response authentication protocol defined in RFC 5802. credentials and Kubernetes Secrets for component principals.
• Config Adapters must be extended to manage SCRAMSCRAM (Salted Challenge Response Authentication Mechanism)A challenge-response authentication protocol defined in RFC 5802. credentials and Secrets Management entries for pipeline principals.
• Transport encryption is always present: via KafkaApache KafkaA distributed event streaming platform. In CIVITAS/CORE it is used as the message bus to transport events, models and data in data flows.-native TLS (without mesh) or via mesh-provided mTLS (with mesh).
• Inter-broker authentication is not covered by this ADRArchitecture Decision RecordA document capturing an architecture decision. Each ADR has a stable identifier and short title and is managed through four lifecycle states: Proposed, Accepted, Deprecated and Superseded. -- it is managed by the KafkaApache KafkaA distributed event streaming platform. In CIVITAS/CORE it is used as the message bus to transport events, models and data in data flows. Operator.
• Credential rotation requires a two-phase approach (create new → update store → restart → remove old) to avoid downtime.
• The SASLSASL (Simple Authentication and Security Layer)A framework defined in RFC 4422 that decouples authentication from application protocols. username must match the principal name used in the authorization layer (ADR 041).

Alternatives

• mTLS as primary mechanism: Rejected because it conflicts with service mesh mTLS (double termination), requires PKI infrastructure, and complicates credential rotation. It would also require different authentication paths for mesh vs. non-mesh deployments.
• SASLSASL (Simple Authentication and Security Layer)A framework defined in RFC 4422 that decouples authentication from application protocols./PLAIN: Rejected because it transmits credentials in cleartext (requires TLS to be secure) and the server must store or access plaintext passwords.
• OAuth/OIDC (SASLSASL (Simple Authentication and Security Layer)A framework defined in RFC 4422 that decouples authentication from application protocols./OAUTHBEARER): Rejected as overengineered for technical service-to-service authentication. Adds dependency on an OAuth provider and token refresh complexity for unattended components.

See also

• Authentication Concept: Full concept with listener configuration, credential lifecycle, and naming conventions
• ADR 043: KafkaApache KafkaA distributed event streaming platform. In CIVITAS/CORE it is used as the message bus to transport events, models and data in data flows. Authorization via OPAOpen Policy AgentThe platform's authorization service: it evaluates 'who may do what' against the CIVITAS/CORE authorization model. OPA is the platform's central Policy Decision Point (PDP) for API authorization.
• ADR 026: Select Message Bus