ADR 043: Kafka Authentication via SASL/SCRAM-SHA-512

Date: 2026-03-27

Status: To Be Reviewed

Decision Makers: Architecture Board

Context

The CIVITAS/CORE V2 data platform uses Apache KafkaApache KafkaA distributed event streaming platform. In CIVITAS/CORE it is used as the message bus to transport events, models and data in data flows. as its central message bus for both configuration events and payload data. Until now, no formal authentication mechanism has been defined for KafkaApache KafkaA distributed event streaming platform. In CIVITAS/CORE it is used as the message bus to transport events, models and data in data flows. clients.

The platform must support two deployment scenarios:

• With service mesh -- Kubernetes environments where mTLS is already provided at the infrastructure level (e.g., Istio, Linkerd)
• Without service mesh -- Environments where KafkaApache KafkaA distributed event streaming platform. In CIVITAS/CORE it is used as the message bus to transport events, models and data in data flows. must handle authentication independently

Clients are predominantly technical users (platform components and dataset pipeline processes). Human users access KafkaApache KafkaA distributed event streaming platform. In CIVITAS/CORE it is used as the message bus to transport events, models and data in data flows. only for administration and troubleshooting. The authentication mechanism must support automated credential provisioning for technical users and integrate with the platform's existing secrets management.

Technical users further divide into two categories with different lifecycle requirements:

• Component principals -- long-lived platform infrastructure (outbox relay, saga orchestrator, config adapters), provisioned at deployment time via Kubernetes Secrets
• Pipeline principals -- dataset-specific producers and consumers, dynamically created and deleted by Config Adapters via the platform Secrets Management

The full concept is documented in the Authentication Concept.

Checked Architecture Principles

• [full] Distributed architecture with unified user experience
• [full] Modular design
• [full] Integration capability through defined interfaces
• [full] Cloud-native architecture -- works with and without service mesh
• [full] Prefer standard solutions over custom development -- SASLSimple Authentication and Security LayerA framework defined in RFC 4422 that decouples authentication from application protocols./SCRAMSalted Challenge Response Authentication MechanismA challenge-response authentication protocol defined in RFC 5802. is a KafkaApache KafkaA distributed event streaming platform. In CIVITAS/CORE it is used as the message bus to transport events, models and data in data flows.-native mechanism
• [full] Self-contained deployment -- no external PKI dependency
• [full] Technological consistency to ensure maintainability
• [full] Security by design -- challenge-response protocol, no cleartext passwords, salted hashes

Decision

SASLSimple Authentication and Security LayerA framework defined in RFC 4422 that decouples authentication from application protocols./SCRAMSalted Challenge Response Authentication MechanismA challenge-response authentication protocol defined in RFC 5802.-SHA-512 is adopted as the primary authentication mechanism for all KafkaApache KafkaA distributed event streaming platform. In CIVITAS/CORE it is used as the message bus to transport events, models and data in data flows. clients in both deployment scenarios.

Key Design Decisions

1. SASLSimple Authentication and Security LayerA framework defined in RFC 4422 that decouples authentication from application protocols./SCRAMSalted Challenge Response Authentication MechanismA challenge-response authentication protocol defined in RFC 5802.-SHA-512 over mTLS for KafkaApache KafkaA distributed event streaming platform. In CIVITAS/CORE it is used as the message bus to transport events, models and data in data flows. authentication -- Works identically with and without service mesh. No dependency on certificate infrastructure. Avoids double TLS termination conflicts with mesh sidecars. Simpler credential rotation (password change vs. certificate reissue).

2. Dual listener configuration -- SASL_SSL (port 9093) for environments without service mesh (KafkaApache KafkaA distributed event streaming platform. In CIVITAS/CORE it is used as the message bus to transport events, models and data in data flows. terminates TLS); SASL_PLAINTEXT (port 9092) for environments with service mesh (sidecar handles TLS).

3. Two-tier credential provisioning -- Component principals are provisioned by the deployment pipeline and distributed via Kubernetes Secrets. Pipeline principals are dynamically managed by Config Adapters and distributed via platform Secrets Management.

4. Config Adapter manages pipeline principal lifecycle -- Config Adapters create SCRAMSalted Challenge Response Authentication MechanismA challenge-response authentication protocol defined in RFC 5802. credentials when datasets are published and delete them when datasets are removed, ensuring credentials do not outlive the resources they protect.

For implementation details, listener configuration, credential rotation procedures, and naming conventions, see the Authentication Concept.

Consequences

• All KafkaApache KafkaA distributed event streaming platform. In CIVITAS/CORE it is used as the message bus to transport events, models and data in data flows. clients must authenticate via SASLSimple Authentication and Security LayerA framework defined in RFC 4422 that decouples authentication from application protocols./SCRAMSalted Challenge Response Authentication MechanismA challenge-response authentication protocol defined in RFC 5802.-SHA-512. Unauthenticated connections are rejected.
• The deployment pipeline must be extended to create SCRAMSalted Challenge Response Authentication MechanismA challenge-response authentication protocol defined in RFC 5802. credentials and Kubernetes Secrets for component principals.
• Config Adapters must be extended to manage SCRAMSalted Challenge Response Authentication MechanismA challenge-response authentication protocol defined in RFC 5802. credentials and Secrets Management entries for pipeline principals.
• Transport encryption is always present: via KafkaApache KafkaA distributed event streaming platform. In CIVITAS/CORE it is used as the message bus to transport events, models and data in data flows.-native TLS (without mesh) or via mesh-provided mTLS (with mesh).
• Inter-broker authentication is not covered by this ADRArchitecture Decision RecordA document capturing an architecture decision. Each ADR has a stable identifier and short title and is managed through four lifecycle states: Proposed, Accepted, Deprecated and Superseded. -- it is managed by the KafkaApache KafkaA distributed event streaming platform. In CIVITAS/CORE it is used as the message bus to transport events, models and data in data flows. Operator.
• Credential rotation requires a two-phase approach (create new → update store → restart → remove old) to avoid downtime.
• The SASLSimple Authentication and Security LayerA framework defined in RFC 4422 that decouples authentication from application protocols. username must match the principal name used in the authorization layer (ADR 041).

Alternatives

• mTLS as primary mechanism: Rejected because it conflicts with service mesh mTLS (double termination), requires PKI infrastructure, and complicates credential rotation. It would also require different authentication paths for mesh vs. non-mesh deployments.
• SASLSimple Authentication and Security LayerA framework defined in RFC 4422 that decouples authentication from application protocols./PLAIN: Rejected because it transmits credentials in cleartext (requires TLS to be secure) and the server must store or access plaintext passwords.
• OAuth/OIDC (SASLSimple Authentication and Security LayerA framework defined in RFC 4422 that decouples authentication from application protocols./OAUTHBEARER): Rejected as overengineered for technical service-to-service authentication. Adds dependency on an OAuth provider and token refresh complexity for unattended components.

See also

• Authentication Concept: Full concept with listener configuration, credential lifecycle, and naming conventions
• ADR 043: KafkaApache KafkaA distributed event streaming platform. In CIVITAS/CORE it is used as the message bus to transport events, models and data in data flows. Authorization via OPAOpen Policy AgentThe platform's authorization service: it evaluates 'who may do what' against the CIVITAS/CORE authorization model. OPA is the platform's central Policy Decision Point (PDP) for API authorization.
• ADR 026: Select Message Bus