Skip to main content
Version: V2-Next

ADR 043: Kafka Authentication via SASL/SCRAM-SHA-512

Date: 2026-03-27

Status: To Be Reviewed

Decision Makers: Architecture Board

Context​

The CIVITAS/CORE V2 data platform uses Apache Kafka as its central message bus for both configuration events and payload data. Until now, no formal authentication mechanism has been defined for Kafka clients.

The platform must support two deployment scenarios:

  • With service mesh -- Kubernetes environments where mTLS is already provided at the infrastructure level (e.g., Istio, Linkerd)
  • Without service mesh -- Environments where Kafka must handle authentication independently

Clients are predominantly technical users (platform components and dataset pipeline processes). Human users access Kafka only for administration and troubleshooting. The authentication mechanism must support automated credential provisioning for technical users and integrate with the platform's existing secrets management.

Technical users further divide into two categories with different lifecycle requirements:

  • Component principals -- long-lived platform infrastructure (outbox relay, saga orchestrator, config adapters), provisioned at deployment time via Kubernetes Secrets
  • Pipeline principals -- dataset-specific producers and consumers, dynamically created and deleted by Config Adapters via the platform Secrets Management

The full concept is documented in the Authentication Concept.

Checked Architecture Principles​

  • [full] Distributed architecture with unified user experience
  • [full] Modular design
  • [full] Integration capability through defined interfaces
  • [full] Cloud-native architecture -- works with and without service mesh
  • [full] Prefer standard solutions over custom development -- SASL/SCRAM is a Kafka-native mechanism
  • [full] Self-contained deployment -- no external PKI dependency
  • [full] Technological consistency to ensure maintainability
  • [full] Security by design -- challenge-response protocol, no cleartext passwords, salted hashes

Decision​

SASL/SCRAM-SHA-512 is adopted as the primary authentication mechanism for all Kafka clients in both deployment scenarios.

Key Design Decisions​

  1. SASL/SCRAM-SHA-512 over mTLS for Kafka authentication -- Works identically with and without service mesh. No dependency on certificate infrastructure. Avoids double TLS termination conflicts with mesh sidecars. Simpler credential rotation (password change vs. certificate reissue).

  2. Dual listener configuration -- SASL_SSL (port 9093) for environments without service mesh (Kafka terminates TLS); SASL_PLAINTEXT (port 9092) for environments with service mesh (sidecar handles TLS).

  3. Two-tier credential provisioning -- Component principals are provisioned by the deployment pipeline and distributed via Kubernetes Secrets. Pipeline principals are dynamically managed by Config Adapters and distributed via platform Secrets Management.

  4. Config Adapter manages pipeline principal lifecycle -- Config Adapters create SCRAM credentials when datasets are published and delete them when datasets are removed, ensuring credentials do not outlive the resources they protect.

For implementation details, listener configuration, credential rotation procedures, and naming conventions, see the Authentication Concept.

Consequences​

  • All Kafka clients must authenticate via SASL/SCRAM-SHA-512. Unauthenticated connections are rejected.
  • The deployment pipeline must be extended to create SCRAM credentials and Kubernetes Secrets for component principals.
  • Config Adapters must be extended to manage SCRAM credentials and Secrets Management entries for pipeline principals.
  • Transport encryption is always present: via Kafka-native TLS (without mesh) or via mesh-provided mTLS (with mesh).
  • Inter-broker authentication is not covered by this ADR -- it is managed by the Kafka Operator.
  • Credential rotation requires a two-phase approach (create new → update store → restart → remove old) to avoid downtime.
  • The SASL username must match the principal name used in the authorization layer (ADR 041).

Alternatives​

  • mTLS as primary mechanism: Rejected because it conflicts with service mesh mTLS (double termination), requires PKI infrastructure, and complicates credential rotation. It would also require different authentication paths for mesh vs. non-mesh deployments.
  • SASL/PLAIN: Rejected because it transmits credentials in cleartext (requires TLS to be secure) and the server must store or access plaintext passwords.
  • OAuth/OIDC (SASL/OAUTHBEARER): Rejected as overengineered for technical service-to-service authentication. Adds dependency on an OAuth provider and token refresh complexity for unattended components.

See also​