Security Hardening

This document describes the security controls implemented in CIVITAS/CORE V2 and possible extensions for operators who want to further harden their deployment.

Already Implemented

The following security measures are built into the default deployment configuration.

Pod Security

All components run with restrictive security contexts defined in defaults/environment/security.yaml.gotmpl:

Control	Setting	Scope
Non-root containers	runAsNonRoot: true, runAsUser: 1000	All pods
Read-only root filesystem	readOnlyRootFilesystem: true	All containers
No privilege escalation	allowPrivilegeEscalation: false	All containers
Dropped capabilities	capabilities.drop: [ALL]	All containers
Seccomp profile	seccompProfile.type: RuntimeDefault	All pods
File system group	fsGroup: 1000	All pods

These defaults are enforced by component value templates and validated by Kyverno policies in CI/CD.

Kyverno Security Policies

info

Our Kyverno Security Policies are currently in a early state and will be expanded over time. We welcome contributions to enhance our security posture.

The deployment repository includes Kyverno policies (in .ci/policies/) that validate manifests at pipeline runtime in the repository. These policies can also be deployed to the cluster for runtime enforcement.

Base policies (all environments):

The base set applies the standard Kyverno Pod Security Standards policies (baseline + restricted profiles). On top of those, CIVITAS/CORE adds these checks:

Policy	Enforces
require-ro-rootfs	Root filesystem must be read-only
validate-probes	Startup, liveness, and readiness probes must be configured
require-ingress-class	Ingress resources must specify an ingressClassName

Production policies (production profile):

Policy	Enforces
require-pdb	PodDisruptionBudget with minAvailable >= 1
require-rolling-update	RollingUpdate with maxUnavailable: 0, maxSurge: 25%
require-minimum-replicas	Either multiple replicas or HPA must be configured
require-resources	CPU and memory requests and limits must be set

Validate policies locally:

# All components
just verify-policies

# Single component
.ci/policies/verify-kyverno-policies.sh <component-name>

Runtime Kyverno Policies

In addition to the shift-left CI scan above, the platform ships in-cluster Kyverno ClusterPolicy objects as the runtime-policies component (enabled by default). They enforce invariants that only exist at runtime — injected sidecars and namespace annotations, and Pods that operators create dynamically — which the CI scan of rendered manifests cannot see.

Prerequisite: Kyverno must be running in the cluster — see Prerequisites → Kyverno.

Policy	Enforces	Scope
require-linkerd-sidecar	Pods in meshed namespaces must carry the injected linkerd-proxy sidecar	Linkerd
require-meshed-namespace-inbound-policy	Meshed namespaces must set a default-inbound-policy annotation	Linkerd
justify-linkerd-inject-opt-out	Opting a workload out of injection requires an explicit justification annotation	Linkerd
restrict-cnpg-operator-pod-labels	The CloudNativePGCloudNativePGA Kubernetes operator for running and managing PostgreSQL clusters. In CIVITAS/CORE it is used to provision and operate Postgres databases. operator may only set app.kubernetes.io/name to its own components	Operators
restrict-strimzi-operator-pod-labels	The Strimzi operator may only set app.kubernetes.io/name to its own components	Operators
protect-operator-owned-labels	Operator-owned label values may only be set by the Pods that operator creates (anti-spoofing)	Operators

The Linkerd policies activate automatically only when the service mesh is enabled. The operator policies key on the creating identity (request.userInfo) and pin to the shared operators' ServiceAccount usernames, protecting the label-based NetworkPolicy authorization against spoofing by dynamically created operator Pods.

Enforcement level is controlled by global.runtimePolicies.failureAction (Audit by default — violations are only reported; switch to Enforce to block at admission once the reports are clean):

global:
  runtimePolicies:
    enabled: true
    failureAction: Audit   # or Enforce

Inspect what is being flagged:

kubectl get clusterpolicyreport,policyreport -A

warning

Keep require-linkerd-sidecar on Audit until Linkerd injection is reliably healthy: under Enforce, if the Linkerd proxy injector is down, no Pod in a meshed namespace will be admitted.

info

The runtime ClusterPolicy objects are cluster-scoped, so they are deployed once per cluster as part of the shared operator layer (helmfile -f deployment/helmfile-operators.yaml sync) — not per instance. On multi-instance clusters this is handled automatically via deployLayer gating; see Deployment → Deploying multiple instances.

RBAC and Least Privilege

• ServiceAccounts: Each component creates its own ServiceAccount with automountServiceAccountToken: false by default
• Etcd RBAC: Etcd uses role-based access control with per-component users (e.g., apisix user with read-write access only to /apisix/* prefix)
• CloudNativePGCloudNativePGA Kubernetes operator for running and managing PostgreSQL clusters. In CIVITAS/CORE it is used to provision and operate Postgres databases.: The operator creates dedicated database roles per component with minimal privileges (no superuser)
• Cluster Admin: Required only for initial deployment (CRD installation); day-to-day operations use namespace-scoped permissions

Secret Management

• Secrets are auto-generated by the secrets component using random passwords (configurable length)
• Secrets are stored as Kubernetes Secrets (base64-encoded in etcd)
• No plaintext secrets in configuration files or Helm values
• Database passwords are generated per component and injected via secret references
• Secret definitions are in components/<component>/secrets.yaml

Network Policies

Network policies are defined per component to restrict pod-to-pod communication. Each policy follows a default-deny model with explicit allow rules.

Component	Allowed Ingress From
PostgreSQL	CloudNativePGCloudNativePGA Kubernetes operator for running and managing PostgreSQL clusters. In CIVITAS/CORE it is used to provision and operate Postgres databases. operator, Portal (backend), KeycloakKeycloakAn open-source Identity and Access Management (IAM) solution providing SSO and OAuth2/OpenID Connect flows. In CIVITAS/CORE it is used to authenticate users and issue JWTs., Authz, FROST, database role setup jobs
Etcd	APISIXApache APISIXAn open-source API gateway for traffic management, security and observability. In CIVITAS/CORE it is used as the centralized entrypoint to route and protect externally exposed APIs., Etcd peers
KafkaApache KafkaA distributed event streaming platform. In CIVITAS/CORE it is used as the message bus to transport events, models and data in data flows.	Portal, KafkaApache KafkaA distributed event streaming platform. In CIVITAS/CORE it is used as the message bus to transport events, models and data in data flows. UI, config-adapters, NiFiApache NiFiA stream processing and connector framework. In CIVITAS/CORE it is the pipeline engine for data integration and transformation (see ADR 047) and implements dataset-defined data flows.
KeycloakKeycloakAn open-source Identity and Access Management (IAM) solution providing SSO and OAuth2/OpenID Connect flows. In CIVITAS/CORE it is used to authenticate users and issue JWTs.	APISIXApache APISIXAn open-source API gateway for traffic management, security and observability. In CIVITAS/CORE it is used as the centralized entrypoint to route and protect externally exposed APIs., config-adapter, keycloakKeycloakAn open-source Identity and Access Management (IAM) solution providing SSO and OAuth2/OpenID Connect flows. In CIVITAS/CORE it is used to authenticate users and issue JWTs.-config-cli

info

Network policies are enabled by default in each component's default-environment.yaml.gotmpl but require a CNI plugin with NetworkPolicy support (e.g., Calico, Cilium). Most managed Kubernetes services support this.

TLS and Encryption

Layer	Implementation
External traffic	TLS termination at Ingress (cert-manager with ClusterIssuer)
Service-to-service	Mandatory mTLS via Linkerd (default cluster-authenticated inbound policy) — see below
Etcd communication	Internal TLS between peers
Database connections	PostgreSQL hostssl entries in pg_hba.conf

Service Mesh Authorization (mTLS Enforcement)

When the Linkerd service mesh is enabled (the default), service-to-service traffic is not just encrypted opportunistically — it is authorized. The prepare step annotates every instance namespace with a mandatory default inbound policy, and public edges are carved out explicitly with Linkerd Server resources.

Setting	Effect	Default
global.serviceMesh.defaultInboundPolicy	Namespace-wide config.linkerd.io/default-inbound-policy. cluster-authenticated = only authenticated, meshed in-cluster clients may connect. Other values: all-authenticated, all-unauthenticated, audit (log-only rollout).	cluster-authenticated
global.serviceMesh.allowUnauthenticatedIngress	Drives the APISIXApache APISIXAn open-source API gateway for traffic management, security and observability. In CIVITAS/CORE it is used as the centralized entrypoint to route and protect externally exposed APIs. data-plane Server accessPolicy. false → all-authenticated (the ingress controller must be meshed). true → all-unauthenticated (accepts plaintext from an unmeshed ingress).	false

Consequences and safe rollout:

• The ingress controller must be meshed when allowUnauthenticatedIngress: false. Once per cluster before deploying, annotate the Ingress-namespace for Linkerd injection — set linkerd.io/inject=enabled and config.linkerd.io/skip-inbound-ports=80,443,8443 on it, then restart the controller. See Prerequisites → Linkerd.
• Kubelet liveness/readiness probes are auto-authorized by Linkerd, so mandatory mTLS does not break health checks.
• Only the public APISIXApache APISIXAn open-source API gateway for traffic management, security and observability. In CIVITAS/CORE it is used as the centralized entrypoint to route and protect externally exposed APIs. data-plane port is exposed as a Server; all other APISIXApache APISIXAn open-source API gateway for traffic management, security and observability. In CIVITAS/CORE it is used as the centralized entrypoint to route and protect externally exposed APIs. ports (admin, control) stay under the mandatory namespace default.
• For a phased rollout, set defaultInboundPolicy: audit first to log what would be denied, then switch to cluster-authenticated once the reports are clean.

Image Security

• Container image tags are pinned to explicit versions (no latest tags); a few images are pinned via the upstream Helm chart's own values rather than in images.yaml
• Multi-architecture builds (amd64/arm64) via CI/CD
• Trivy vulnerability scanning in CI pipeline (container-qa stage)
• SBOMSoftware Bill of MaterialsA machine-readable inventory of a software artifact's components, generated in CycloneDX format as part of the supply-chain security measures (alongside dependency pinning, vulnerability scanning, signed artifacts, and SLSA-aligned provenance). generation via Syft (CycloneDX format) in CI pipeline
• Images are built from known base images with minimal attack surface

CI/CD Security

• Pre-commit hooks enforce code quality (YAML lint, Helm lint, formatting)
• Kyverno policy validation in CI pipeline
• Container image scanning with Trivy (GitLab report format)
• Manual approval gates for staging and production deployments

Possible Extensions

The following measures are not part of the default deployment but can be implemented by operators to further harden their environment.

Runtime Policy Enforcement

In-cluster runtime enforcement is now built in via the runtime-policies component — see Runtime Kyverno Policies. To harden further, you can also deploy the shift-left CI policy sets (.ci/policies/base/, .ci/policies/production/) to the cluster for continuous enforcement of the manifest-level rules, and switch the runtime policies from Audit to Enforce:

# Enforce the built-in runtime policies (once reports are clean)
# global.runtimePolicies.failureAction: Enforce

# Additionally enforce the shift-left policy sets in-cluster
kubectl apply -f .ci/policies/base/
kubectl apply -f .ci/policies/production/  # for production clusters

warning

Test policies in Audit mode before switching to Enforce to avoid blocking legitimate workloads.

External Secret Management

Replace Kubernetes Secrets with an external secret manager for enhanced security:

• External Secrets Operator with HashiCorp VaultHashiCorp VaultA secrets management system used as the platform's secrets backend. CIVITAS/CORE supports HashiCorp Vault (BSL) and its API-compatible open-source fork OpenBao. The platform does not implement its own encryption or secret storage., AWS Secrets Manager, or Azure Key VaultHashiCorp VaultA secrets management system used as the platform's secrets backend. CIVITAS/CORE supports HashiCorp Vault (BSL) and its API-compatible open-source fork OpenBao. The platform does not implement its own encryption or secret storage.
• Encrypt secrets at rest using Kubernetes EncryptionConfiguration

Pod Security Standards

Apply Kubernetes Pod Security Standards at the namespace level:

kubectl label namespace <instanceSlug> \
  pod-security.kubernetes.io/enforce=restricted \
  pod-security.kubernetes.io/warn=restricted

Image Signing and Verification

• Sign container images with Cosign in the CI pipeline
• Verify signatures at admission time using Kyverno's image verification policies

Network Policy Hardening

• Enable network policies for all environments (not just production)
• Add egress policies to restrict outbound traffic from pods
• Fine-grained service mesh authorization: the namespace-wide mandatory mTLS policy is already enforced by default (see Service Mesh Authorization). To go further, add per-workload Linkerd Server / AuthorizationPolicy resources that restrict which meshed clients may reach a given service, beyond the namespace default

Audit Logging

• Enable Kubernetes audit logging to track API server requests
• Forward audit logs to your centralized logging stack (see Monitoring & Logging)

Regular Vulnerability Scanning

• Run Trivy scans on deployed images on a schedule (not just in CI)
• Use Trivy Operator for continuous in-cluster vulnerability scanning