Security Hardening

This document describes the security controls implemented in CIVITAS/CORE V2 and possible extensions for operators who want to further harden their deployment.

Already Implemented

The following security measures are built into the default deployment configuration.

Pod Security

All components run with restrictive security contexts defined in defaults/environment/security.yaml.gotmpl:

Control	Setting	Scope
Non-root containers	runAsNonRoot: true, runAsUser: 1000	All pods
Read-only root filesystem	readOnlyRootFilesystem: true	All containers
No privilege escalation	allowPrivilegeEscalation: false	All containers
Dropped capabilities	capabilities.drop: [ALL]	All containers
Seccomp profile	seccompProfile.type: RuntimeDefault	All pods
File system group	fsGroup: 1000	All pods

These defaults are enforced by component value templates and validated by Kyverno policies in CI/CD.

Kyverno Security Policies

info

Our Kyverno Security Policies are currently in a early state and will be expanded over time. We welcome contributions to enhance our security posture.

The deployment repository includes Kyverno policies (in .ci/policies/) that validate manifests at pipeline runtime in the repository. These policies can also be deployed to the cluster for runtime enforcement.

Base policies (all environments):

The base set applies the standard Kyverno Pod Security Standards policies (baseline + restricted profiles). On top of those, CIVITAS/CORE adds these checks:

Policy	Enforces
require-ro-rootfs	Root filesystem must be read-only
validate-probes	Startup, liveness, and readiness probes must be configured
require-ingress-class	Ingress resources must specify an ingressClassName

Production policies (production profile):

Policy	Enforces
require-pdb	PodDisruptionBudget with minAvailable >= 1
require-rolling-update	RollingUpdate with maxUnavailable: 0, maxSurge: 25%
require-minimum-replicas	Either multiple replicas or HPA must be configured
require-resources	CPU and memory requests and limits must be set

Validate policies locally:

# All components
just verify-policies

# Single component
.ci/policies/verify-kyverno-policies.sh <component-name>

Runtime Kyverno Policies

In addition to the shift-left CI scan above, the platform ships in-cluster Kyverno ClusterPolicy objects as the runtime-policies component (enabled by default). They enforce invariants that only exist at runtime — injected sidecars and namespace annotations, and Pods that operators create dynamically — which the CI scan of rendered manifests cannot see.

Prerequisite: Kyverno must be running in the cluster — see Prerequisites → Kyverno.

Policy	Enforces	Scope
require-linkerd-sidecar	Pods in meshed namespaces must carry the injected linkerd-proxy sidecar	Linkerd
require-meshed-namespace-inbound-policy	Meshed namespaces must set a default-inbound-policy annotation	Linkerd
justify-linkerd-inject-opt-out	Opting a workload out of injection requires an explicit justification annotation	Linkerd
restrict-cnpg-operator-pod-labels	The CloudNativePG operator may only set app.kubernetes.io/name to its own components	Operators
restrict-strimzi-operator-pod-labels	The Strimzi operator may only set app.kubernetes.io/name to its own components	Operators
protect-operator-owned-labels	Operator-owned label values may only be set by the Pods that operator creates (anti-spoofing)	Operators

The Linkerd policies activate automatically only when the service mesh is enabled. The operator policies key on the creating identity (request.userInfo) and pin to the shared operators' ServiceAccount usernames, protecting the label-based NetworkPolicy authorization against spoofing by dynamically created operator Pods. All rules match only CREATE/UPDATE operations, so Enforce never blocks deleting an already-admitted, non-compliant resource (e.g. a Pod created before injection was healthy).

Enforcement level is controlled by global.runtimePolicies.failureAction (Enforce by default — violations block admission). Civitas's own components are tested against these policies as part of the standard deployment, so no rollout step is needed for this deployment itself — Enforce is expected to be safe out of the box for civitas/core only clusters.

global:
  runtimePolicies:
    enabled: true
    failureAction: Enforce   # or Audit

Inspect what is being flagged, when Audit is activated:

kubectl get clusterpolicyreport,policyreport -A

When Enforce is activated, you have to look in the events for warnings.

warning

If the Linkerd proxy injector is down, require-linkerd-sidecar under Enforce will block admission of any new or updated Pod in a meshed namespace (existing Pods keep running). If injection reliability is a concern, set global.runtimePolicies.failureAction: Audit until it is restored.

Impact on shared clusters

These ClusterPolicy objects are cluster-scoped and match cluster-wide — none of them are restricted to Civitas's own namespaces, so on a cluster shared with other tenants/workloads Enforce mode can affect resources this platform does not own:

• require-linkerd-sidecar, require-meshed-namespace-inbound-policy, and justify-linkerd-inject-opt-out trigger on any Namespace/Pod in the cluster that carries the relevant linkerd.io/inject (or config.linkerd.io/default-inbound-policy) annotation — including namespaces belonging to other teams that independently use Linkerd, not just Civitas instance namespaces.
• restrict-strimzi-operator-pod-labels and restrict-cnpg-operator-pod-labels match on the ServiceAccount's trailing name only (strimzi-cluster-operator / postgres-operator-cloudnative-pg), regardless of namespace. A separate, unrelated Strimzi or CloudNativePG operator installation elsewhere in the same cluster would also be restricted to Civitas's app.kubernetes.io/name allowlist, and could have legitimate Pods denied.
• protect-operator-owned-labels denies any Pod in the cluster — not just ones created by a Civitas operator — that carries app.kubernetes.io/name: kafka, app.kubernetes.io/name: postgresql, or a cnpg.io/jobRole label, unless it was created by Civitas's own Strimzi/CloudNativePG operator ServiceAccount. Third-party charts commonly used elsewhere on a shared cluster (e.g. Bitnami's Kafka/PostgreSQL charts) set exactly these labels and are created by kube-controller-manager, not the Civitas operator — so under Enforce those unrelated Pods would be denied at admission. This is the most likely of the runtime policies to break an unrelated workload on a shared cluster.

If this cluster hosts anything other than this Civitas deployment, do not leave the default Enforce in place blind. Instead:

1. Before deploying (or before switching an existing cluster to a newer version of this platform), set global.runtimePolicies.failureAction: Audit.
2. Deploy/upgrade, then let the cluster run under real traffic for a while and inspect kubectl get clusterpolicyreport,policyreport -A for violations attributed to non-Civitas workloads.
3. Only once those reports are clean, switch back to global.runtimePolicies.failureAction: Enforce (the default) to get the full protection.

Civitas's own components don't need this rollout — they are tested against these policies under Enforce already. This step is only to protect other workloads on a shared cluster from being unexpectedly blocked.

info

The runtime ClusterPolicy objects are cluster-scoped, so they are deployed once per cluster as part of the shared operator layer (helmfile -f deployment/helmfile-operators.yaml sync) — not per instance. On multi-instance clusters this is handled automatically via deployLayer gating; see Deployment → Deploying multiple instances.

RBAC and Least Privilege

• ServiceAccounts: Each component creates its own ServiceAccount with automountServiceAccountToken: false by default
• Etcd RBAC: Etcd uses role-based access control with per-component users (e.g., apisix user with read-write access only to /apisix/* prefix)
• CloudNativePG: The operator creates dedicated database roles per component with minimal privileges (no superuser)
• Cluster Admin: Required only for initial deployment (CRD installation); day-to-day operations use namespace-scoped permissions

Secret Management

• Secrets are auto-generated by the secrets component using random passwords (configurable length)
• Secrets are stored as Kubernetes Secrets (base64-encoded in etcd)
• No plaintext secrets in configuration files or Helm values
• Database passwords are generated per component and injected via secret references
• Secret definitions are in components/<component>/secrets.yaml

Network Policies

Network policies are defined per component to restrict pod-to-pod communication. Each policy follows a default-deny model with explicit allow rules.

Component	Allowed Ingress From
PostgreSQL	CloudNativePG operator, Portal (backend), Keycloak, Authz, FROST, database role setup jobs
Etcd	APISIX, Etcd peers
Kafka	Portal, Kafka UI, config-adapters, NiFi
Keycloak	APISIX, config-adapter, keycloak-config-cli

info

Network policies are enabled by default in each component's default-environment.yaml.gotmpl but require a CNI plugin with NetworkPolicy support (e.g., Calico, Cilium). Most managed Kubernetes services support this.

TLS and Encryption

Layer	Implementation
External traffic	TLS termination at Ingress (cert-manager with ClusterIssuer)
Service-to-service	Mandatory mTLS via Linkerd (default cluster-authenticated inbound policy) — see below
Etcd communication	Internal TLS between peers
Database connections	PostgreSQL hostssl entries in pg_hba.conf

Service Mesh Authorization (mTLS Enforcement)

When the Linkerd service mesh is enabled (the default), service-to-service traffic is not just encrypted opportunistically — it is authorized. The prepare step annotates every instance namespace with a mandatory default inbound policy, and public edges are carved out explicitly with Linkerd Server resources.

Setting	Effect	Default
global.serviceMesh.defaultInboundPolicy	Namespace-wide config.linkerd.io/default-inbound-policy. cluster-authenticated = only authenticated, meshed in-cluster clients may connect. Other values: all-authenticated, all-unauthenticated, audit (log-only rollout).	cluster-authenticated
global.serviceMesh.allowUnauthenticatedIngress	Drives the APISIX data-plane Server accessPolicy. false → all-authenticated (the ingress controller must be meshed). true → all-unauthenticated (accepts plaintext from an unmeshed ingress).	false

Consequences and safe rollout:

• The ingress controller must be meshed when allowUnauthenticatedIngress: false. Once per cluster before deploying, annotate the Ingress-namespace for Linkerd injection — set linkerd.io/inject=enabled and config.linkerd.io/skip-inbound-ports=80,443,8443 on it, then restart the controller. See Prerequisites → Linkerd.
• Kubelet liveness/readiness probes are auto-authorized by Linkerd, so mandatory mTLS does not break health checks.
• Only the public APISIX data-plane port is exposed as a Server; all other APISIX ports (admin, control) stay under the mandatory namespace default.
• For a phased rollout, set defaultInboundPolicy: audit first to log what would be denied, then switch to cluster-authenticated once the reports are clean.

Image Security

• Container image tags are pinned to explicit versions (no latest tags); a few images are pinned via the upstream Helm chart's own values rather than in images.yaml
• Multi-architecture builds (amd64/arm64) via CI/CD
• Trivy vulnerability scanning in CI pipeline (container-qa stage)
• SBOM generation via Syft (CycloneDX format) in CI pipeline
• Images are built from known base images with minimal attack surface

CI/CD Security

• Pre-commit hooks enforce code quality (YAML lint, Helm lint, formatting)
• Kyverno policy validation in CI pipeline
• Container image scanning with Trivy (GitLab report format)
• Manual approval gates for staging and production deployments

Possible Extensions

The following measures are not part of the default deployment but can be implemented by operators to further harden their environment.

Runtime Policy Enforcement

In-cluster runtime enforcement is now built in via the runtime-policies component, and enforces (Enforce) by default — see Runtime Kyverno Policies. To harden further, you can additionally deploy the shift-left CI policy sets (.ci/policies/base/, .ci/policies/production/) to the cluster for continuous, in-cluster reporting of the manifest-level rules:

kubectl apply -f .ci/policies/base/
kubectl apply -f .ci/policies/production/  # for production clusters

warning

These policy sets are Audit-only — not every Civitas component currently satisfies every rule, so Enforce is not supported for them yet and would block legitimate deployments. Use them for reporting only.

External Secret Management

Replace Kubernetes Secrets with an external secret manager for enhanced security:

• External Secrets Operator with HashiCorp Vault, AWS Secrets Manager, or Azure Key Vault
• Encrypt secrets at rest using Kubernetes EncryptionConfiguration

Pod Security Standards

Apply Kubernetes Pod Security Standards at the namespace level:

kubectl label namespace <instanceSlug> \
  pod-security.kubernetes.io/enforce=restricted \
  pod-security.kubernetes.io/warn=restricted

Image Signing and Verification

• Sign container images with Cosign in the CI pipeline
• Verify signatures at admission time using Kyverno's image verification policies

Network Policy Hardening

• Enable network policies for all environments (not just production)
• Add egress policies to restrict outbound traffic from pods
• Fine-grained service mesh authorization: the namespace-wide mandatory mTLS policy is already enforced by default (see Service Mesh Authorization). To go further, add per-workload Linkerd Server / AuthorizationPolicy resources that restrict which meshed clients may reach a given service, beyond the namespace default

Audit Logging

• Enable Kubernetes audit logging to track API server requests
• Forward audit logs to your centralized logging stack (see Monitoring & Logging)

Regular Vulnerability Scanning

• Run Trivy scans on deployed images on a schedule (not just in CI)
• Use Trivy Operator for continuous in-cluster vulnerability scanning