Security Hardening

This document describes the security controls implemented in Civitas Core V2 and possible extensions for operators who want to further harden their deployment.

Already Implemented

The following security measures are built into the default deployment configuration.

Pod Security

All components run with restrictive security contexts defined in defaults/environment/security.yaml.gotmpl:

Control	Setting	Scope
Non-root containers	runAsNonRoot: true, runAsUser: 1000	All pods
Read-only root filesystem	readOnlyRootFilesystem: true	All containers
No privilege escalation	allowPrivilegeEscalation: false	All containers
Dropped capabilities	capabilities.drop: [ALL]	All containers
Seccomp profile	seccompProfile.type: RuntimeDefault	All pods
File system group	fsGroup: 1000	All pods

These defaults are enforced by component value templates and validated by Kyverno policies in CI/CD.

Kyverno Security Policies

info

Our Kyverno Security Policies are currently in a early state and will be expanded over time. We welcome contributions to enhance our security posture.

The deployment repository includes Kyverno policies (in .ci/policies/) that validate manifests at pipeline runtime in the repository. These policies can also be deployed to the cluster for runtime enforcement.

Base policies (all environments):

Policy	Enforces
drop-all-capabilities	All capabilities must be explicitly dropped
read-only-root-filesystem	Root filesystem must be read-only
probes	Startup, liveness, and readiness probes must be configured
require-ingress-class	Ingress resources must specify an ingressClassName

Production policies (production profile):

Policy	Enforces
require-pdb	PodDisruptionBudget with minAvailable >= 1
require-rolling-update	RollingUpdate with maxUnavailable: 0, maxSurge: 25%
require-replicas-or-hpa	Either multiple replicas or HPA must be configured
require-resources	CPU and memory requests and limits must be set

Validate policies locally:

# All components
make verify-policies

# Single component
.ci/policies/verify-kyverno-policies.sh <component-name>

RBAC and Least Privilege

• ServiceAccounts: Each component creates its own ServiceAccount with automountServiceAccountToken: false by default
• Etcd RBAC: Etcd uses role-based access control with per-component users (e.g., apisix user with read-write access only to /apisix/* prefix)
• CloudNativePG: The operator creates dedicated database roles per component with minimal privileges (no superuser)
• Cluster Admin: Required only for initial deployment (CRD installation); day-to-day operations use namespace-scoped permissions

Secret Management

• Secrets are auto-generated by the secrets component using random passwords (configurable length)
• Secrets are stored as Kubernetes Secrets (base64-encoded in etcd)
• No plaintext secrets in configuration files or Helm values
• Database passwords are generated per component and injected via secret references
• Secret definitions are in components/<component>/secrets.yaml

Network Policies

Network policies are defined per component to restrict pod-to-pod communication. Each policy follows a default-deny model with explicit allow rules.

Component	Allowed Ingress From
PostgreSQL	CloudNativePG operator, Portal (backend), Keycloak, Authz, FROST, database role setup jobs
Etcd	APISIX, Etcd peers
Kafka	Apicurio, Portal, Kafka UI
Keycloak	APISIX, config-adapter, keycloak-config-cli
Apicurio	Model Atlas

info

Network policies are enabled by default in each component's default-environment.yaml.gotmpl but require a CNI plugin with NetworkPolicy support (e.g., Calico, Cilium). Most managed Kubernetes services support this.

TLS and Encryption

Layer	Implementation
External traffic	TLS termination at Ingress (cert-manager with ClusterIssuer)
Service-to-service	mTLS via Linkerd service mesh (optional but recommended)
Etcd communication	Internal TLS between peers
Database connections	PostgreSQL hostssl entries in pg_hba.conf

Image Security

• All container image tags are pinned (no latest tags)
• Multi-architecture builds (amd64/arm64) via CI/CD
• Trivy vulnerability scanning in CI pipeline (container-qa stage)
• SBOM generation via Syft (CycloneDX format) in CI pipeline
• Images are built from known base images with minimal attack surface

CI/CD Security

• Pre-commit hooks enforce code quality (YAML lint, Helm lint, formatting)
• Kyverno policy validation in CI pipeline
• Container image scanning with Trivy (GitLab report format)
• Manual approval gates for staging and production deployments

Possible Extensions

The following measures are not part of the default deployment but can be implemented by operators to further harden their environment.

Runtime Policy Enforcement

Deploy Kyverno policies to the cluster for runtime enforcement (beyond CI/CD validation):

# Install Kyverno
helm repo add kyverno https://kyverno.github.io/kyverno/
helm install kyverno kyverno/kyverno --namespace kyverno --create-namespace

# Apply Civitas policies (change from audit to enforce mode as needed)
kubectl apply -f .ci/policies/base/
kubectl apply -f .ci/policies/production/  # for production clusters

warning

Test policies in audit mode before switching to enforce to avoid blocking legitimate workloads.

External Secret Management

Replace Kubernetes Secrets with an external secret manager for enhanced security:

• External Secrets Operator with HashiCorp Vault, AWS Secrets Manager, or Azure Key Vault
• Encrypt secrets at rest using Kubernetes EncryptionConfiguration

Pod Security Standards

Apply Kubernetes Pod Security Standards at the namespace level:

kubectl label namespace <instanceSlug> \
  pod-security.kubernetes.io/enforce=restricted \
  pod-security.kubernetes.io/warn=restricted

Image Signing and Verification

• Sign container images with Cosign in the CI pipeline
• Verify signatures at admission time using Kyverno's image verification policies

Network Policy Hardening

• Enable network policies for all environments (not just production)
• Add egress policies to restrict outbound traffic from pods
• Consider a service mesh policy layer (e.g., Linkerd authorization policies) for L7 traffic control

Audit Logging

• Enable Kubernetes audit logging to track API server requests
• Forward audit logs to your centralized logging stack (see Monitoring & Logging)

Regular Vulnerability Scanning

• Run Trivy scans on deployed images on a schedule (not just in CI)
• Use Trivy Operator for continuous in-cluster vulnerability scanning