Test Levels & Matrix

This page describes the test levels and test types of CIVITAS/CORE and the mandatory tests for each component. It is part of the Testing Concept.

Test Levels

Each level has its own purpose, owner, and pass or fail criteria.

Test level	Purpose	Typical tools and artifacts	Primary environment
Unit test	Business logic and small, isolated units	JUnit, Mockito, Vitest	Local, CI
API test	Endpoints, status codes, and flows	Bruno, HTTP clients, tests based on OpenAPI	Local, CI
Integration test	Interaction of several components	Testcontainers, Docker Compose, real services	Local, CI
E2E (developer)	Happy path with reusable preconditions	Playwright, Bruno, test library	Local, CI, Develop
E2E (community) / UAT	Real functional processes, including side effects	Community use cases, manual tests	Staging
System test	The platform as a whole	CI with VCluster, Robot Framework	CI, VCluster
Security test	Vulnerabilities and misconfigurations	Automated security checks, penetration test	Security test environment
Usability test	Clarity and ease of use	UX reviews, manual scenarios	Staging, demo environment
Documentation review	Clarity and completeness of the documentation	Review checklists, functional review	Internal, community

Unit Tests

Unit tests are the first line of defense. They are fast, need no external infrastructure, and test one functional or technical decision precisely. They are especially important for backend business logic, adapter logic, state machines and command builders, frontend component logic, and validation and transformation rules.

API Tests

API tests check endpoints, payloads, response codes, and repeatable flows from a functional perspective. They do not replace integration tests that involve several components. The Bruno collections for the Portal Backend are in api/portal-backend in the civitas-core-platform repository.

Integration Tests

Integration tests are most valuable where several systems interact:

• Portal Backend and Config Adapter.
• Frontend and authentication.
• Backend and database.
• Adapters and external target systems.
• Orchestration and event handling.

They expose typical interface risks: incorrect payloads, unclear state transitions, incompatible versions, missing infrastructure assumptions, and incorrect assumptions about authentication or authorization.

Contract Tests

Contract tests protect critical interfaces. They check the compatibility between producing and consuming systems before defects reach E2E or system tests. They verify the minimum expectations for payloads, status codes, fields, and error behavior. They are relevant for these interfaces:

• Portal Backend and Config Adapter.
• Frontend and Backend.
• Backend and event or saga processing.
• Adapters and external target systems.
• Authentication and authorization.

System Tests

System tests check the platform as a complete system. They show whether the parts work together, even when each part works on its own. CI with VCluster is the mandatory environment. It is isolated, reproducible, close to the real platform topology, and independent of lengthy manual setup.

System tests follow a Hero Case. It covers the CIVITAS/CORE feature set on the happy path. E2E tests cover side effects.

End-to-End Tests

End-to-end tests have two parts because they pursue different goals. Both use a small set of core use cases: ideally one to three central ones that cover as many aspects of the platform as possible. New cases are added iteratively. Stable tests matter more than many tests.

Developer E2E: happy path with preconditions. The developer E2E suite covers the happy path. Developers can test a feature without setting up the full chain of preconditions by hand. The preconditions are provided automatically. When a feature is complete and stable, it becomes a precondition for the following tests. See Test Library.

Community E2E (Quality Assurance Working Group). The community E2E suite covers complete functional processes, including side effects. It is close to real use and therefore valuable for functional acceptance. The happy path alone is not enough: follow-on effects and interactions must be visible. The use cases come from the community, which defines the functionally relevant processes. The manual process is described in User Acceptance Tests.

Smoke Tests

Smoke tests are short, fast stability checks. They show whether a build or a deployment is basically usable. They check that a new environment starts, that core services and connections work at a basic level, and they reveal obvious configuration errors early. They do not replace functional tests, but they must pass before deeper tests start.

Security Tests

Security tests check the platform for vulnerabilities, misconfigurations, and unintended access:

• Automated security checks.
• Access control checks.
• Configuration checks.
• An external penetration test by Port Zero.

They run where real risks arise. They therefore cover the interaction of authentication, permissions, data access, and operational settings, not only individual components. For secure development practice, see the Secure Development Guide.

Documentation Review

Documentation is part of the test strategy because it directly affects usability. A review checks functional correctness, sufficient completeness for the target audience, consistency with the implementation and the environment, and usability for the community and internal users.

Flaky Tests

An unstable test is a quality defect. The target state handles flaky tests separately:

• Identify unstable tests explicitly.
• Analyze the root cause promptly.
• Quarantine or deactivate a test only as a temporary measure.
• Return the test to the regular run after the cause is fixed.
• Check regularly for recurring instability.

This keeps the test pipeline meaningful.

Test Matrix by Component

Each core component has an explicit test matrix. It is the reference for planning, implementation, and review, and it prevents implicit gaps in test coverage.

Component	Mandatory test types	Quality goal
portal-backend	Unit, API, integration, and contract tests; system-level smoke checks; relevant E2E flow steps	Correct business logic, stable API, robust persistence, correct integration with Config Adapter and authentication
portal-frontend	Unit and component tests; authentication and backend contracts; Playwright E2E; selected community flows	Stable user guidance, consistent interaction, correct authentication, reliable E2E flows
authz	Unit and integration tests; Playwright E2E; permission contracts	Correct access decisions, stable role logic, reproducible 401/403 and permitted-access cases
config-adapter	Unit tests; adapter integration tests; contract tests; saga state machine tests; orchestrated end-to-end scenarios	Deterministic event processing, idempotent adapter logic, correct orchestration, reliable external integrations
model-forge	Unit tests; adapter integration tests; contract tests	Correct business logic, stable API, robust persistence, correct model handling
Deployment and infrastructure	Deployment validation, smoke tests, system tests, connectivity checks	Stable deployment, reachable core components, correct connectivity, reliable operation

The mandatory environments for each component follow from Test Environments.