portal-backend Integration & Division of Labor

Two boundaries keep portal-backend thin, and this page describes both. The first is about content: who is allowed to understand the model payload. The second is about persistence: where data physically lives. They are independent concerns, so they are stated separately. The content boundary comes first because it is the one most easily eroded — model interpretation tends to leak upward into the host one convenient map.get(...) at a time, and each leak recreates in the host a piece of knowledge that already exists (and is tested) in the frontend, in Model Forge or in config-adapter.

Prime directive: the host sees only the envelope

Frontend, the model-forge library and config-adapter understand the model payload by its content. portal-backend knows only the envelope: name, version, URN, dependencies.

This is a hard layering rule, not a preference. Any model interpretation that ends up in portal-backend is by definition misplaced — it belongs in model-forge (if it is a runtime derivation) or in the frontend (if it can be computed when the document is authored).

Party	Understands the payload…	Role
Frontend	by content	authors the clean, schema-valid CORE document — nodes/edges, Elements, mapping fields, styles
model-forge library	by content	single source of truth for the schemas; validation, URN authority, dependency graph
config-adapter	by content	interprets the graph document into an engine flow (GraphParser, NodeKind, DataStructureSchema, NiFi MappingNodeType)
portal-backend	envelope only	custodian & orchestrator: stores URN pins + relations, forwards the opaque payload, resolves references at the URN level

The model vocabulary is deliberately not one shared Java type. Each content-aware party re-derives it from the canonical JSON Schemas in model-forge-runtime (pipeline.schema.json, datastructure.schema.json, mapping.schema.json, …): Model Forge validates against them, config-adapter has its own hand-written readers, the frontend consumes generated Zod types. portal-backend derives nothing — it is the only party that never parses node, field or transform content.

What the envelope is, in code

• Entities are thin shells. Pipeline, DataStructureVersion, DataSink and DataSource hold only name, version, the URN pins (modelLogicalUrn/modelUrn, configurationUrn) and FK relations — no node, element, mapping or field columns. This is the JPA face of the same cut described under the persistence cut.
• The payload travels as an opaque Map<String,Object> (PipelineInputDTO.model/styles), is stored verbatim in the registry, and is never deserialized into model types. As PipelineService puts it, the backend "stores the model verbatim … and never parses the pipeline's contents."
• The only door to Model Forge types is the anti-corruption layer ModelRegistryGateway — "the only portal-backend package that depends on de.civitascore.modelforge.*; Model Forge types never leak past it."
• URN structure is touched only inside that ACL, and only through the authorities UrnParser (Model Forge) / CoreUrn (config-adapter-api). Everywhere else in the host a URN is an opaque string.

Allowed and forbidden in portal-backend

Allowed

• Hold, compare and forward URNs as opaque strings; match version drift with CoreUrn.sameArtifact / logicalUrn.
• Store and fetch the payload through ModelRegistryGateway (PayloadKind = opaque JSON).
• Query the dependency graph via modelForge.dependents(…) — never reconstruct it from the payload, never mirror it into host tables ahead of a dedicated work package.
• Wire FK relations and resolve dataSourceIds/dataSinkIds (UUIDs / configuration URNs) — these are envelope references, not content.

Forbidden

• Parsing or interpreting the payload: no reaching into nodes, edges, fields, sourceRef/sinkRef/mappingRef, $defs, x-core-*, transform kinds or element semantics.
• Any new de.civitascore.modelforge.* import outside ModelRegistryGateway.
• Importing config-adapter content types (GraphParser, NodeKind, PipelineGraph, DataStructureSchema, MappingNodeType, …).
• Re-implementing URN layout (splitting segments, recomputing the disambiguator, duplicating the pattern).
• Model validation — that is exclusively Model Forge's job against the schemas.

When you reach for model content

If host code seems to need to look inside the payload, the logic is in the wrong place. Move it:

1. Prefer the frontend. If the result is fixed at authoring time, the frontend should pre-compute it and ship it inside the envelope/payload it already builds.
2. Otherwise the model-forge contract. If it is a genuine runtime derivation, expose a method on the ModelForge facade that returns the result in envelope terms (URNs, ArtifactId, dependency edges, or a finished result) and call it through the ACL.

Rule of thumb: the host may consume the result of an interpretation (in envelope form); it may never perform the interpretation itself.

Resolving a pipeline's Mappings without reading its content

DataSetSagaPublisher.buildMappings() must ship every Mapping a pipeline uses into the saga trigger, because config-adapter is callback-free — a mapping's transform rules have to be self-contained in the payload it receives. It does this without reading the pipeline document: it asks Model Forge's dependency graph which Mappings the pipeline depends on (ModelRegistryGateway.dependencyUrnsOfType(pipelineUrn, "mapping")), then fetches each Mapping's content by URN. Dependencies are part of the envelope, so this stays on the correct side of the cut.

This is the shape of correct host orchestration: its inputs and outputs are URNs and pins, never model bodies. An earlier version walked the pipeline's nodes[] and read each mappingRef directly — the single place content semantics had leaked into the host. That leak has been closed; do not reintroduce it. If you need to know something else about a pipeline's content, ask the dependency graph or add a model-forge contract method — never parse the document in the host.

Enforcing the boundary

The ArchUnit rule shown under Hard rules the cut relies on below already confines every de.civitascore.modelforge.. import to the ..modelregistry.. gateway; the content boundary reuses that guard. For a host diff, the review checklist is:

• no new modelforge.* import outside ModelRegistryGateway;
• no .get("nodes") / .get("edges") / .get("fields") / mappingRef / sourceRef / sinkRef in new host code;
• no new manual URN splitting outside the ACL;
• new entity/DTO fields carry only name, version, URN or relations;
• any interpretive logic written instead lives in the frontend or in model-forge.

The persistence cut: host DB vs. registry

Model Forge is consumed as an embedded library, not a standalone service (see Architecture). The host is portal-backend from civitas-core-platform: it persists its own management entities (DataStructure, DataStructureVersion, Pipeline, ...) via JPA/Hibernate, while the model content itself lives in Model Forge's registry. Both systems persist things — just not the same things. This page describes where that line runs.

Guiding principle

The host owns everything about management, lifecycle, relationships and authorization. Model Forge owns the model content itself: schema documents, their versions, formats and reference graph.

Both sides keep their existing persistence technology — nothing is migrated off JPA or off Model Forge's JDBC registry. What moves is data ownership, not storage engine.

Why this isn't a real overlap

At first glance both systems seem to store "DataStructures, Pipelines, DataSets" — in practice they store different things:

Host concept (JPA)	Model Forge concept (registry)	Relationship
DataStructure entity	datastructure-typed artifact (a grouping of Elements)	Management object vs. model manifest. The host owns name, status, permissions; Model Forge owns the content.
DataStructureVersion.model (jsonb)	element artifacts + their grouping, content in artifact_representation	The actual overlap. Today a host column, becomes a registry artifact.
DataStructureVersion.version (host-chosen string)	artifact_version.version (backend SemVer)	Target: one version concept, owned by Model Forge; the host mirrors it (see ADR 057).
DataSet entity (aggregate: pipelines, sinks, ...)	dataset artifact (composition manifest with *Refs)	Same name, disjoint concepts today.
Pipeline.model / DataSource / DataSink (host jsonb columns)	pipeline / datasource / datasink artifacts	Disjoint today; a later, separate decision could converge them.

There is no table- or foreign-key collision — the collision is mostly terminology, plus the one genuine double role of DataStructureVersion.model. That column is exactly where the cut runs.

Ownership by concern

Concern	Owner	Lives in
Name/description shown in the UI, platform status (DRAFT/AVAILABLE)	Host	public.data_structure*
Authorization (assignments), FK integrity to data_sources	Host	public.*
styles/modelName; the version number is a mirror, not an input	Host	public.data_structure_versions
JSON Schema / XSD content	Model Forge	model_forge.artifact_representation
XSD → JSON Schema conversion (eager at first import; the original stays retrievable)	Model Forge, triggered by the integration (ADR 061)	model_forge.artifact_representation
Version assignment (SemVer, bump kind), format variants, content hash	Model Forge	model_forge.artifact_version, artifact_representation
Reference graph between models ($ref, imports)	Model Forge	model_forge.artifact_reference
Derived views (inlined/bundled), search	Model Forge	computed on demand

Two decisions worth calling out because they move authority that used to sit with the host or its users:

• Versioning. Model Forge is the sole version authority (ADR 057). The host no longer lets a user type a version string; it sends a change class (patch/minor/major) and mirrors back whatever version Model Forge assigned.
• Status/workflow. Model Forge carries no lifecycle state at all — no draft/review/approved stages, no per-artifact status column. The host's own status (DRAFT/AVAILABLE, release rules, in-use locks) is the only status in the system. There is exactly one place a caller can go to ask "is this thing usable" — the host — never two places that could disagree.

Topology: one database, two schemas

                    host application (one Spring Boot app, one JVM)
  ┌──────────────────────────────────────────────────────────────────────┐
  │  Controllers / Security / Authorization        (host responsibility) │
  │        │                                                             │
  │        │ JPA (Hibernate)              Model Forge facade             │
  │        ▼                                    │                        │
  │  host entities                        model-forge-runtime            │
  │        │                                    │                        │
  │        │                                    ▼                        │
  │        │                            (registry adapter)               │
  │        │                                    │ JdbcClient             │
  │        └────────────────┬───────────────────┘                        │
  │                         ▼                                            │
  │      one DataSource, one PlatformTransactionManager                  │
  └─────────────────────────┼────────────────────────────────────────────┘
                            ▼
                       PostgreSQL
     ┌─────────────────────────┬────────────────────────────────┐
     │ Schema `public`         │ Schema `model_forge`           │
     │ host management tables  │ artifact                       │
     │   └─ a URN column ───────────► (versioned URN, no FK)    │
     │                         │ artifact_version               │
     │                         │ artifact_representation        │
     │                         │ artifact_reference, ...        │
     │ flyway_schema_history   │ model_forge_schema_history     │
     └─────────────────────────┴────────────────────────────────┘

One database, two schemas, one DataSource, two independent Flyway histories, one PlatformTransactionManager. Hibernate only ever sees its own mapped entities and never touches model_forge — there is no ORM validation conflict.

Hard rules the cut relies on

1. No foreign key crosses the schema boundary. The host references artifacts only by a versioned URN (a plain text column). Consistency comes from the shared transaction (below), not a database constraint.
2. The host never touches model_forge.* tables directly — no JPA mapping, no native SQL, no view over them. The only door in is the ModelForge facade.
3. Model Forge knows no host tables and no host entities. It only ever receives URNs, JSON content and commands from the facade.
4. Each side migrates only its own schema. The host's own Flyway history and Model Forge's modelForgeFlyway (see Architecture) stay strictly separate.

A host is expected to enforce rule 2 structurally, for example with an ArchUnit rule that confines every import of Model Forge classes to one package (an anti-corruption layer / gateway):

@ArchTest
static final ArchRule model_forge_only_via_gateway =
    noClasses()
        .that().resideOutsideOfPackage("..modelregistry..")
        .should().dependOnClassesThat()
        .resideInAPackage("de.civitascore.modelforge..");

That gateway is also where the vocabulary gap gets absorbed: a host may still think in its own terms (e.g. "DataStructure" meaning a single type), while Model Forge uses the vocabulary from the Glossary (Element vs. DataStructure, ADR 060). The gateway is the one place that translates between the two, so the difference never leaks into the rest of the host's codebase.

One transaction, not two systems

Sharing a DataSource is more than an operational convenience — it is what makes the cut safe without distributed-transaction machinery:

• Model Forge writes through JdbcClient, which obtains its connection via DataSourceUtils — it automatically joins whatever transaction is already open on that DataSource. A JpaTransactionManager supports mixed JPA-and-JDBC access to the same DataSource within one transaction.
• A host write that saves management metadata and calls into Model Forge is therefore one commit. No two-phase commit, no outbox, no compensating actions. If the facade throws, the whole operation rolls back — there is no way to end up with a version but no artifact, or an artifact but no version.
• This relies on the single-writer assumption from Architecture: one write-capable host instance. Model Forge's in-memory caches are rebuilt from PostgreSQL, so they survive a restart; multiple concurrent writers would need that assumption revisited first.

What does not move

• Persistence technology stays put on both sides. The host does not adopt JDBC for its management data, and Model Forge does not adopt JPA for the registry — see ADR 049 for why explicit SQL is the right tool for dynamic, versioned artifact payloads.
• Two separate databases was considered and rejected: it would break the one-transaction guarantee above and reintroduce eventual consistency between two systems that otherwise share nothing hard to keep in sync.
• Authorization stays entirely with the host. Model Forge registers no security of its own (see Architecture); whoever gets past the host's controller is trusted to invoke the facade operation behind it.

Net effect

There is no second system competing for the same data. The host remains the place for management, relationships and permissions; Model Forge becomes the place for model content, its versions and its reference graph. The two meet in one database, one transaction and a single text column — not in shared tables, not in shared entities, and not through a second service to keep available and in sync.