Version: V2-Next

CORE-IR Syntax and Semantics

This document describes the CORE Intermediate Representation (CORE-IR), i.e. the JSON-based exchange format between Model Forge, UI, registry and downstream generators.

The normative technical basis are the JSON Schemas in the backend (browsable in the CORE-IR Schema Documentation linked in the API section):

dataset.schema.json
datastructure.schema.json
artifact-envelope.schema.json
mapping.schema.json
pipeline.schema.json
datasource.schema.json
datasink.schema.json
core-schema-extensions.schema.json

the JSON Schema Reference lists them all with downloads. The individual schemas remain the maintained source of truth; the large schema is composed from them.

Basic model

CORE-IR consists of reusable artifacts:

Artifact	Content	Identity
`DataSet`	Composition manifest for artifacts that belong together	`id`
`Element`	JSON Schema or XSD-based structure definition	`$id`
`DataStructure`	Named, versioned grouping of Elements	`id`
`Mapping`	Field mapping between two Elements	`id`
`Pipeline`	Graph of sources, transformations and sinks	`id`
`DataSource`	Inbound connection and payload structure	`id`
`DataSink`	Outbound connection and payload structure	`id`
`ArtifactEnvelope`	Transport/import envelope for artifact metadata and content	`artifactId` + `firstVersion.version`

Elements are JSON Schema documents and therefore use the JSON Schema identity $id. The CORE domain artifacts use id.

All artifact IDs are versioned CORE URNs:

urn:core:{scope}:{owner}:{artifactType}:{domain}:{name}:{disambiguator}:{version}

Examples:

urn:core:platform:civitas:dataset:common:air-quality:zgpsfmmplg:1.0.0
urn:core:platform:civitas:element:common:SensorReading:d28s38wfmi:1.0.0
urn:core:platform:civitas:mapping:common:sensor-to-observation:0qyaberslc:1.0.0

`$schema`, `id` and `$id`

$schema describes the schema/artifact syntax of the document. It is not the domain identity of the artifact.

{
  "$schema": "https://civitasconnect.digital/core/mapping/v1",
  "id": "urn:core:platform:civitas:mapping:common:sensor-to-observation:0qyaberslc:1.0.0"
}

Rules:

CORE domain artifacts use id.
Elements use $id.
Every exported artifact contains $schema.
References point to the domain identity, i.e. to id or $id, not to $schema.
Short forms such as "SensorReading" are not valid references.

Artifact envelope

The artifact envelope is the uniform transport form for artifacts when metadata is needed in addition to the actual content. It separates the logical identity (artifactId), the artifact type, and the initial content (firstVersion.content) when creating an artifact.

CORE uses no separate top-level $id in the envelope. The unique versioned artifact URN is derived from artifactId and firstVersion.version, so the envelope does not duplicate identity.

{
  "$schema": "https://civitasconnect.digital/core/artifact-envelope/v1",
  "artifactId": "urn:core:platform:civitas:element:common:AirQualityEnvelope:uxqttdplsw",
  "artifactType": "XSD",
  "groupId": "elements",
  "title": "AirQualityEnvelope",
  "firstVersion": {
    "version": "1.0.0",
    "content": {
      "contentType": "application/xml",
      "content": "<xs:schema xmlns:xs=\"http://www.w3.org/2001/XMLSchema\">...</xs:schema>"
    }
  }
}

Field	Meaning
`$schema`	Schema of the envelope: `https://civitasconnect.digital/core/artifact-envelope/v1`
`artifactId`	The logical, version-free CORE URN that identifies the artifact. Elements always use the `element` type — the format is not in the URN, so a `:xsd:` URN is rejected.
`artifactType`	Content format of the embedded `content` (`JSON` \| `JSON_SCHEMA` \| `XSD` \| `MAPPING` \| `PIPELINE` \| `DATASOURCE` \| `DATASINK` \| `DATASET`) — an import hint that selects the stored representation, not the URN's artifact-type segment.
`groupId`	Global artifact group, e.g. `elements`
`firstVersion.version`	Version of the initial artifact content
`firstVersion.content.contentType`	Media type of the content
`firstVersion.content.content`	The actual artifact content. JSON Schema as an object, XSD as an XML string.

The versioned CORE URN of the artifact is {artifactId}:{firstVersion.version}.

For JSON Schema, the content itself may also carry $id. In that case this $id must semantically match the versioned envelope URN derived above. For XSD there is no JSON $id in the content; there, artifactId + firstVersion.version is the authoritative CORE identity.

DataSet

A DataSet is the bracketing document for a coherent integration. It is stored and exchanged as a manifest that references its members by URN:

{
  "$schema": "https://civitasconnect.digital/core-dataset/v1",
  "id": "urn:core:platform:civitas:dataset:common:air-quality:zgpsfmmplg:1.0.0",
  "title": "Air Quality",
  "datastructureRefs": [
    "urn:core:platform:civitas:datastructure:common:air-quality:d28s38wfmi:1.0.0"
  ],
  "mappingRefs": [
    "urn:core:platform:civitas:mapping:common:sensor-to-observation:0qyaberslc:1.0.0"
  ],
  "pipelineRefs": [
    "urn:core:platform:civitas:pipeline:common:air-quality-ingest:qubihav71f:1.0.0"
  ],
  "dataSourceRefs": [
    "urn:core:platform:civitas:datasource:common:aq-mqtt-source:yv3oxe088a:1.0.0"
  ],
  "dataSinkRefs": [
    "urn:core:platform:civitas:datasink:common:aq-http-sink:s7mqw72qmh:1.0.0"
  ]
}

DataSet validation is structural: the manifest is validated against the bundled CORE DataSet JSON Schema. Cross-member reference checks — e.g. that every *Refs entry resolves to an existing artifact — are not part of this validation.

Element

An Element is a JSON Schema or a schema derived from XSD. The domain identity is in $id.

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "urn:core:platform:civitas:element:common:SensorReading:d28s38wfmi:1.0.0",
  "title": "SensorReading",
  "type": "object",
  "properties": {
    "sensorId": { "type": "string" },
    "temperature": { "type": "number" }
  },
  "required": ["sensorId"]
}

If an Element version is authored as XSD, its JSON Schema projection points back at the Element's own URN via x-xsd-source — the version that carries the source XSD representation.

DataStructure

A DataStructure is a named, versioned grouping of Elements — a stored artifact (not a View) that records which Elements belong together, e.g. all entities of one imported JSON Schema document. It carries no member content of its own; the members are stored separately and referenced by URN.

A DataStructure is itself a JSON Schema: a $defs library with one entry per member, whose value is a $ref to that member's CORE URN.

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "urn:core:platform:civitas:datastructure:common:air-quality:zgpsfmmplg:1.0.0",
  "title": "Air Quality",
  "$defs": {
    "SensorReading": {
      "$ref": "urn:core:platform:civitas:element:common:SensorReading:d28s38wfmi:1.0.0"
    }
  }
}

$id is a versioned CORE URN with the datastructure artifact-type segment. Each $defs entry is a $ref to a member Element and produces one datastructure-ref edge. An optional top-level $ref designates one member as the root shape, so a root-shaped schema can be derived for a client that needs one; because its target is already a $defs member it adds no second edge. title and description are optional metadata.

A read gives the library back as stored. A bundled or inlined read embeds the member Elements again.

Mapping

A Mapping describes a declarative mapping between two Elements. source and target are global Element URNs.

{
  "$schema": "https://civitasconnect.digital/core/mapping/v1",
  "id": "urn:core:platform:civitas:mapping:common:sensor-to-observation:0qyaberslc:1.0.0",
  "source": "urn:core:platform:civitas:element:common:SensorReading:d28s38wfmi:1.0.0",
  "target": "urn:core:platform:civitas:element:common:Observation:ulhry9fjx6:1.0.0",
  "fields": {
    "$.result": { "op": "copy", "input": "$.temperature" },
    "$.source": { "op": "const", "value": "mqtt" },
    "$.name": {
      "op": "concat",
      "inputs": ["$.sensorId", "$.metric"],
      "separator": ":"
    }
  }
}

Field operations:

Operation	Required fields	Semantics
`copy`	`input`	Copies a value from the source object
`concat`	`inputs`	Concatenation of multiple source values (optional `separator`)
`const`	`value`	Writes a constant value
`toString`	`input`	Converts the input value to its string representation
`toInt`	`input`	Parses the input value as an integer
`toFloat`	`input`	Parses the input value as a floating-point number
`toDate`	`input`, `pattern`	Parses the input string into a date/time value using a date pattern (e.g. `yyyy-MM-dd`)
`format`	`input`, `pattern`	Formats the input date/time value into a string using a date pattern

As a short form, a field value may be a string. Semantically this is a copy mapping with the string as the source path.

DataSource and DataSink

DataSources and DataSinks describe technical connections and the Element of the respective payload.

{
  "$schema": "https://civitasconnect.digital/core/datasource/v1",
  "id": "urn:core:platform:civitas:datasource:common:aq-mqtt-source:yv3oxe088a:1.0.0",
  "connectionType": "mqtt",
  "brokerUrl": "tcp://mqtt.example.test:1883",
  "topic": "city/air-quality/+/reading",
  "element": "urn:core:platform:civitas:element:common:SensorReading:d28s38wfmi:1.0.0"
}

{
  "$schema": "https://civitasconnect.digital/core/datasink/v1",
  "id": "urn:core:platform:civitas:datasink:common:aq-http-sink:s7mqw72qmh:1.0.0",
  "connectionType": "http",
  "url": "https://data.example.test/observations",
  "method": "POST",
  "element": "urn:core:platform:civitas:element:common:Observation:ulhry9fjx6:1.0.0"
}

element is a foreign key (x-core-ref): its value is the CORE URN ($id) of the Element that describes the payload — a reference, not an embedded schema.

Pipeline

A Pipeline is a directed graph of nodes and edges.

{
  "$schema": "https://civitasconnect.digital/core/pipeline/v1",
  "id": "urn:core:platform:civitas:pipeline:common:air-quality-ingest:qubihav71f:1.0.0",
  "nodes": [
    { "id": "start", "kind": "start", "x-ui-position": { "x": 80, "y": 180 } },
    {
      "id": "source",
      "kind": "source",
      "sourceRef": "urn:core:platform:civitas:datasource:common:aq-mqtt-source:yv3oxe088a:1.0.0",
      "x-ui-position": { "x": 240, "y": 180 }
    },
    {
      "id": "mapping",
      "kind": "mapping",
      "mappingRef": "urn:core:platform:civitas:mapping:common:sensor-to-observation:0qyaberslc:1.0.0",
      "x-ui-position": { "x": 420, "y": 180 }
    },
    {
      "id": "sink",
      "kind": "sink",
      "sinkRef": "urn:core:platform:civitas:datasink:common:aq-http-sink:s7mqw72qmh:1.0.0",
      "x-ui-position": { "x": 600, "y": 180 }
    },
    { "id": "end", "kind": "end", "x-ui-position": { "x": 760, "y": 180 } }
  ],
  "edges": [
    { "id": "e1", "source": "start", "target": "source", "kind": "control" },
    { "id": "e2", "source": "source", "target": "mapping", "kind": "data" },
    { "id": "e3", "source": "mapping", "target": "sink", "kind": "data" },
    { "id": "e4", "source": "sink", "target": "end", "kind": "control" }
  ]
}

Node kinds:

`kind`	Reference fields	Semantics
`start`	none	Start point
`end`	none	End point
`source`	`sourceRef`	Reads from a DataSource
`filter`	`expression`	Filters records
`enrich`	`lookupSourceRef`, `lookupKey`	Enriches via a lookup source
`mapping`	`mappingRef`	Applies a mapping
`sink`	`sinkRef`	Writes to a DataSink
`split`	none	Splits a flow into multiple records

Pipeline edges reference node IDs within the same pipeline. These node IDs are local graph IDs, not CORE URNs.

References between artifacts

CORE artifacts reference each other by CORE URN — a foreign key. The field is a plain string holding the target's URN; the target is not embedded. The x-core-ref annotation marks such a field and names the artifact type the URN points at.

This is the counterpart to a JSON Schema $ref, which embeds the referenced schema (the value at that position is an object shaped like the target):

$ref — composition: the value is (an instance of) the target.
x-core-ref — association / foreign key: the value names the target.

References are always global URNs; existence is checked by registry-aware services (see Validation). A reference may pin a concrete version (…:Foo:1.0.0) or use the latest token (…:Foo:latest) — see URN Format; it is stored verbatim and resolved to a concrete version on read.

Examples of x-core-ref fields: DataSet.datastructureRefs[] / mappingRefs[] / …, Mapping.source / Mapping.target, DataSource.element, Pipeline.nodes[].sourceRef / mappingRef / sinkRef.

Example: a foreign key between Elements

A Baum (tree) references the Strasse (street) it stands on — by reference, not by embedding. stehtAn is a plain string field annotated as a foreign key:

// Element: Strasse
{
  "$id": "urn:core:platform:civitas:element:common:Strasse:u8pwgr2zzg:1.0.0",
  "title": "Strasse",
  "type": "object",
  "properties": {
    "id":   { "type": "string" },
    "name": { "type": "string" }
  }
}

// Element: Baum — "stehtAn" is a foreign key to the Strasse Element
{
  "$id": "urn:core:platform:civitas:element:common:Baum:vd1c1ziwlu:1.0.0",
  "title": "Baum",
  "type": "object",
  "properties": {
    "name":    { "type": "string" },
    "stehtAn": {
      "type": "string",
      "x-core-ref": { "type": "urn:core:platform:civitas:element:common:Strasse:u8pwgr2zzg:1.0.0" }
    }
  }
}

The type names the concrete target Element (Strasse), so the foreign key is strongly typed: x-core-ref references the Strasse Element (artifact), and on import Model Forge checks that this Element exists in the registry (see Validation). In instance data the stehtAn field then holds an application-level key that identifies a Strasse record — the street is pointed at, not copied in:

// instance of Strasse
{ "$schema": "urn:core:platform:civitas:element:common:Strasse:u8pwgr2zzg:1.0.0",
  "id": "42", "name": "BerlinerAllee" }

// instance of Baum — stehtAn references the Strasse above
{ "$schema": "urn:core:platform:civitas:element:common:Baum:vd1c1ziwlu:1.0.0",
  "name": "Fichte", "stehtAn": "42" }

Note the two levels: the schema-level x-core-ref.type is a CORE artifact URN (the Strasse Element — this is what the registry existence check resolves), while the instance value ("42") is the application's own join key into Strasse records and is not a CORE URN. Model Forge validates the schema-level reference, not instance join keys.

Had stehtAn used $ref instead, a Baum instance would have to embed the whole Strasse object inline rather than reference it.

`x-core-ref`

x-core-ref is a CORE-specific JSON Schema annotation. It marks a string field as a foreign key: the value is the CORE URN of another artifact. JSON Schema alone cannot express this, so the keyword records the target type.

The only key is type — the CORE URN of the referenced artifact. It is either a concrete artifact URN (a strongly-typed foreign key, e.g. to a specific Element — its existence is checked against the registry) or an urn:core:type:<Kind> category URN (used by the generic CORE-IR artifact meta-schemas, where the concrete target is not fixed):

{ "type": "urn:core:platform:civitas:element:common:Strasse:u8pwgr2zzg:1.0.0" }
{ "type": "urn:core:type:Element" }
{ "type": "urn:core:type:Mapping" }
{ "type": "urn:core:type:Pipeline" }
{ "type": "urn:core:type:DataSource" }
{ "type": "urn:core:type:DataSink" }

There is no scope (every reference is a global URN) and no collection / key field. Not allowed (examples of rejected shapes):

{ "type": "#/$defs/Element" }
{ "type": "Element" }
{ "type": "urn:core:type:Element", "scope": "global" }
{ "type": "urn:core:type:Element", "collection": "elements" }

Other `x-*` attributes

All allowed CORE extensions are declared in core-schema-extensions.schema.json. New x-* attributes must first be defined there as a shape.

`x-ui-position`

x-ui-position stores canvas coordinates for pipeline nodes.

{
  "id": "mapping",
  "kind": "mapping",
  "mappingRef": "urn:core:platform:civitas:mapping:common:sensor-to-observation:0qyaberslc:1.0.0",
  "x-ui-position": { "x": 420, "y": 180 }
}

Semantics: pure UI annotation, no domain effect on mapping, validation or generator logic. Replaces a domain-level position field — both must not exist at the same time.

`x-xsd-source`

x-xsd-source points from a JSON Schema projection back at the Element URN whose version stores the source XSD representation it was converted from.

{
  "$id": "urn:core:platform:civitas:element:common:AirQualityJson:alhvv5pu1y:1.0.0",
  "x-xsd-source": "urn:core:platform:civitas:element:common:AirQuality:wyonmizavr:1.0.0",
  "type": "object"
}

Semantics: documents the origin of the JSON Schema view and links the XSD and the JSON Schema projection at the domain level. It is not a substitute for $id.

Validation

x-core-ref is a typed annotation, not a JSON-Schema-level assertion:

Structural validation — JSON Schema checks required fields, types, URN patterns (pattern: "^urn:core:") and allowed node/operation types. x-core-ref itself is a non-validating annotation here (it records the target type).
Referential validation (registry-aware) — whether a referenced URN resolves to an existing artifact is the registry's concern, not pure JSON Schema. On Element import/update Model Forge checks every concrete x-core-ref target against the registry (ReferenceExistenceValidator): an unknown target is rejected with diagnostic code unresolved-core-ref, and a type that is neither a well-formed CORE URN nor an urn:core:type:<Kind> category marker is rejected with invalid-core-ref. Targets created by the same request and urn:core:type:<Kind> category markers are not checked, and the check is skipped when no registry is configured.

Backend: x-core-ref is registered as an annotation keyword (no in-document validator); existence is enforced by ReferenceExistenceValidator.
Governance: a test verifies that all x-* attributes are declared in the extension schema.

Design rules

Use CORE URNs as reference values, not plain names.
x-core-ref (foreign key, by URN) and $ref (embedding) are distinct — pick reference vs. composition deliberately.
Use $id for Elements, id for CORE artifacts.
Use x-ui-position for canvas positions, not position.
Introduce new x-* attributes only if they are described and tested in the extension schema.
Keep DataSet manifests lean: refs instead of duplicated content, provided a global store is available.

Terms

Term	Meaning
Artifact	Persistable CORE unit such as Mapping, Pipeline or Element
DataSet manifest	Lean DataSet with `*Refs` to global artifacts
`x-core-ref`	Foreign key: the field holds the CORE URN of another artifact
`$ref`	Embedding: the referenced schema is composed in (the value is the target)
`type` (in `x-core-ref`)	CORE URN of the referenced artifact: a concrete artifact URN (strongly-typed FK) or an `urn:core:type:...` category URN

Basic model​

$schema, id and $id​

Artifact envelope​

DataSet​

Element​

DataStructure​

Mapping​

DataSource and DataSink​

Pipeline​

References between artifacts​

Example: a foreign key between Elements​

x-core-ref​

Other x-* attributes​

x-ui-position​

x-xsd-source​

Validation​

Design rules​

Terms​