Deletion & DataSet Membership Policy

Status: implemented (Model Forge + portal-backend). See "Implementation" below.

This document defines when a CORE artifact may be deleted, how DataSet membership affects deletion, and how to find artifacts that belong to no DataSet. It is the authoritative reference for the delete behaviour of ModelForge.deleteArtifact(…) and the portal-backend delete endpoints.

Artifact kinds and references

Every stored artifact (DataStructure, Element, Mapping, Pipeline, DataSource, DataSink, DataSet) can reference other artifacts. Model Forge records each reference as a typed edge in artifact_reference (reference_type), for example:

• datastructure-ref — a DataStructure grouping one of its member Elements.
• mapping-source / mapping-target — a Mapping binding its DataStructures.
• pipeline-node — a Pipeline node referencing a source / sink / mapping.
• dataset-ref — a DataSet manifest referencing one of its members (introduced with the DataSet artifact; see below).

Referential integrity is the baseline: an artifact that another artifact still points at must not silently disappear, or the reference would dangle.

DataSet membership

A DataSet is a first-class artifact — a manifest that references its members by versioned CORE URN (pipelineRefs, dataSinkRefs, plus derived dataSourceRefs / mappingRefs / datastructureRefs). Membership is expressed as dataset-ref edges from the DataSet manifest to each member.

• Pipeline — the only exclusive member: a Pipeline carries a @NotNull dataSet FK, so it belongs to exactly one DataSet. It is linked by writing it with the optional dataSet argument (dataset-ref edge manifest → pipeline).
• Reusable artifacts — DataSources, DataSinks, Mappings and DataStructures are reusable and may belong to several DataSets at once.

Membership of a reusable artifact in a DataSet arises two ways (kept consistent):

1. Auto-link through Pipelines — when a Pipeline is written with dataSet = D, Model Forge links the Pipeline and every artifact it references (its sources/sinks/mappings and, transitively, their DataStructures) as members of D.
2. Explicit assignment — a reusable artifact can be added to (or removed from) a DataSet directly, independent of any Pipeline using it.

Both produce dataset-ref edges from the DataSet manifest to the member; an artifact can therefore be a member of zero, one, or many DataSets.

Delete rules

Let D(x) = the set of DataSets whose manifest references artifact x (dataset-ref in-edges), and N(x) = every non-DataSet artifact that references x (e.g. a Pipeline referencing a Mapping).

Deleting a member artifact x

1. Non-DataSet references block, always. If N(x) is non-empty, the delete is rejected with ArtifactInUseException listing the blockers. You must remove/update those referencing artifacts first. (A Mapping used by a Pipeline cannot be deleted while that Pipeline exists.)
2. DataSet membership is count-based:
   • |D(x)| = 0 (orphan) → deletable.
   • |D(x)| = 1 → deletable, and Model Forge removes x from that one DataSet's manifest as part of the delete (auto-unlink).
   • |D(x)| ≥ 2 → blocked. The member is shared across DataSets; remove it from the DataSets first (all but at most one), then delete.

So a member is deletable exactly when it has no non-DataSet referrers and is in at most one DataSet.

Deleting a DataSet S

Deleting a DataSet deletes only its manifest — its members are not deleted (a DataSet is a grouping, not an owner). The manifest's outgoing dataset-ref edges disappear with it, so each former member's D(x) shrinks by one (a member that was in only S becomes an orphan, still present and reusable).

Nothing references a DataSet manifest, so a DataSet is never blocked by member references.

cascade and force

Two opt-in modifiers exist; both require deliberate intent and are dangerous.

• cascade — delete the target and its members, descending the reference graph. A member is cascade-deleted only if it becomes an orphan once the target is gone (no other non-DataSet referrer, in no other DataSet); shared members are kept. Cascade recurses and always stops at anything still in use. Deleting a DataSet with cascade therefore removes the manifest plus every member that was exclusive to it, keeping members shared with other DataSets or used by surviving Pipelines.

• force — override the blocking checks and delete the target regardless of referrers. This can leave dangling references and must be reserved for administrative repair; it is never the default and should be gated + audited. Prefer removing the blockers, or cascade, over force.

  The platform does not expose force. It is a parameter of the Model Forge facade, and ModelRegistryGateway always passes false; the gateway's own deleteArtifact has no force parameter at all. A force delete therefore cannot be triggered through the portal backend API. The paragraph above describes what the library can do, not what an operator can call.

The safe default is deleteArtifact(x) = no cascade, no force: it deletes only when the rules above permit, and never touches anything else (beyond the single-DataSet auto-unlink).

Finding orphans (artifacts in no DataSet), by type

It must be possible to list artifacts of a given type that belong to no DataSet — e.g. "all Mappings / Pipelines / DataSources not in any DataSet" — so they can be reviewed, assigned, or cleaned up. This is a generic membership query, expressed as: artifacts of kind T with no dataset-ref in-edge.

Model Forge exposes this generically (kind + membership filter) rather than one method per type, e.g. an orphans query taking an artifact kind and returning the matching ArtifactSummary list. The portal-backend surfaces it as a read-only maintenance endpoint (filterable by type).

Facade note: today the facade offers search (by text/type/format) and dependents (reverse edges) — the building blocks — but not a single orphan-by-membership query. This policy adds that generic query rather than composing it client-side per type.

Summary table

Target	Referenced by non-DataSet artifact?	In N DataSets	Default delete
Member	yes	any	blocked (remove referrers first)
Member	no	0	deleted
Member	no	1	deleted + unlinked from that DataSet
Member	no	≥2	blocked (remove from DataSets first)
DataSet	(n/a — nothing references it)	—	deleted, members kept

cascade extends a delete downward into exclusively-owned members; force overrides the blocks and may dangle references — use with great care.

Implementation

• Model Forge — saveArtifact/createArtifact take an optional dataSet URN; when set, the artifact is linked into that DataSet's manifest, and for a Pipeline its whole reference closure (sources/sinks/mappings and their DataStructures) is linked too. Membership is stored as uniform dataset-ref edges. deleteArtifact(id, cascade, force) applies the policy above; orphans(kind) returns artifacts of a kind with no dataset-ref in-edge; linkToDataSet/unlinkFromDataSet are the explicit-assignment operations.
• portal-backend — a DataSet's manifest is created when the dataset is created (DataSet.manifestLogicalUrn) and deleted with the dataset (members kept). PipelineService passes its dataset's URN on save (auto-link). Endpoints: GET /v1/orphans?type=…, POST /v1/datasets/{id}/members ({artifactUrn}), DELETE /v1/datasets/{id}/members?artifactUrn=….
• Verified — the roundtrip E2E asserts a saved pipeline links its entire closure into the dataset manifest as dataset-ref members, and the full suite stays green (the policy does not break existing deletes).