ADR 051: DataSet as an artifact manifest
Date: 2026-07-15
Status: Accepted
Decision Makers: @derlinne, @luckey
Context
A DataSetDatasetA data-related element that contains processed data and makes it available for consumption. A Dataset is populated via Pipelines and carries Metadata and access permissions. composes several sub-resources that also need to exist and evolve independently.
Decision
The DataSetDatasetA data-related element that contains processed data and makes it available for consumption. A Dataset is populated via Pipelines and carries Metadata and access permissions. is a lightweight manifest in the registry that references its sub-resources by URN (elementRefs, mappingRefs, pipelineRefs, dataSourceRefs, dataSinkRefs). Sub-resources are independent artifacts.
Rationale
- Each resource has its own versioning history.
- Compositions (one schema in multiple DataSetsDatasetA data-related element that contains processed data and makes it available for consumption. A Dataset is populated via Pipelines and carries Metadata and access permissions.) are possible without data duplication.
- Clear separation: DataSetDatasetA data-related element that contains processed data and makes it available for consumption. A Dataset is populated via Pipelines and carries Metadata and access permissions. = composition container, sub-resources = reusable building blocks.
Consequences
A DataSetDatasetA data-related element that contains processed data and makes it available for consumption. A Dataset is populated via Pipelines and carries Metadata and access permissions. read returns the stored manifest; consumers fetch referenced artifacts individually by their URNs.