Connect external Data

Who is this guide for?

Role: Data Architect, Data Steward

You need to collaborate with: Data Owner, Data Gatekeeper

Goal: You want to connect an external Data sourceData sourceA data-related element that represents the origin of data. It defines how data is connected, accessed, and ingested into the Platform, such as an external database or sensor network. to the Platform and make it available for further processing.

Required Permissions:

• Data Permissions: Create, Read, Update, and Release Data structureData structureA versioned data-related element that defines the schema and organization of data. It specifies how data is structured and interpreted within the Platform.; Create, Read, Update, and Release Data sourceData sourceA data-related element that represents the origin of data. It defines how data is connected, accessed, and ingested into the Platform, such as an external database or sensor network.
• System Permissions: Read Group, Read Role

What you will achieve

After completing this guide, you will have:

• Created a Data structureData structureA versioned data-related element that defines the schema and organization of data. It specifies how data is structured and interpreted within the Platform.
• Connected a Data sourceData sourceA data-related element that represents the origin of data. It defines how data is connected, accessed, and ingested into the Platform, such as an external database or sensor network.
• Linked the Data sourceData sourceA data-related element that represents the origin of data. It defines how data is connected, accessed, and ingested into the Platform, such as an external database or sensor network. to the Data structureData structureA versioned data-related element that defines the schema and organization of data. It specifies how data is structured and interpreted within the Platform.
• Managed access permissions
• Made the Data sourceData sourceA data-related element that represents the origin of data. It defines how data is connected, accessed, and ingested into the Platform, such as an external database or sensor network. available for pipelines

Your collaborators

This process includes a handover between roles:

• A Data Architect or Data Steward prepares the Data structuresData structureA versioned data-related element that defines the schema and organization of data. It specifies how data is structured and interpreted within the Platform. and Data sourcesData sourceA data-related element that represents the origin of data. It defines how data is connected, accessed, and ingested into the Platform, such as an external database or sensor network.
• A Data Owner or Data Gatekeeper must review and release them

→ You cannot complete this guide alone if you do not have release permissions.

Before you start

• Verify that you can create Data structuresData structureA versioned data-related element that defines the schema and organization of data. It specifies how data is structured and interpreted within the Platform. and Data sourcesData sourceA data-related element that represents the origin of data. It defines how data is connected, accessed, and ingested into the Platform, such as an external database or sensor network.. If these options are unavailable, contact your TenantTenantAn isolated organizational partition that owns Data pools, Datasets, Users, Groups, and Roles. All access rules exist within their Tenant, and the Tenant is the widest Scope of a Role. Currently, one Tenant corresponds to the Platform. Admin to request access.
• Identify a Data Owner or Data Gatekeeper who can authorize the release of your Data-related elementsData-related elementsA collective term for Datasets, Data structures, Data pools and Data sources..

Understanding your source data

Before connecting a Data sourceData sourceA data-related element that represents the origin of data. It defines how data is connected, accessed, and ingested into the Platform, such as an external database or sensor network., make sure you understand the structure and meaning of your source data. In CIVITAS/CORE, a Data sourceData sourceA data-related element that represents the origin of data. It defines how data is connected, accessed, and ingested into the Platform, such as an external database or sensor network. always must comply with a Data structureData structureA versioned data-related element that defines the schema and organization of data. It specifies how data is structured and interpreted within the Platform..

This means:

• the data of the Data sourceData sourceA data-related element that represents the origin of data. It defines how data is connected, accessed, and ingested into the Platform, such as an external database or sensor network. must match a defined structure definition
• relationships between entities must be clear
• data fields must be mapped consistently

Example use case: Integrate Smart Meter energy usage data

In this guide, we demonstrate this process using a use case which collects energy usage data from Smart Meters ("TAF10 data") as an example. It shows how to ingest sensor data and store it as Observations in a SensorThings APISensorThings APIA standardized API based on the OGC SensorThings API specification for accessing time series and IoT data. It enables structured retrieval and management of observations and related entities. backend.

→ You will find additional context from the Smart Meter energy use case in the highlighted boxes throughout this guide.

Requirements

Two types of data are required:

• Master data → defines devices, sensors, and datastreams
• Measurement data → continuous sensor values received via MQTT

Make sure you have access to

• a PostgreSQL database (master data)
• a MQTT broker (measurement data)

Before measurement data can be processed, the master data must be made available.

→ In this guide, you will connect both Data sourcesData sourceA data-related element that represents the origin of data. It defines how data is connected, accessed, and ingested into the Platform, such as an external database or sensor network. required for this setup.

Step-by-step guide

Step 1: Create a Data structure

1. Go to Data structuresData structureA versioned data-related element that defines the schema and organization of data. It specifies how data is structured and interpreted within the Platform.

2. Click Create Data structureData structureA versioned data-related element that defines the schema and organization of data. It specifies how data is structured and interpreted within the Platform.
3. Enter Base information and hit Create and continue

Example use case: Integrate Smart Meter energy usage data

Create a Data structureData structureA versioned data-related element that defines the schema and organization of data. It specifies how data is structured and interpreted within the Platform. for the master data required to model devices and their energy usage datastreams.

Example base information:

• Name: Smart Meter Energy master data
• Description: Smart meter device and energy datastream structure

Step 2: Define the Data structure

→ In CIVITAS/CORE, a Data structureData structureA versioned data-related element that defines the schema and organization of data. It specifies how data is structured and interpreted within the Platform. is versioned. Each version represents a specific structure definition.

You can define the structure manually or use an existing structure as a starting point. The Data structureData structureA versioned data-related element that defines the schema and organization of data. It specifies how data is structured and interpreted within the Platform. Editor supports:

• Manual definition: Create Classes, Attributes, Enums, and Relationships directly in the Editor.
• Import from file: Import a Data structureData structureA versioned data-related element that defines the schema and organization of data. It specifies how data is structured and interpreted within the Platform. from a JSON file, for example one exported from another CIVITAS/CORE instance.
• Standard structures: Use predefined structures for supported use cases, such as SensorThings.

1. Create a new version in the Versions tab
2. Define your structure in the Structure Definition tab
3. Open the Version Information tab
4. Enter a Description and hit Save

Learn how to design Data structures

Create Classes, add Attributes, connect Classes, and configure Relationships → Data structure Editor

Example use case: Integrate Smart Meter energy usage data

For the Smart Meter use case, you can use a SensorThings Standard structure as a starting point instead of defining the complete structure manually.

The structure represents the SensorThings entities required for the use case, such as Things, Sensors, Observed Properties, and Datastreams.

This structure will later be used for the master data from the PostgreSQL Data sourceData sourceA data-related element that represents the origin of data. It defines how data is connected, accessed, and ingested into the Platform, such as an external database or sensor network..

Step 3: Manage access permissions of the Data structure

Required Permissions

Update Data structureData structureA versioned data-related element that defines the schema and organization of data. It specifies how data is structured and interpreted within the Platform. is required to manage access. Adding a new Group–Role Assignment additionally requires the Read Group and Read Role system permissions. Existing Assignments can be removed with Update Data structureData structureA versioned data-related element that defines the schema and organization of data. It specifies how data is structured and interpreted within the Platform. without these system permissions.

1. Open the Access tab
2. Add a Group

3. Assign a Role to the Group

4. Repeat these steps to add more Groups and Roles
5. Hit Save and Exit

Learn how authorisation works

• Groups that have platform-wide permissions on all Data-related elementsData-related elementsA collective term for Datasets, Data structures, Data pools and Data sources. will have access to this Data structureData structureA versioned data-related element that defines the schema and organization of data. It specifies how data is structured and interpreted within the Platform. automatically.
• Removing your own Group Role assignment may result in the loss of your permissions for this Data structureData structureA versioned data-related element that defines the schema and organization of data. It specifies how data is structured and interpreted within the Platform..
• Access in CIVITAS/CORE is defined by assigning a Role to a Group within a specific Scope, f.e. this Data structureData structureA versioned data-related element that defines the schema and organization of data. It specifies how data is structured and interpreted within the Platform.. → Learn more about the authorisation

Step 4: Continue in Draft for iterative editing

Keep the Data structureData structureA versioned data-related element that defines the schema and organization of data. It specifies how data is structured and interpreted within the Platform. and its Version in Draft and continue with the next step.

While in Draft, you can continue editing the Data structureData structureA versioned data-related element that defines the schema and organization of data. It specifies how data is structured and interpreted within the Platform. Version and its structure definition as you build and adjust the complete data flow. This supports an iterative workflow before releasing the Data structureData structureA versioned data-related element that defines the schema and organization of data. It specifies how data is structured and interpreted within the Platform..

Release the Data structure

If you want to release the Data structureData structureA versioned data-related element that defines the schema and organization of data. It specifies how data is structured and interpreted within the Platform. at this point and you have the required Release Data structureData structureA versioned data-related element that defines the schema and organization of data. It specifies how data is structured and interpreted within the Platform. Permission, you can set the Version and Data structureData structureA versioned data-related element that defines the schema and organization of data. It specifies how data is structured and interpreted within the Platform. to Available.

Otherwise, continue working with the Draft Data structureData structureA versioned data-related element that defines the schema and organization of data. It specifies how data is structured and interpreted within the Platform.. It can be released later when the complete DatasetDatasetA data-related element that contains processed data and makes it available for consumption. A Dataset is populated via Pipelines and carries Metadata and access permissions. setup is finished.

Editing options may become restricted after release or when the Data structureData structureA versioned data-related element that defines the schema and organization of data. It specifies how data is structured and interpreted within the Platform. is in use. → Learn more about Status lifecycle and editing

Outcome

The Data structureData structureA versioned data-related element that defines the schema and organization of data. It specifies how data is structured and interpreted within the Platform. and its Version can now be used in Draft status in Data sourcesData sourceA data-related element that represents the origin of data. It defines how data is connected, accessed, and ingested into the Platform, such as an external database or sensor network. and Pipelines.

Step 5: Create a Data source

1. Go to Data sourcesData sourceA data-related element that represents the origin of data. It defines how data is connected, accessed, and ingested into the Platform, such as an external database or sensor network.

2. Click Create Data sourceData sourceA data-related element that represents the origin of data. It defines how data is connected, accessed, and ingested into the Platform, such as an external database or sensor network.
3. Enter Base Information and hit Create and continue

Example use case: Integrate Smart Meter energy usage data

For this use case start with the PostgreSQL data sourceData sourceA data-related element that represents the origin of data. It defines how data is connected, accessed, and ingested into the Platform, such as an external database or sensor network.. It provides the master data required before measurement data can be processed.

Example base information:

• Name: Smart Meter PostgreSQL master data
• Description: Master data sourceData sourceA data-related element that represents the origin of data. It defines how data is connected, accessed, and ingested into the Platform, such as an external database or sensor network. for devices and datastreams used in Smart Meter

Step 6: Configure the Connector of the Data source

1. Open the Connector tab
2. Select a Connector type
3. Enter the required Configuration details and hit Save

Example use case: Integrate Smart Meter energy usage data

This PostgreSQL Data sourceData sourceA data-related element that represents the origin of data. It defines how data is connected, accessed, and ingested into the Platform, such as an external database or sensor network. provides the master data used to provision SensorThings entities in the backend.

Example: PostgreSQL database

Step 7: Assign a Data structure to the Data source

1. Open the Data structureData structureA versioned data-related element that defines the schema and organization of data. It specifies how data is structured and interpreted within the Platform. tab
2. Click Import and select the Data structureData structureA versioned data-related element that defines the schema and organization of data. It specifies how data is structured and interpreted within the Platform. from Platform you created earlier

Example use case: Integrate Smart Meter energy usage data

Assign the Data structureData structureA versioned data-related element that defines the schema and organization of data. It specifies how data is structured and interpreted within the Platform. that models the SensorThings master data.

Step 8: Configure Data source availability

1. Open the Data poolsData poolA governed, centralized collection of Datasets that are managed together within the Platform. It provides a shared place to organize, discover, and access data, including associated metadata, ownership, and access permissions. With Data pools users can cluster their Datasets according to their organizational structure (e.g., by Departments, Offices). Formerly called Data space. tab

2. Decide

   Either: You keep the default setting "Approve for all Data poolsData poolA governed, centralized collection of Datasets that are managed together within the Platform. It provides a shared place to organize, discover, and access data, including associated metadata, ownership, and access permissions. With Data pools users can cluster their Datasets according to their organizational structure (e.g., by Departments, Offices). Formerly called Data space." (DE: Für alle Datenpools freigegeben) enabled. This makes the Data sourceData sourceA data-related element that represents the origin of data. It defines how data is connected, accessed, and ingested into the Platform, such as an external database or sensor network. available to DatasetsDatasetA data-related element that contains processed data and makes it available for consumption. A Dataset is populated via Pipelines and carries Metadata and access permissions. in all Data poolsData poolA governed, centralized collection of Datasets that are managed together within the Platform. It provides a shared place to organize, discover, and access data, including associated metadata, ownership, and access permissions. With Data pools users can cluster their Datasets according to their organizational structure (e.g., by Departments, Offices). Formerly called Data space., as well as to DatasetsDatasetA data-related element that contains processed data and makes it available for consumption. A Dataset is populated via Pipelines and carries Metadata and access permissions. without a Data poolData poolA governed, centralized collection of Datasets that are managed together within the Platform. It provides a shared place to organize, discover, and access data, including associated metadata, ownership, and access permissions. With Data pools users can cluster their Datasets according to their organizational structure (e.g., by Departments, Offices). Formerly called Data space. assignment.

   Or: You disable the setting "Approve for all Data poolsData poolA governed, centralized collection of Datasets that are managed together within the Platform. It provides a shared place to organize, discover, and access data, including associated metadata, ownership, and access permissions. With Data pools users can cluster their Datasets according to their organizational structure (e.g., by Departments, Offices). Formerly called Data space." (DE: Für alle Datenpools freigegeben). In this case, you can specify the individual Data poolsData poolA governed, centralized collection of Datasets that are managed together within the Platform. It provides a shared place to organize, discover, and access data, including associated metadata, ownership, and access permissions. With Data pools users can cluster their Datasets according to their organizational structure (e.g., by Departments, Offices). Formerly called Data space. for which this Data sourceData sourceA data-related element that represents the origin of data. It defines how data is connected, accessed, and ingested into the Platform, such as an external database or sensor network. should be released. If you disable the default setting without adding any specific Data poolsData poolA governed, centralized collection of Datasets that are managed together within the Platform. It provides a shared place to organize, discover, and access data, including associated metadata, ownership, and access permissions. With Data pools users can cluster their Datasets according to their organizational structure (e.g., by Departments, Offices). Formerly called Data space., the Data sourceData sourceA data-related element that represents the origin of data. It defines how data is connected, accessed, and ingested into the Platform, such as an external database or sensor network. cannot be used in any DatasetDatasetA data-related element that contains processed data and makes it available for consumption. A Dataset is populated via Pipelines and carries Metadata and access permissions..

3. Hit Save

Step 9: Manage access permissions of the Data source

Required Permissions

Update Data sourceData sourceA data-related element that represents the origin of data. It defines how data is connected, accessed, and ingested into the Platform, such as an external database or sensor network. is required to manage access. Adding a new Group–Role Assignment additionally requires the Read Group and Read Role system permissions. Existing Assignments can be removed with Update Data sourceData sourceA data-related element that represents the origin of data. It defines how data is connected, accessed, and ingested into the Platform, such as an external database or sensor network. without these system permissions.

1. Open the Access tab
2. Add a Group

3. Assign a Role to the Group

4. Repeat these steps to add more Groups and Roles
5. Hit Save and Exit

Step 10: Keep the Data source in Draft for iterative editing

Keep the Data sourceData sourceA data-related element that represents the origin of data. It defines how data is connected, accessed, and ingested into the Platform, such as an external database or sensor network. in Draft and continue with the next in your workflow.

While in Draft, you can continue editing the Data sourceData sourceA data-related element that represents the origin of data. It defines how data is connected, accessed, and ingested into the Platform, such as an external database or sensor network. as you build and adjust the complete data flow.

Release the Data source

If you want to release the Data sourceData sourceA data-related element that represents the origin of data. It defines how data is connected, accessed, and ingested into the Platform, such as an external database or sensor network. at this point and you have the required Release Data sourceData sourceA data-related element that represents the origin of data. It defines how data is connected, accessed, and ingested into the Platform, such as an external database or sensor network. Permission, you can set the Data sourceData sourceA data-related element that represents the origin of data. It defines how data is connected, accessed, and ingested into the Platform, such as an external database or sensor network. to Available.

For an iterative workflow, you can keep it in Draft and release it later. Editing options may become restricted after release or when the Data sourceData sourceA data-related element that represents the origin of data. It defines how data is connected, accessed, and ingested into the Platform, such as an external database or sensor network. is in use. → Learn more about Status lifecycle and editing

Outcome

The Data sourceData sourceA data-related element that represents the origin of data. It defines how data is connected, accessed, and ingested into the Platform, such as an external database or sensor network. can now be used in Draft status in Pipelines according to its configured Data poolData poolA governed, centralized collection of Datasets that are managed together within the Platform. It provides a shared place to organize, discover, and access data, including associated metadata, ownership, and access permissions. With Data pools users can cluster their Datasets according to their organizational structure (e.g., by Departments, Offices). Formerly called Data space. availability.

Summary

You have successfully:

• Created and defined a Data structureData structureA versioned data-related element that defines the schema and organization of data. It specifies how data is structured and interpreted within the Platform.
• Connected and configured a Data sourceData sourceA data-related element that represents the origin of data. It defines how data is connected, accessed, and ingested into the Platform, such as an external database or sensor network.
• Configured the Data sourceData sourceA data-related element that represents the origin of data. It defines how data is connected, accessed, and ingested into the Platform, such as an external database or sensor network. availability for Data poolsData poolA governed, centralized collection of Datasets that are managed together within the Platform. It provides a shared place to organize, discover, and access data, including associated metadata, ownership, and access permissions. With Data pools users can cluster their Datasets according to their organizational structure (e.g., by Departments, Offices). Formerly called Data space.
• Configured access permissions
• Prepared the Data structureData structureA versioned data-related element that defines the schema and organization of data. It specifies how data is structured and interpreted within the Platform. and Data sourceData sourceA data-related element that represents the origin of data. It defines how data is connected, accessed, and ingested into the Platform, such as an external database or sensor network. for use in Pipelines

Example use case: Integrate Smart Meter energy usage data

Next: Repeat this process for measurement data

Create a second Data structureData structureA versioned data-related element that defines the schema and organization of data. It specifies how data is structured and interpreted within the Platform. and Data sourceData sourceA data-related element that represents the origin of data. It defines how data is connected, accessed, and ingested into the Platform, such as an external database or sensor network. for the incoming measurement data, this time using MQTT instead of PostgreSQL.

Data structureData structureA versioned data-related element that defines the schema and organization of data. It specifies how data is structured and interpreted within the Platform. Create a Data structureData structureA versioned data-related element that defines the schema and organization of data. It specifies how data is structured and interpreted within the Platform. that describes the format of the incoming smart meter measurements.

Example Base information:

• Name: Smart Meter Energy measurement data
• Description: Structure for incoming smart meter measurement data

The Structure Definition should represent the incoming MQTT messages, for example:

• device identifier
• timestamp
• measurement value

Data sourceData sourceA data-related element that represents the origin of data. It defines how data is connected, accessed, and ingested into the Platform, such as an external database or sensor network. Create an MQTT Data sourceData sourceA data-related element that represents the origin of data. It defines how data is connected, accessed, and ingested into the Platform, such as an external database or sensor network. for continuous measurement data.

Example Base information:

• Name: Smart Meter MQTT measurement data
• Description: Measurement data sourceData sourceA data-related element that represents the origin of data. It defines how data is connected, accessed, and ingested into the Platform, such as an external database or sensor network. for incoming Smart Meter energy values

In the Connector tab, select MQTT and enter the required configuration.

Example:

• Broker: mqtt.example.com
• Topic: taf10/sensors

Assign the Data structureData structureA versioned data-related element that defines the schema and organization of data. It specifies how data is structured and interpreted within the Platform. that describes the incoming MQTT messages to the Data sourceData sourceA data-related element that represents the origin of data. It defines how data is connected, accessed, and ingested into the Platform, such as an external database or sensor network..

The measurement data will later be transformed and stored as Observations in a Sensor Data storageData storageA Pipeline node that persists data inside the Platform so that it can be provided through an API. The Platform offers the Sensor Data storage, which stores sensor data for the SensorThings API, and the Geospatial Data storage, which stores geospatial data in tables for WFS/WMS APIs. The Data storages used in the Pipelines of a Dataset determine which API types the Dataset can offer. and linked to the corresponding Datastreams.

Outcome

You now have the Data sourcesData sourceA data-related element that represents the origin of data. It defines how data is connected, accessed, and ingested into the Platform, such as an external database or sensor network. and Data structuresData structureA versioned data-related element that defines the schema and organization of data. It specifies how data is structured and interpreted within the Platform. for both master data and measurement data.

Next

You have connected your Data sourcesData sourceA data-related element that represents the origin of data. It defines how data is connected, accessed, and ingested into the Platform, such as an external database or sensor network. and prepared them for further processing. In the next guide, you will:

• build pipelines
• transform your data
• prepare it for consumption

→ Continue with Persist, transform & provide Data