Migrate from CIVITAS/CORE v1

Who is this guide for?

Role: Data Architect, Data Steward

You need to collaborate with: the operator of your v1 Platform or source system, Data Owner, Data Gatekeeper

Goal: You want to move existing data from CIVITAS/CORE v1, or from another system, into CIVITAS/CORE v2.

Before you start

CIVITAS/CORE v2 has no separate migration tool. You migrate data with the same building blocks you use for any other data integration: Data sourcesData sourceA data-related element that represents the origin of data. It defines how data is connected, accessed, and ingested into the Platform, such as an external database or sensor network., Pipelines, Mappings and Storage nodes.

This guide gives you an overview of the approach and points you to the guides that explain each step in detail.

Understand the migration path

Path	Source	When to use
SQL	A table in a migration database (PostgreSQL)	Master data and moderate data volumes; the default path
MQTT	Messages published to an MQTT broker topic	Large data volumes, e.g. long measurement histories, that you want to deliver in packages

Step 1: Provide the source data

The SQL connector reads data from a PostgreSQL table. Copy the data you want to migrate into a separate migration or transfer database and connect the Platform to this copy.

For CIVITAS/CORE v1, ask the operator of your v1 Platform for the copy, e.g. as a database dump. Data from other systems is exported into a table of the migration database.

Do not connect databases of running systems

Reading large amounts of data puts load on a production system, and its data may change while you migrate. A copy gives you a stable state that you can migrate and check without affecting the running system.

Prepare the data in the migration database

Use views to select and combine the columns you need — the simpler the source table, the simpler the Mapping. Grant the Platform read-only access.

Step 2: Connect the source

Describe the source table with a Data structure and connect the migration database as a Data sourceData sourceA data-related element that represents the origin of data. It defines how data is connected, accessed, and ingested into the Platform, such as an external database or sensor network. with the SQL connector.

→ Connect external Data

Step 3: Map and store the data

Create a DatasetDatasetA data-related element that contains processed data and makes it available for consumption. A Dataset is populated via Pipelines and carries Metadata and access permissions. with a Pipeline that maps the data into the target structure and stores it. Choose the target by the kind of data:

Data	Target	Learn more
Sensor master data and measurements	Sensor Data Storage	Work with sensor Data
Geospatial data, e.g. points, lines, polygons	Geospatial Data Storage	Work with geospatial Data

→ Persist, transform & provide Data

Migrating sensor data

Migrate the master data first, then the measurements. Map the identifiers of your v1 data to the references of the entities, so a second run does not create duplicates. → Map the references

Step 4: Migrate large data volumes via MQTT

For large data volumes, e.g. measurement histories over several years, deliver the data in packages via an MQTT broker instead. Connect the topic as a Data sourceData sourceA data-related element that represents the origin of data. It defines how data is connected, accessed, and ingested into the Platform, such as an external database or sensor network. with the MQTT connector, then map and store the data as described in Step 3.

→ Connect external Data

Step 5: Release and check the result

Have the DatasetDatasetA data-related element that contains processed data and makes it available for consumption. A Dataset is populated via Pipelines and carries Metadata and access permissions. released and check the migrated data through its APIs.

→ Release Data

→ Consume Data

Summary

You migrate data into CIVITAS/CORE v2 with the standard tools of the Platform:

• copy the existing data into a migration database
• connect it with the SQL connector
• map it and store it in the Sensor Data Storage or the Geospatial Data Storage
• use MQTT to deliver large data volumes in packages
• release the DatasetDatasetA data-related element that contains processed data and makes it available for consumption. A Dataset is populated via Pipelines and carries Metadata and access permissions. and check the result through its APIs