Enabling MQTT with TLS Encryption

Using secure MQTT with CIVITAS/CORE

Enabling the TLS feature of APISIX for MQTT is a half automated feature in CIVITAS/CORE. This means, at least one step has to be done manually.

There are three steps, that have to be executed, to get a running configuration.

1. Configure MQTT Settings in the inventory, to use TLS
2. Run the core platform installation
3. Run the separate playbook to gather a Let's Encrypt certificate for the MQTT port

These three steps are described in detail in the following chapters. But first the rough architecture of the MQTT integration is explained.

Overview

To allow MQTT as part of the provided platform interfaces, the CIVITAS/CORE platform routes the MQTT Requests through the API Management APISIX. For APISIX, the TCP stream proxy is used. This means, APISIX opens an additional port (configured in the inventory) and accepts general TCP Connections on this port. Details how this can be configured directly in APISIX can be found here but this is included in the platform configuration, when you activate the MQTT Support in the inventory.

Next, we have to create a stream route using the mqtt_proxy plugin. But because we do not want to provide unauthenticated access to the MQTT port, provide a new mqtt-auth-proxy plugin. This plugin enhances the standard mqtt-plugin with a Keycloak integration. The needed settings for communicating with Keycloak are set by the deployment script. Unauthenticated access is not supported.

APISIX operates only as a gateway for MQTT. All requests are forwarded to the FROST-Servers' MQTT broker. So the protocol on MQTT is SensorThingsAPI in this case.

Next we have to expose the MQTT port to the internet. To archive this, the CIVITAS/CORE platform creates an additional service on the kubernetes cluster, that can be connected to the internet via a LoadBalancer or NodePortConnection. Any other way is suitable, too. This can be configured in the Inventory.

Due to the fact that authentication in MQTT is done by providing username and password as part of the connect message, we need a secure way to transfer this from the client to the APISIX Gateway. So the next step is to enable TLS for MQTT. Here you have different options, from which we provide a half automated script for one of them.

The options are:

• Provide self-signed certificates
• Use a Let's Encrypt certificate
• Use any other officially signed certificate

In all cases, the certificate must match with the SNI of the MQTT service, which can be configured in the inventory.

Configuring secure MQTT with CIVITAS/CORE

The following guide describes the process to use Let's Encrypt certificates and configure the platform by the half automated script. The last step has to be replaced for the other options.

Configure MQTT Settings in the inventory to use TLS

In the APISIX Part of the inventory the following MQTT-specific settings are available.

            stream: # enable for MQTT Integration
              enable: true
              mqtt_service_type: "LoadBalancer" # "ClusterIP", "NodePort" or "LoadBalancer"
              mqtt_service_port: 1883
              enable_tls_for_mqtt: true 
              mqtt_service_sni: "mqtt.{{ DOMAIN }}"
              dns01_wait_minutes: 5 

The enable option allows to enable or disable the full external support of APISIX for MQTT. Set it to true to start with the configuration.

Next you must decide on the service type for the exposed MQTT Service described above. The following options are available:

• ClusterIP
• NodePort
• LoadBalancer

In each case an external available public IP must be allocated. This typically happens in different ways, depending on your hosters infrastructure. You have to lookup the IP after the deployment over your hosters management interface.

As an example: Usage of hetzner.de as cloud hoster and configuration of the service as type LoadBalancer: When the deployment is done, a new loadbalancer entry in the cloud management UI is shown. The Public IP of this LoadBalancer is needed

With the parameter mqtt_service_port you can decide on the exposed port of the MQTT service. The default is 1883. For secure MQTT also 8883 is commonly used. But at the end it is your choice.

Until here, you would expose MQTT without TLS. To change this, you have to set enable_tls_for_mqtt to true.

Finally, you have to configure which hostname is used for TLS. This host name is later used to request the certificate from Let's Encrypt and to identify the right certificate for the TLS route in APISIX.

The parameter mqtt_service_sni is defaulted to mqtt.{{ DOMAIN }}.

The parameter dns01_wait_minutes is defaulted to 5 and defines the time, the certificate playbook will wait, before it tries to request the created certificate after setting up the request for it.

IMPORTANT: This DNS lookup for this hostname can be configured after the next step of the deployment. For the deployment the lookup is not actively used - it is only used to configure the right settings for later. The IP Address is assigned to a LoadBalancer after the next step.

Run the core platform installation

You can run the deployment scripts as described in the Installation Guide

The installation can be done with the initial deployment or with later updates.

Run the Let's Encrypt certificate generation playbook

We provide the mqtt_tls_configuration.yml playbook in the CIVITAS/CORE repository in the tools folder.

The playbook creates a certificate request for the above configured MQTT Hostname (SNI) based in the DNS-01 Challenge. Details about that can be found here

The scripts create the Let's Encrypt requests and needed key automatically. After sending the request to Let's Encrypt it waits for the defined time from the inventory. After this time, the script checks for the ready certificates again and stores them locally.

Finally this certificate and the needed key are stored and after that, pushed to the SSL Section of the APISIX Installation.

To run the playbook simply execute the following command:

ansible-playbook -i <your_inventory> mqtt_tls_configuration.yml

The playbook will stop running and wait the defined time with the following output:

TASK [print challenge] ***********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": {
        "_acme-challenge.<yourdomain>": [
            "EU12XoBitgBXrsNERi-uv-U2C9PwxjFBQAAA&E5y8ZI"
        ]
    }
}

TASK [Pause for 5 minutes to configure DNS] **************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
Pausing for 300 seconds
(ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort)

The output before the wait statements shows the needed data for the DNS-01 challenge. Depending on your way to configure your DNS you now have to create a DNS Record like:

_acme-challenge.<yourdomain> TXT "EU12XoBitgBXrsNERi-uv-U2C9PwxjFBQDAAA&E5y8ZI"

After that, the script will proceed and configure APISIX with the new certificate.