FROST MQTT & WebSocket (WSS) – Access via APISIX

This document describes how to access the FROST MQTT backend via APISIX in Kubernetes.

Summary (tl;dr)

• Port 8883 (MQTTS): Authentication/authorization possible (MQTT clients, e.g., IoT)
• Port 9876 (WSS): Public only, no authentication (e.g., browser, dashboards)
• All settings are controlled via the inventory (see variables below)
• DNS of mqtt-Subdomain must point to the APISIX Data Plane LoadBalancer IP
• Certificate is managed automatically by the APISIX Ingress Controller

Using secure MQTT with CIVITAS/CORE

Enabling the TLS feature of APISIX for MQTT is an automated feature in CIVITAS/CORE. There are three steps that have to be executed, to get a running configuration.

1. Configure MQTT Settings in the inventory
2. Run the core platform installation
3. Make sure that the subdomain mqtt.{{ DOMAIN }} is routed to the APISIX ingress controller

These three steps are described in detail in the following. But first the rough architecture of the MQTT integration is explained.

Overview

To allow MQTT as part of the provided platform interfaces, the CIVITAS/CORE platform routes the MQTT Requests through the APISIX API Gateway. For APISIX, the TCP stream proxy is used. This means, APISIX opens an additional port (configured in the inventory) and accepts general TCP Connections on this port. Details on how this can be configured directly in APISIX can be found here but this is included in the platform configuration, when you activate the MQTT Support in the inventory.

Next, we have to create a stream route using the mqtt_proxy plugin. But because we do not want to provide unauthenticated access to the MQTT port, CIVITAS/CORE provides a new mqtt-auth-proxy plugin. This plugin enhances the standard mqtt-plugin with a Keycloak integration. The needed settings for communicating with Keycloak are set by the deployment script. Unauthenticated access is still allowed (can be disabled) to access public dataspaces.

APISIX operates only as a gateway for MQTT. All requests are forwarded to the FROST-Servers' MQTT broker. So the protocol on MQTT is SensorThings API in this case.

Next we have to expose the MQTT port to the internet. To archive this, the CIVITAS/CORE platform allows to deploy the APISIX gatway as LoadBalancer service on the Kubernetes cluster. Other methods can also be used. This can be configured in the Inventory.

Due to the fact that authentication in MQTT is done by providing username and password as part of the connect message, we need a secure way to transfer this from the client to the APISIX Gateway. So the next step is to enable TLS for MQTT. The CIVITAS/CORE platform uses Let's Encrypt certificates for this purpose.

1. Classic Network Flow: MQTTS (Port 8883)

Authentication/authorization possible via the mqtt-auth-proxy plugin.

(MQTT Client)
   |
   |  (MQTTS, port 8883)
   v
+-------------------------+
|    APISIX Data Plane    |
|   (MQTTS Stream Route)  |
+-------------------------+
            |
            v
   +------------------------+
   |  FROST MQTT            |
   |  (e.g. Mosquitto)      |
   |  (Port: 1883, no TLS)  |
   +------------------------+

2. WebSocket Network Flow: WSS (Port 9876)

Public, without authentication/authorization.

(Browser/Client)
   |
   |  (WSS, port 9876, path: /mqtt)
   v
+-------------------------+
|    APISIX Data Plane    |
|    (HTTP Route)         |
+-------------------------+
            |
            v
   +--------------------+
   |   FROST MQTT       |
   | (WebSocket module) |
   | (Port: 9876)       |
   +--------------------+

Prerequisite: Inventory Configuration

The following variables must be set correctly in your inventory:

inv_access:
  apisix:
    ingress_controller:
      enable: true                   # Enable APISIX Ingress Controller
    data_plane:
      enable: true                   # Enable APISIX Data Plane
      services_type: "LoadBalancer"  # Make Data Plane externally reachable
      mqtt_service_port: 8883        # MQTTS (port 8883, default)
      mqtt_websocket_port: 9876      # WSS  (port 9876, default)
inv_cm:
  frost:
    mqtt:
      enable: true                   # Enable FROST MQTT functionality

Note: To completely disable one of the ports, set its value to 0:

• inv_access.apisix.data_plane.mqtt_service_port = 0
• inv_access.apisix.data_plane.mqtt_websocket_port = 0

DNS and Certificate Management

• The APISIX Data Plane receives its own (public) IP address as soon as the Service is set as LoadBalancer.
• The subdomain mqtt.{{ DOMAIN }} must point to this IP (DNS A record).
• The Let's Encrypt TLS certificate for mqtt.{{ DOMAIN }} is automatically obtained and managed by the APISIX Ingress Controller.

Special Notes & Recommendations

• Authentication/Authorization

  • WSS endpoint (9876): Currently without authentication! Only public FROST dataspaces are accessible.

    Attention: Access is completely open via WSS.

  • MQTTS endpoint (8883): Authentication via APISIX plugin mqtt-auth-proxy is possible (e.g., with Keycloak). By default, anonymous connections are also allowed!

    • To block anonymous access, set allow_anonymous to false in the APISIX StreamRoute for MQTT.

Running the installation

You can run the deployment scripts as described in the Installation Guide

The installation can be done with the initial deployment or with later updates, e.g. via

cc_cli exec -i <your_inventory_file> -t acc_keycloak,acc_mqtt,acc_apis