Showcasing some embedded device management functions in one use case

Introduction

This demo use case shows an example of how qbee handles remote device access, configuration management for different Linux settings, OTA software updates and complete application software installations.

The agenda is as follows:

  • Installation & bootstrapping of qbee
  • Configuration examples
    • NTP
    • User password
    • ssh keys
    • firewall
  • Full OTA software and package update
  • remote ssh access / remote web server access
  • MQTT setup using Mosquitto & templating
  • Node-RED setup & templating, remote access to Node-RED on an edge device
  • Bonus “Minecraft Server” (showing Java software installation example)

We created a video to accompy this document.

For the demo we use a SYS TEC CTR-700 but this can be just as well deployed to any Raspberry Pi, odroid, Beagleboard or other Linux device.

qbee-systec-ctr-700

First a user account is created on qbee.io. The bootstrap keys are created (in this case for a specific group since the device should auto bootstrap to the group "Systec". Then the qbee agent is installed and bootstrapped.

Installation and bootstrapping

This process installs the qbee agent on the device. The bootstrap keys allows qbee to assign the device to the correct user and the correct position in the device tree. Multiple bootstrap keys can be used for different groups/customers and these keys can easily be revoked at any time preventing further devices to bootstrap with them.

This will look as follows:

qbee-install-bootstrap

The result is that the remote device comes online in qbee and it is possible to see device information and embedded performance metrics.

qbee-demo-device-screen

qbee-demo-metrics

Now the devices will inherit configuration from the group (in this case ssh keys and user passwords) and new configuration can be set. All the configuration that is configured in this demo is applied to the group Systec. This will enable all devices (or groups) that reside in this tree branch to inherit this configuration. Also all devices that will be bootstrapped later into that group will automatically inherit and deploy this configuration.

Initial configuration

In this demo 4 configurations are applied. They are the following:

Next time the agent connects it will download the new configuration file and converge to the new configuration. This configuration creates an audit entry and a log file entry allowing to see who did which configuration as well as for which devices it was applied. In this case some Norwegian NTP servers were defined, some ssh keys and a new user root password. In addition the firewall is configured to drop all. With qbee no ports need to be open but it is still possible to get full secure remote access through the built in VPN.

qbee-demo-audit

qbee-demo-logs

The next task includes a full OTA debian package update. Through package management qbee can select if all available packages should be updated or only specific ones. qbee can fetch packages from any repository or from the internal file manager.

Full OTA package update

For this demo a full over-the-air system update is selected. In the package management configuration it can be selected if all or specific packages should be updated. A reboot option after update is available.

Update with pre-condition

There is a field called "pre-condition". This allows to check if a condition is true /bin/true or if a script exits with "0". This allows to build very complex logic. From checking for certain update times to checking voltages being applied to any of the ADCs of the device.

Having defined and commited the "update all" policy the device updates 235 packages. Which packages will be updated can be seen in the device software inventory tab.

qbee-demo-sw-inventory

Remote ssh access

With qbee it is possible to access the remote device behind any firewall or NAT. This can be done from the device tab or from the "remote console" screen. The first opens a new window while in the later it is possible to open it in screen or as a new window. qbee offers also the option to access the remote device from the local machine through the local terminal.

The next topic in this use case example is about installing an MQTT publisher (Mosquitto) and sending MQTT data to a MQTT receiver. For this a script is used that measures the CPU temperature and sends this to an MQTT service. Most embedded devices such as Raspberry Pis or similar ones have an internal temperature reading. This part also introduces the possibility to template scripts or configuration files.

Send CPU temperature data via MQTT

Mosquitto and the Mosquitto-clients package is installed automatically and the service is permanently started and monitored. This is done with "Configure -> Software management".

qbee-demo-mosquitto

Distribute the script that measures CPU temperature and do the templating

"Configure -> File distribution" is used to play out the script below from the file manager. But instead of playing it out as a script with .sh ending a .tmpl ending is selected indicating that this is a template file that first needs to be expanded with the correct key-value pairs. qbee allows to abstract and expose anything within a script or configuration file as key-value pairs by using the so-called mustache notation - two double brackets as displayed here {{key}}. This can be exposed through file distribution by checking the "Template" check box. Then any number of key-value parameters can be defined. When the script is first distributed the place holders will be written with the correct values and the "command to run" will be invoked. The same happens any time these parameters will be changed and committed.

qbee-demo-cpu_script

Explaining the script

First the script checks if a CPU_temp process is already running. If so it gets killed. Then the temperature is received and some checks apply (NaN...). The temperature is then handed to the mosquitto publisher. The interesting part here is that the MQTT topic is defined by the "MQTT-topic" value "ctr-700/temp" and the frequency of measurement by the sleep "delay" with value "5" seconds.

cpu_temp.sh
#!/usr/bin/env bash

PIDFILE="/var/run/$(basename $0).pid"

# Kill any previously running process
if [[ -f $PIDFILE ]]; then
  kill $(cat $PIDFILE) 2> /dev/null
fi

echo $$ > $PIDFILE

while true; do
  t=$(cat /sys/class/thermal/thermal_zone0/temp)
  re='^[0-9]+$'
  if ! [[ $t =~ $re ]]; then
    echo "error: Not a number" >&2
  else
    t=$((($t + 1) / 1000))
    mosquitto_pub -h MY_MQTT_SERVER.com -p 99999 -u MQTT_user -P MQTT_password -t {{MQTT-topic}} -m "$t °C"
  fi
  sleep {{delay}}
done> /dev/null 2>&1 &

Explaining the command to run

The script is defined as a bash script and started by calling bash /usr/local/bin/cpu_temp.sh > /dev/null 2>&1 &. This runs the script in the background and the appendix > /dev/null 2>&1 & pipes bash output to /dev/null. This command will run any time the file is changed or a key-value pair changes.

The output is send to CloudMQTT in order to have a simple means of visualizing the data from the device. The correct topic is received and the data comes in with 5 seconds intervals. In the video this is changed and both topic and interval change.

qbee-demo-cloudmqtt

Importing configuration with qbee

It is possible to import and export configuration as json. Below is the json that can be imported in file distribution. The script needs to be uploaded in file manager with cpu_temp.tmpl to the root level.

cpu_temp.tmpl
    {
      "enabled": true,
      "files": [
        {
          "templates": [
            {
              "source": "/cpu_temp.tmpl",
              "destination": "/usr/local/bin/cpu_temp.sh",
              "is_template": true
            }
          ],
          "parameters": [
            {
              "key": "MQTT-topic",
              "value": "ctr-700/temperature"
            },
            {
              "key": "delay",
              "value": "10"
            }
          ],
          "command": "bash /usr/local/bin/cpu_temp.sh > /dev/null 2>&1 &",
          "inheritance_scope": "full"
        }
    ]
    }

Defining a process watch to start and keep the process running

As mentioned qbee will restart the script through file distribution any time there is a change. After a reboot or when the process should die it will not be restarted. Therefore the process watch configuration is very useful. qbee will check at every run if the process is present (or basent) and run the command. Below is the json for import to watch the CPU_temp script.

{
  "enabled": true,
  "processes": [
    {
      "name": "/usr/local/bin/cpu_temp.sh",
      "policy": "Present",
      "command": "bash /usr/local/bin/cpu_temp.sh > /dev/null 2>&1 &"
    }
  ]
}

Node Red setup and templating

Both a standard Raspbian Raspberry Pi image as well as the CTR-700 have a Node Red installation already pre-installed. In this example we do the following:

  • Distribute Node Red settings & credentials
  • Distribute a flow file with templating
  • Create a system service access script
  • Do a remote web server login to the remote Node Red

Below is the full json for the file distribution containing both the previous exampes (with changed ke-values) as well as the Node-RED ones. By now it should be easy to read these json files. Alternatively they can be imported to any device through file distribution without saving to get an UI view. All this is also shown in the video about this example.

file distribution json
{
  "enabled": true,
  "files": [
    {
      "templates": [
        {
          "source": "/cpu_temp.tmpl",
          "destination": "/usr/local/bin/cpu_temp.sh",
          "is_template": true
        }
      ],
      "parameters": [
        {
          "key": "MQTT-topic",
          "value": "ctr-700/temperature"
        },
        {
          "key": "delay",
          "value": "10"
        }
      ],
      "command": "bash /usr/local/bin/cpu_temp.sh > /dev/null 2>&1 &",
      "inheritance_scope": "full"
    },
    {
      "templates": [
        {
          "source": "/node-red/settings.js",
          "destination": "/root/.node-red/settings.js",
          "is_template": false
        },
        {
          "source": "/node-red/flows_cred.json",
          "destination": "/root/.node-red/flows_cred.json",
          "is_template": false
        },
        {
          "source": "/node-red/flows.tmpl",
          "destination": "/root/.node-red/flows.json",
          "is_template": true
        }
      ],
      "parameters": [
        {
          "key": "weather",
          "value": "very nice"
        },
        {
          "key": "temp",
          "value": "30"
        }
      ],
      "command": "systemctl restart node-red",
      "inheritance_scope": "full"
    },
    {
      "templates": [
        {
          "source": "/node-red-enable.tmpl",
          "destination": "/usr/local/bin/node-red-enable.sh",
          "is_template": true
        }
      ],
      "parameters": [
        {
          "key": "enable/disable",
          "value": "enable"
        }
      ],
      "command": "bash /usr/local/bin/node-red-enable.sh > /dev/null 2>&1 &",
      "inheritance_scope": "full"
    }
  ]
}

Node Red credentials handling

Node Red is a fantastic tool that has reached production quality with an awesome amount of integrations. But it is very important that the credentials get handled correctly, otherwise it is very difficult to use it in an automated fashion. There are two places Node Red keeps the password to decode credentials:

  • in settings.js (credentialSecret: "a-secret-key")
  • in .config.json (_credentialSecret, but this is the secret hash)

In a new installation of Node Red that never saved credentials through the flow editor the _credentialSecret is not present. Then Node Red reads the "a-secret-key" from the settings file and uses that. No problem for automated deployment of flows and secrets. But if the _credentialSecret exists in .config.json and it is not the hash from the current settings.js but a random hash or from a previous settings.js then this will cause a problem. In this case it needs to be removed in the .config.json or the correct .config.json needs to be played out as well.

In short: If it is a clean Node-RED installation the following mechanism will work out of the box. Otherwise a strategy for the .config.json needs to be developed. Often the .config.json file has device specific information (specific nodes) and therefore it is often preferable to use the devices' local version.

Automated Node Red deployments get developed by starting on one machine. These are the actions that need to be taken (and please make sure that you follow the credential handling information above):

  • edit the settings.js. Two paramteres need to be set. First, uncomment and define credentialSecret: "a-secret-key" then define that the flow file name is defined as such flowFile: 'flows.json', instead of inserting the machine name.
  • restart Node Red
  • create a flow file and deploy it (thus creating an updated flows_cred.json file with the encrypted MQTT server credentials )
  • exit Node Red

For this simple demo we create a weather station. This flow json file can be copied into Node Red. Only the MQTT part needs to be adjusted to reflect your server. This is not a flows.json but a flows.tmpl and two key-value templating actions are performed with mustache notation. These the double brackets {{weather}} and {{temp}} for payload. These will later be replaced by qbee with values.

flows.tmpl
[
    {
        "id": "5d23c98c.7b266",
        "type": "tab",
        "label": "Flow 1",
        "disabled": false,
        "info": ""
    },
    {
        "id": "b14795ea.ec2168",
        "type": "inject",
        "z": "5d23c98c.7b266",
        "name": "weather",
        "topic": "weather",
        "payload": "{{weather}}",
        "payloadType": "str",
        "repeat": "10",
        "crontab": "",
        "once": true,
        "onceDelay": 0.1,
        "x": 171.5,
        "y": 175,
        "wires": [
            [
                "4aa7eeb3.7fa398"
            ]
        ]
    },
    {
        "id": "4aa7eeb3.7fa398",
        "type": "function",
        "z": "5d23c98c.7b266",
        "name": "weather station",
        "func": "var payload = msg.payload;\n\nmsg.payload=\"The weather is \"+msg.payload;\n\nreturn msg;",
        "outputs": 1,
        "noerr": 0,
        "x": 373.5,
        "y": 175,
        "wires": [
            [
                "30800128.36b06e",
                "6aaa640f.8634cc"
            ]
        ]
    },
    {
        "id": "30800128.36b06e",
        "type": "debug",
        "z": "5d23c98c.7b266",
        "name": "",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "false",
        "x": 586.5,
        "y": 174,
        "wires": []
    },
    {
        "id": "5d6ae0e7.c3b5e8",
        "type": "inject",
        "z": "5d23c98c.7b266",
        "name": "temp",
        "topic": "temp",
        "payload": "{{temp}}",
        "payloadType": "num",
        "repeat": "10",
        "crontab": "",
        "once": true,
        "onceDelay": 0.1,
        "x": 160,
        "y": 256,
        "wires": [
            [
                "2f10cd67.eced22"
            ]
        ]
    },
    {
        "id": "2f10cd67.eced22",
        "type": "function",
        "z": "5d23c98c.7b266",
        "name": "temp station",
        "func": "var payload = msg.payload;\n\nmsg.payload=\"The temperature is \"+msg.payload;\n\nreturn msg;",
        "outputs": 1,
        "noerr": 0,
        "x": 365,
        "y": 255,
        "wires": [
            [
                "37951f74.afcd6",
                "6aaa640f.8634cc"
            ]
        ]
    },
    {
        "id": "37951f74.afcd6",
        "type": "debug",
        "z": "5d23c98c.7b266",
        "name": "",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "false",
        "x": 591,
        "y": 261,
        "wires": []
    },
    {
        "id": "6aaa640f.8634cc",
        "type": "mqtt out",
        "z": "5d23c98c.7b266",
        "name": "cloudmqtt",
        "topic": "weather",
        "qos": "",
        "retain": "",
        "broker": "31539024.dca0f8",
        "x": 608.5,
        "y": 350,
        "wires": []
    },
    {
        "id": "31539024.dca0f8",
        "type": "mqtt-broker",
        "z": "",
        "name": "CloudMQTT",
        "broker": "farmer.cloudmqtt.com",
        "port": "13179",
        "clientid": "",
        "usetls": false,
        "compatmode": true,
        "keepalive": "60",
        "cleansession": true,
        "birthTopic": "",
        "birthQos": "0",
        "birthRetain": "false",
        "birthPayload": "",
        "closeTopic": "",
        "closeQos": "0",
        "closeRetain": "false",
        "closePayload": "",
        "willTopic": "",
        "willQos": "0",
        "willRetain": "false",
        "willPayload": ""
    }
]

This flow file looks as follows. Two injector nodes inject the variable "weather" and "temperature". The function block turns it into a sentence and then it is published to MQTT.

qbee-demo-nodered-flow

Tip

The mustache notation with the curly brackets can be done in the json file or even in the editor. But then, obviously, the flow cannot run anymore.

Now the flows.json, flows_cred.json and the settings.js files can be uploaded via the file manager into the folder "node-red". Then the file distributions moves these into the correct remote .node-red folder. For flows.json templating is enabled and the "weather" and "temperature" is defined as key-value parameters. In order to apply the parameters Node Red needs to be restarted. This is done by invoking the service restart command systemctl restart node-red through command to run. This will also run any time any value changes.

Restart the node-red service any time a change occurs

Whenever a flow file changes the service needs to be restarted. This happens automatically when defined in "command to run"

In order to make the Node Red demo complete there is one more example. This makes it possible to enable/disable Node Red on a defined group of devices.

A simple means of enabling/disabling services on group or device level

Sometimes it is important that services can be turned on or off for a large froup of devices. This can be achieved by running a simple script and templating the respective condition.

The scrip will look like this and is called node-red-enable.tmpl. It needs to reside in the root folder (or the file distribution path can be changed). The variable that is exposed here is "enable/disable". Obviously this can be set to enable, disable, start or stop

node-red-enable.tmpl

#!/usr/bin/env bash

systemctl {{enable/disable}} node-red

Enabling this for group "Systec" will enable and autostart the service on all devices in Systec or subgroups from Systec. Below the MQTT messages from Node Red and the CPU temperature script are displayed. Note how the topic for the script changed as well as the update rate.

qbee-demo-multiple-mqtt-messages

Remote access to Node Red

qbee.io provides a desktop tool called "qbee-connect" which is available for all major OS platforms. This allows full secure remote access through the build in VPN service. Remote ports can be mapped to local host ports and accessed with services such as https, http, ssh, scp, VNC and many more.

The following screen shows how the remote Node Red port 1880 is mapped to a local port. Then the Node Red UI can be accessed through "localhost:50218" from the remote machine across firewalls.

qbee-demo-qbee-connect

qbee-demo-node-red-access

Remote ssh access with local terminal

It is possible to use the OS local terminal and access the remote device through ssh or even use remote file copy with scp. The ssh command can be copied from qbee-connect. Below you see the ssh command and an scp example:

  • ssh -p 50217 root@localhost
  • scp -P 50217 root@localhost:/var/log/syslog syslog_ctr-700

qbee-demo-local-terminal

Playing out a Mincecraft Server and playing remotely

This excercise gives an example how to play out, configure and use a remote Java app with qbee. The intention is to give ideas and inspiration what else can be done with qbee.

The base configuration is again a file distribution. The following file distribution is used (can be used on top of the two other ones). Here a minecraft server.jar java file is played out. In addition the server.properties file is distributed. This file has been slightly modified to reduce the overall Minecraft world size and complexity and to limit the amount of players. The next file is a service description that starts Minecraft as a service. This is shown below. The "command to run" systemctl daemon-reload reloads the daemon thus picking up the minecraft.service and makes it available. The a chained systemctl start minecraft starts it.

File distribution has a file order

In this case it is not only important what is distributed but also in which order. First the server file is delivered. Then the property file and the service file is distributed and minecraft is started. The first time it writes its own server.properties file, thus overwriting ours. It also extracts an eula.txt file that says eula=false. Then the manually created EULA file with eula=true overwrites the previous. In additon with the next qbee agent run the properties.service does not match with the one from file manager. At this point it gets distributed again (overwriting the one that came with the server.jar) and the "command to run" runs again restarting Minecraft. Now all files are exactly the way they are supposed to be and the Minecraft world gets created.

file distribution exmample
{
  "enabled": true,
  "files": [
    {
      "templates": [
        {
          "source": "/minecraft/server.jar",
          "destination": "/usr/local/bin/minecraft/server.jar",
          "is_template": false
        },
        {
          "source": "/minecraft/server.properties",
          "destination": "/usr/local/bin/minecraft/server.properties",
          "is_template": false
        },
        {
          "source": "/minecraft/minecraft.service",
          "destination": "/etc/systemd/system/minecraft.service",
          "is_template": false
        }
      ],
      "command": "systemctl daemon-reload && systemctl start minecraft",
      "inheritance_scope": "full"
    },
    {
      "templates": [
        {
          "source": "/minecraft/eula.txt",
          "destination": "/usr/local/bin/minecraft/eula.txt",
          "is_template": false
        }
      ],
      "inheritance_scope": "full"
    }
  ]
}

Here is the minecraft.service service description. This is extremely basic, not complete and should not be used like this. But this illustrates the way it is done:

minecraft.service
[Unit]
Description=Minecraft Server
After=network.target

[Service]
Type=simple
User=root
Nice=1
KillMode=none
SuccessExitStatus=0 1

WorkingDirectory=/usr/local/bin/minecraft
ExecStart=/usr/bin/java -Xms512M -Xmx1008M -jar server.jar nogui


[Install]
WantedBy=multi-user.target

Making the Minecraft server autostart

Again, with process watch the service is monitored and restarted. Add the following to the previous process watch:

{
  "name": "/usr/bin/java",
  "policy": "Present",
  "command": "systemctl start minecraft"
}
This time the process is identified as a java process matched by the expression "/usr/bin/java".

Get a coffee

Running a minecraft server on a small device with limited memory and Armv7 CPU like a Raspberry Pi or CTR-700 takes a while to calculate the world. So it can take up to one hour before the server accepts connections. But then it works fairly responsive.

When everything is initialized it is possible to connect to the remote device through the qbee VPN. On the device all ports are closed by the firewall as configured in the beginning of this tutorial.

Accessing the minecraft server over the qbee VPN

The server is configured to serve Minecraft on the standard port 25565. No authentification has been configured in the properties file. qbee connect is used to open a secure connection on port 25565 and map it to a localhost port through its internal VPN. In this case this is port 56337. Now the local Minecraft application needs to connect to the remote server localhost:56337.

qbee-demo-connect-minecraft

qbee-demo-minecraft-connect

qbee-demo-minecraft-gameplay

This demo was done on a ctr-700. It is just as possible to run all this on a Raspberry Pi or similar devices. The main difference between the ctr-700 and the Raspberry Pi is that the ctr-700 has an internal flash and does not run off an SD-card.

For industrial use cases please do not run off an SD-card, always use eMMC

While the Raspberry Pi is a great device we strongly advise against running these in production from SD-card. Please use an industrial device (there are industrial Raspberry Pi devices with eMMC flash available) or an industry controller like the Systec CTR-700 that also has internal flash in form of an eMMC.

Please do not hesitate to reach out for questions

Thanks for your attention. For further questions or more details please send us a mail or contact us through our chat solution. What can we do for you? We are always looking out for interesting use cases.