Configure

Configure

This is one of the key tasks of qbee.io. Here the configuration for your devices is defined. In order to understand how this works we need to explain the way qbee.io works:

Configuration Management with qbee.io

qbee.io is a state based configuration management tool. The local qbee agent on the device queries the centralized server with a set interval and checks if new configuration information is available. If not it just checks if it still complies to the current state and goes to sleep again. Otherwise it downloads the new configuration and then converges towards the new configuration. This concept has many advantages as the agent can operate and maintain configuration even if the network is down. Also devices that come later into a group (spare parts, replacements) will always get the latest configuration with latest ssh keys and user definitions. Some practical implications are:

  • configuration is only becoming valid if you commit it

  • it will take at least until the next connect interval of the agent until it receives the configuration

  • the qbee agent will report success or an error if it cannot reach its state

  • the qbee agent maintains its defined state. If you manage password or firewall settings and you change it manually qbee will revert it in the next run (and send you a log message about this). This is also true for files that you play out.

  • in very few cases the agent cannot converge to its defined state. Then an error message is displayed (for example you play out a file where you do not have correct access rights, or a process should be started that does not exist)

In the configure menu the top selector defines for which device or group of devices configuration is applied. "All devices" will enable this for the main branch of the tree. All subsequent groups and devices that do not have their inheritance disabled will inherit this configuration. You can do multiple configuration changes at once, but it is recommended to always do one after the other.

qbee-configuration-inheritance

In the image above the group "hardware devices" has inherited its settings from group "All devices". The menus are greyed out but show the configuration. By pressing the orange "Edit settings" button this inheritance can be broken. Then hardware devices (and all its group below in the branch of the tree) will inherit the new configuration. Breaking inheritance and defining new settings will cause a green "save" button to appear. New configurations need to be saved. Then an information box pops up showing all impacted groups and devices.

qbee-pop-up

Here you can cancel or save. Then the configuration is written. At this point in time the configuration is not yet committed.

Tip

Any time new configuration is committed the system generates a list of all devices that are impacted by this. Please check if this is as expected before you continue.

qbee-commit-configuration

First when you press the orange commit button the configuration change gets active on the server and can be picked up by the relevant devices. A pop-up will allow you to add a short comment to this commit. Then it will propagate through the targeted devices with the defined connect interval of the devices. Progress can be followed in the log window, in the log tab for specific devices or in the audit menu. This configuration change has an entry that will track this specific change throughout the device park.

Warning

Make sure you save configuration AND commit it. Otherwise nothing will happen (only the respective config and the commit button turn orange).

Tip

Follow the progress of your configuration change either in "Logs" or through the "Audit" view for the specific configuration change. In audit click on the time stamp and select "show reports". These will refresh automatically.

Note

Configuration can only be active after having been committed and picked up by a device after the next qbee agent run.

It is possible to import and export configurations. Either to keep track of those or to migrate settings between different branches of the tree. Pressing "export" will export the json configuration to local download. An import is done by pasting the json into the form and then pressing "import". The inheritance part of the json should be disregarded when copying across the tree.

qbee-import-export

This was the general procedure how to configure and commit a configuration. Below are the different configuration options explained.

Settings

Settings

The settings menu is very simple. Selections that can be made include metric collection, report collection, remote console and process inventory. Each item can individually be selected as on or off. Metrics, reports and process information are transmitted every time the agent connects. How often the agent connects is defined in "agent scheduling interval". The number represents minutes. Choosing a fast update rate (e.g. 5 minutes) enables the system to respond very rapidly to configuration changes. However, on a mobile connection it might make sense to increase the time interval to save bandwidth. The software and user inventory is not updated every time the system reports back. So for these topics some delay might be experienced. Where possible the agent compresses the information to save bandwidth and only differential changes are transmitted.

Tip

Select the correct scheduling interval. If data traffic is of no issue we recommend 5 minutes for fast response time. If you want lower overhead from the tool please reduce this rate. If minimum bandwidth is the ultimate goal it is possible to disable metrics and reports as well as remote login.

Info

Setting changes are logged in the audit log but do not generate any log messages from the device. Please go to "Analytics -> Settings" to see an overview of current settings for all devices.

Warning

Gathering process information gives valuable information but also at a cost of increased data usage.

qbee-settings

System

Users

Here users can be added or removed. The password for users can be set in the security tab under passwords.

Info

If a new user is added and the user is assigned a password during the same commit it takes two runs of the qbee agent to converge and set the password.

Warning

If a user is removed the home directory of that user will be deleted without any further warning!!!

qbee-user-configuration

SSH Keys

In ssh keys all valid ssh keys for the users can be defined. By pressing the plus button additional users and ssh keys can be defined. If one or more ssh key is assigned to a user the previous keys will be overwritten.

qbee-management-of-ssh-keys

Package Management

With package management it is possible to automatically update debian packages from the repositories that are defined on the device. These could be either generic platform specific repositories (for example a specific version of the Raspbian stable branch) or own defined repositories. The latter allows to easily add own applications and manage which version gets deployed. Any time there is a new version in the repository it is automatically played out.

Tip

Create your own repository with your own applications. Define different versions like -stable, -test or -experimental. Then you can define different groups that automatically update any time one of the packages is updated. This way you can rapidly update your experimental lab setup and then move this into testing. Some customers define two test environments: internal for lab testing and external for real testing on a customer site. The possibility to assign these different versions to different groups makes development very fast and easy.

Tip

Customers have made very good experiences using repository management tools like aptly or packagecloud.

With enable/disable package management is controlled for the device / group or tree branch. The next selection defines if any pre-condition is used. Leave blank if no pre-condition is required.

Note

Pre-condition checks if a script/executable exits successfully with "exit=0". Only then the update is performed. A simple form of doing this is to use "/bin/true" which always returns 0 and thus always updates. The idea here is that more complicated conditions can be evaluated. For example time slots can be checked (only upgrade at night), machine conditions can be checked or certain environmental variables (one customer actually has a local web server UI and a technician needs to trigger an update window manually which then allows updates for 3 hours. At this time all accumulated updates get played out at once).

At this point a decision needs to be made. It is either possible to upgrade all packages from the repositories that actually have new versions available or specific packages can be watched and maintained. The image below shows an "update all" configuration.

qbee-package-management

While the window below shows a selective auto update of the packages "openssl" and "npm".

By clicking the "plus" you can get more boxes to define more packages.

qbee-package-management

It is important to define if you want to reboot after upgrade or not. If this is selected the device will immediately reboot after all current updates are completed.

Tip

Decide if you want to update all or single packages. As many packages as needed can be defined with the plus buttons.

Available packages for upgrading can be seen under device -> inventory -> software.

qbee-inventory-libraries

In the audit or log trail it is possible to see that updates have been completed.

qbee-audit-package-management


File Distribution

This is one of the most complex configuration policies. Please also refer to the question marks behind the form items to get additional information. File Distribution allows you to perform a multitude of different tasks:

  • Ensure that one or more files are distributed to the devices. If the file gets changed on the device it will be replaced with the original one defined as source file.

  • Convert a configuration file into a templating file and do key-value templating.

  • Create any kind of script, play it out through file distribution and run it

In general any file that should be distributed can be uploaded through the file manager. In the file distribution UI it is important to define the correct source and destination path. This is always the full path. A file is then played out to the according destination. This file has a checksum. Any changes to the file will be detected by the qbee agent and causes the file to be overwritten with the original version from the source path. This effectively restores any changes to the original version automatically as well as that it allows to play out new updated files.

Info

Any change to the file on the local device will be reverted by the next run of the qbee agent.

Warning

Please be aware that qbee agent checks if a file hash on the target machine is different than the file hash in the qbee file manager. This can consume a lot of bandwidth if either one of the following cases appears:

  • A file on the target changes between qbee agent runs
    This causes qbee to download the file again and again
  • A file is written to a temporary directory
    If a file is donwloaded to /tmp on the target and /tmp is a temporary directory that is cleaned after every reboot then the qbee agent will download the file again and again after each reboot. So, when using file distribution we recommend to put installation files in a permanent directory to avoid extensive downloads.

The first time the file is played out or any time there is a change of the file on the device or in the key-value configuration the "command to run" is executed. A checkbox determines if it runs in the background or not.

Here is an example that plays out the LoRaWAN Loriot network server to a Kerlink ifemto basestation

qbee-file-distribution-embedded-device

Info

It is possible to run chained commands like && . Please see this example:
/bin/chmod -x /tmp/loriot-install.sh && /tmp/loriot-install.sh -f

Tip

Multiple files can be played out. They will be played out in sequential order. So it is possible to play out a tarball and then play out an installation script in the second file distribution and run this with the second "command to run" definition. Only after that the first file has been played out the second file will be downloaded and run.

Tip

Sometimes it is needed to deliver a file to a target device that can be changed on the target without being re-deployed and over-written. We call this a one-off file distribution. The recommended way to solve this is to deliver a file "file1_tmp" to the device. Then deliver a "script.sh" that checks if the file "file1" exists on the device. If so it does not do anything. Otherwise it will copy "file1_tmp" to the correct location as "file1". After that it will never re-distribute the file even if "file1" is changed locally. Just deploying "file1" directly would result in a new file distribution and an overwrite process any time the file is changed on the device.

One very useful feature of qbee.io is to expose parts of regular config files for remote configuration through key-value pairs. This allows for many different use cases. Instead of pushing out different configuration files for different customers you can use the same basic config file and then configure some configurations individually. One example could be that different customers need to get directed to different MQTT servers. This can be very easily done. Just take the original configuration file and expose any accessible paramters through mustache notation - two double brackets as displayed here {{server}}.

Original config file configuration.conf:

[broker]
username = "my_username"
password = "James Bond"
server = "customer_a.mqtt.someserver.com"
port = 8883
tls = 0

[LPWAN system]
system-id = 12

Now it is possible to access one or all configuration options of this file remotely. Please copy this file. Let’s call it configuration.conf.tmpl. Then all configuration parameters that are in scope need to be exposed with {{your_key}} to expose them as key-value in the UI.

[broker]
username = "{{username}}"
password = "{{password}}"
server = "{{server}}"
port = 8883
tls = 0

[LPWAN system]
system-id = {{system-id}}

This file can now be played out. It is then possible to select a specific group of devices from the group selector and assign that group a different mqtt server or a different user name or password.

Tip

Structure your device groups in a way that it fits your configuration needs. For example
RPI → Customers → Customer_A → Test
RPI → Customers → Customer_A → Production
Then you can use different MQTT servers for testing and production.

qbee-file-distribution-embedded-device2

Please note that the "Command to run" gets executed any time a parameter is changed. Usually this command would restart the application. This in return reloads the configuration file and allows the new configuration to become active. In the following example you see the feedback the system has generated on a successful reconfiguration of a node-red flow file:

qbee-file-distribution-embedded-device3

Tip

Only expose the parameters that you really are interested in. They can be anywhere in a long configuration file. This way you can expose the three or four relevant parameters within a configuration file with many hundred lines.

Connectivity Watchdog

Astonishingly enough many embedded Linux systems that use mobile networks do not have a good watchdog functionality. However, it is quite frequently that connectivity is lost. One example could be a reconfiguration of the sim card in the mobile modem. Many times the Linux stack does not notice that connectivity is lost and the device goes offline. This can only be fixed through a reboot. This can cause large costs when a service technician needs to go on-site to do this. The alternative to ask a customer to power cycle is not very attractive either.

Note

The qbee watchdog is based on agent scheduling interval (in minutes). Please see the second image below. For this example the interval is set to 5 minutes.
The watchdog then has a threshold number of attempts before triggering. In this example this is set to 5.
This means that after 5 x 5 minutes = 25 minutes of no connectivity the device is automatically rebooted.

That is why qbee provides users with a basic watchdog functionality. Just configure it in the configuration section under the "connectivity watchdog". Any time the qbee agent runs it checks if access to the qbee server exists. If it does all is OK. If not it counts one failed attempt. This failed attempts counter gets increased any time the qbee server does not respond. If the defined threshold in the configuration is hit another last sanity check against a DNS server (8.8.8.8) is done. If that server cannot be reached the device is forced into a reboot.

qbee-connectivity-watchdog

It is important to note that the base time span between qbee agent runs is determined in settings. This can be anything from 5 minutes to much longer time periods.

qbee-settings

Process Watch

This configuration can keep processes running or stopped. A defined process name will be evaluated with regards to if it is running (present) or not (absent). If the condition is not met the command defined in "command to run" will be executed. This could for example restart a process. On the other hand this can also prevent processes to run.

Info

This evaluation is performed with the frequency with which the qbee agent runs. If it runs every 5 minutes then a process would be restarted at the latest 5 minutes after termination. Like all qbee configuration this will also be maintained should the network connection be down.

qbee-process-configuration

NTP

Here one or more NTP servers can be defined.
In the example image below an individual NTP configuration is given to some LoRaWAN gateways. One use case is to point to a local NTP server. In this case we point to a local Norwegian ntp server for the devices located in Norway. Please note also that we apply this configuration to the group "Kerlink" only. When pressing "save settings" a window appears and informs about which devices will be impacted by this configuration change.

qbee-ntp-configuration

Software

Software Management

Software management is used when Debian packages are played out. qbee will then handle the packages and play them out to all units that are in scope. This is useful for two different type of software management. You can play out and maintain files from repositories and/or the qbee file manager. The system distinguishes between them by checking if there is a package ending. So a debian package with name "node-red" will be fetched from any of the repositories that are configured for the Linux system. A package that has a debian compatible file name and a .deb package ending will be played out from qbee's internal file manager. For more information please see below. If the package defines a service qbee software management will automatically restart the service after any update.

Note

Debian packages defined by the service name and no ".deb" ending will be provided from the configured repository on the target device.

Packages that do have a debian compatible naming convention and a ".deb" ending will be provided by qbee from "software-management/packages".

If a service is defined it will be restarted after the update or after any key-value template changes

Warning

It is important that you adhere to the Debian naming convention for this to work.
A typical example is the file name used below: tinymesh-agent_0.0.3_all.deb
Or, in a more general notation: package_version_architecture.package-type

These packages need to be uploaded to the specific upload folder "software-management/packages".

qbee provides a fixed folder for this in in the upload folder. This is called "software-management/packages". If you want to use key-value templating on these packages via configuration files these files need to reside in "software-management/config-templates"

Below is an example that automatically installs a package for a Tinymesh sensor system to handle the radio communication and cloud connection. This debian package tinymesh-agent_0.0.3_all.deb is installed from the file manager storage space and then moved to each device in scope of the "iRPI" group. Also a template file called tinymesh-agent-conf.tmpl was uploaded to "software-management/config-templates". It gets resolved into the final configuration file for the tinymesh package with the following path on the target system /etc/tinymesh-agent/tinymesh-agent.conf.

qbee-application-updates

What really happens here is that original config file tinymesh-agent.conf is transposed into a templating file. Any key-value pair that should be exposed is replaced by a mustache notation with the double curly brackets.

[broker]
username = "my_username"
password = "James Bond"
server = "customer_a.mqtt.someserver.com"
port = 8883
tls = 0

[LPWAN system]
system-id = 12

Now it is possible to access one or all configuration options of this file remotely. Please rewrite this file into the template file tinymesh-agent-conf.tmpl. Then all configuration parameters that are in scope need to be exposed with {{your_key}} to expose them as key-value in the UI.

[broker]
username = "{{username}}"
password = "{{password}}"
server = "{{server}}"
port = 8883
tls = 0

[LPWAN system]
system-id = {{system-id}}

Committing this configuration will then trigger all devices in the iRPI group to install/upgrade the package (if this has not been installed yet) and then configure the new key-value pairs. It will also restart the service for any configuration change. In the case of the example above the audit trail shows that 2 devices have received this update already. They download the changed template file tinymesh-agent-conf.tmpl. Then the variables are applied to the real configuration file /etc/tinymesh-agent/tinymesh-agent.conf. As a last step the application * tinymesh-agent_0.0.3_all.deb* is restarted to apply the new configuration.

qbee-software-management

Software Install (tar.gz)

This configuration option allows to play out files in the tarball format tarballs in format -.tar.gz. The file is uploaded to the file manager and selected with its path in "Application name". This option allows to run pre and post commands. This can be used to stop a process or prepare before moving the tarball to the target system. The post command can do additional changes or start the application.

Info

It is possible to run chained commands with && . Please see this example:
/bin/chmod -x /tmp/loriot-install.sh && /tmp/loriot-install.sh -f

qbee-tarball

Security

Firewall

The firewall is used to configure the security of the devices. The functionality is developed to satisfy most user demands. If you need more detailed firewall configurations it is possible to create those through our Ansible integration. By default qbee opens its own port. So the firewall can be configured with default policy "drop" and qbee will work anyhow. The same is true for all remote access to the devices. Both the remote shell access and the remote access to any port do not demand any open ports in the firewall. All this is handled by the qbee agent through its own VPN.

Info

It is possible to configure the firewall with default policy "drop". qbee and remote access will work independent of this.

Rules can be created for TCP or UDP. It is possible to limit the IP access range or use "ANY" in order to allow any IP to connect. The following example shows a firewall configuration that drops all connections by default but allows HTTPS access through port 443. This could be used for allowing external web server access.

qbee-firewall-configuration1

So it is possible to drop all and define custom rules which allow access or accept all and drop or reject certain ports.

Tip

This example opens external HTTPS access to the device. This is potentially dangerous as it weakens the security concept. With qbee-connect you would be able to provide the same access without exposing any external ports since it is routed through the qbee VPN.

Password

In the passowrd configuration pane you can set your different user passwords. Please note that the passwords need to be inserted as a hash, not a clear text password. Depending on what hash you create the password will be set accordingly. We recommend that you use SHA-512 or stronger and not MD-5. Even if you do not know the previous password qbee will overwrite and create a new password.

Info

If a new user is added and the user is assigned a password during the same commit it takes two runs of the qbee agent to converge and set the password.

qbee-password-configuration

Tip

Please refer to your Linux distribution how to create password hashes. On many system this will produce a SHA-512 hash:
mkpasswd -m sha-512 -s