This is the user manual for https://qbee.io .

Document

This document contains both quick start guides to on-boarding devices, navigating the qbee.io application as well as in-depth references.

This icon will indicate practical tips or best practices.
This icon will highlight very important information.

Introduction to qbee.io

qbee.io is a tool to configure, manage, monitor and administrate a large number of Linux devices. In contrast to other libraries such as Chef or Puppet qbee.io is specifically designed to work both with regular Linux devices as well as with embedded devices. It uses a RAM disk to prevent flash wear and is optimized to run using a minimum amount of resources. With qbee.io you can access your devices and servers behind multi-level firewalls and beyond NAT routers as well as on mobile networks. Basic configuration management is easily available through the GUI and through the qbee-connect tool it is possible to seamlessly combine qbee.io with Ansible. In addition to providing automation, configuration and access qbee.io collects metrics such as CPU load, file system usage and much more. This is supplemented with inventory discovery of libraries, open ports, registered users and also running process metrics.

Mission

Manage, secure and orchestrate new and legacy (embedded) Linux IoT devices. Secure and control the Internet of Things by using a server automation mindset with technology specifically developed for embedded devices.

In contrast to many other tools qbee.io uses a local agent on the remote device. This agent wakes at pre-determined intervals and establishes a secure connection to a central server. If there is a new configuration available the agent will download the new configuration and converge towards the indicated state. This state will constantly be monitored and maintained. If for some reason it is impossible to achieve the desired state an error message is issued. This concept allows offline devices (for example industrial controllers sitting in stock) to immediately converge towards the now current configuration. Since the device itself issues the connection qbee.io works across most firewalls and also across mobile networks. But qbee.io is not only a configuration management tool. It contains a plethora of functions that will make your life easier, all implemented through one highly secure SSL encrypted communication port using modern elliptic curve cryptography (ECDSA P-521).

qbee.io features

Metrics reporting

qbee.io allows to collect metrics from all devices. Both CPU load, memory and disk usage can be checked. This gives you a detailed view into your system and allows you to see problems before they occur. All metrics can be constantly monitored and alerts can be initiated if thresholds are crossed.

Inventory reports

Understand what ports are open, which users are on your system, what password types are use (insecure MD5 or sha-512), how old passwords are. qbee.io gathers an inventory of all libraries that are present on the embedded device. This helps you to keep track of dependencies and different versions. If necessary the system can also show your most active processes through a "top" type of representation.

Grouping

All devices can be ordered in a tree structure through different groups. This allows to control and configure all devices very granularly independent if you want to order them according to customer, version, geographical location or any custom build logic.

Configuration management

This is the key functionality of qbee.io. Through smart grouping and configuration profiles it is possible to control thousands of devices through a defined set of configurations. Define firewall rules or software configurations through a simple and consistent interface across different device types. If you have additional configuration needs qbee.io offers a way to integrate with Ansible.

Remote access

With qbee.io the same security mechanism that is used for metrics and configuration management can also be used for remote console access. The tool has a build in web based remote console. This enables a quick way to access remote devices, even behind a firewall. In addition to that we offer a local client called qbee-connect that allows to tunnel any port or protocol via https into a local browser through OpenVPN. This allows you to access SCADA systems, configuration tools or node-red instances as if they were running on your local machine. Get ssh access through your terminal or make all your units available to Ansible with our connector.

Vulnerability checks

qbee.io can create an inventory of all installed libraries. This is compared against the CVE and NIST security database. Any possible vulnerabilities are visualized with their according score and a link to additional information.

Security hardening

Use qbee.io to ensure that your firewall is configured correctly. With remote access handled by qbee.io you can drop all connections to any port. In addition it is easy to become aware of all users on a system (also the default support users from equipment vendors) and remove them or assign strong passwords. Periodically change user passwords or rotate ssh keys. If a legacy system shows a vulnerability of a library this library can be exchanged if needed.

Alerts

The alert engine allows you to trigger on thresholds or error messages. This allows to detect issues with memory leaks, file system issues like log files that grow out of proportion or high CPU load. Make sure that these issues will get handled pro-actively.

Application or library updates

If the embedded device has a way of updating firmware files qbee.io can be used for over-the-air (OTA) firmware updates. In addition it is very easy to play out new applications or update existing ones. Automate your complete file distribution.

Configuration file templating

qbee.io can template any configuration file. This allows you to expose parts of the configuration easily through mustache templating and make this available as key-value pairs in the qbee.io UI. This way you can for example assign different MQTT servers to different customers while still maintaining a common configuration for the application. We have also created an example how to do node-red templating allowing you to manage node-red flows remotely.

Getting started

Overview

Starting with qbee.io is done in some easy steps:
  • Create a user account with qbee.io

  • Locate the agent for your specific device or ask us to build one

  • Install the agent

  • Bootstrap the agent to qbee.io to create the initial trust

Creating a user account

Please apply for a user account on qbee.io

Select the appropriate agent for your device

qbee.io is cross-compiled for your device. This enables us to run on basically any device that supports basic Linux from full server to limited embedded device. We continuously update the library of available agent packages. We build specific libraries for devices with customized Linux and provide general packages that will run on many platforms that follow common Linux flavour styles. If you run a device which is based on Ubuntu Linux you can try the according package for your CPU architecture (e.g. Armhf or AMD64) and in many cases this will work out of the box. If you have a more customized Linux or a limited Linux please get in touch and we will see how we can help you.

Installing the agent

Donwload the agent to your device. Install it according to the manual provided with your package or see here: Metrics reporting
After successful installation the agent explains how to bootstrap.

Bootstrapping a device

Bootstrapping is the process of securely pairing the device with the qbee.io server. Here the initial trust between the device and the server is established through the exchange of keys. A typical bootstrap command will look like this:

sudo /opt/devctrl/bootstrap/bootstrap.sh -s <ip-of-policyhost>

This initiates the communication with the server. You will see a feedback on the agent command line how the bootstrap process is commencing. During the process the new device has to be accepted in the qbee.io web app.
Check if the IP of the device and the request time is correct and then approve the device. accept_device

When the device is approved it takes a while before it is visible in the tree. Eventually it will appear under a group called "unassigned devices". This is important as there is no configuration associated with this group. First if you move the device into a group that has configuration or if you create a group and then define configuration this device is in a defined state.

Any new device needs to be accepted in the "Pending devices" tab. When that is done it will appear under "unassigned devices" in the tree. Move it to the correct group and it will receive the valid configuration for that group.

qbee.io web app

Overview

This is the initial dashboard for qbee.io. It will give a quick overview over what has happened in the system. The three donuts give an overview over status messages for the time periods of day, week and month. Status messages have 4 different states:

  • Info:
    Basic information about events that have happened. No action needed.

  • Warning:
    Information that has impacted the device but usually no action needed. A typical example is the restart of an application.

  • Error:
    This an indication that something has not happened the way it was intended. This is a status message that usually demands action.

  • Critical:
    Here immediate action is needed as something is critically wrong and needs fixing.

Devices

The device tab shows you an overview over all devices and the overall tree structure. Selecting a specific device will give you more information about the device.

Tree

The tree creates the logical structure in which all devices exist. This allows devices to inherit configuration and you can select to configure different configurations to different branches of the tree.

All new devices will appear in an "unassigned group". Please move them from there into the correct group of the tree structure and then save the tree. They will then apply the configuration of that group.

In the tree you can move devices, create and rename groups. All changes will only be active after the save button has been pressed. Only empty groups can be deleted.

First, group your inventory logically. Best practice is to group servers and network devices by their What (function, vendor, revision), Where (location or region), and When (development, testing or staging):
What: Multitech, mLinux
Where: Norway, Oslo, floor_19, building_A
When: dev, test, staging, prod

If you are using the Ansible functionality of our qbee-connect tool then this tree will be exposed into the Ansible inventory.

Device Tab

This tab shows some basic device and network information.
devices
Beside the name an edit menu can be called. Here you can position the device on the map. The default localization is IP address resolution. It is possible to override this with coordinates. The address function is for information purposes only.

If a console connection is available the green button will open a new window with a shell. It is possible to select which user you want to use for login.

Please use the browsers "paste" function to copy information (e.g. password). The shortcuts like ctrl+C do not work!

The graphs tab shows you various metric information. You can use the selector to adjust the time period. By opening a rectangle with your mouse you can zoom in into interesting areas.

metrics

The next tab shows you a log of all events that happened for this device. You can sort by time, level or labe as well as with a free text search.

The inventory tab gives you various information. What information is populated here is defined in settings in configuration. So if you miss something here please go to settings and adjust these accordingly.

The amount of inventory that is collected is defined in configuration→settings. Only changes are reported thus saving bandwidth. Inventory is not real time. It can have a certain delay. Please look for the time stamp of the last inventory.
Activating the process inventory will use additional bandwidth as this is reported every time the qbee agent reports back. On mobile networks we recommend to use this only for debugging.

The following inventory items are available:

  • Ports:
    All defined ports are shown and which process is utilizing those. The 0.0.0.0:xxxx ports are internal ones.

  • Users:
    We show user name, shell, home directory, password age and type of password algorithm. This is very useful to discover pre-defined users that should not be on the device or identify passwords that were not changed.

Please do not use weak passoword algorithms like MD5 any more.
  • Software:
    Here you see all software packages that are installed. This helps to identify both the version and the availability. The search tap can be used to simply find and verify versions such as "python3" to get all the python3 libraries available.

By checking the CVE box we check all libraries in scope against the CVE/NIST vulnerability database. This allows you quickly to review potential threats to your device.
  • Processes:
    This gives you an insight into your system exposing all processes running and what their rerspective resource usage is.

Configure

This is one of the key tasks of qbee.io. Here you define the configuration for your devices. In order to understand how this works we need to explain the way qbee.io works again.

Configuration Management with qbee.io

qbee.io is a state based configuration management tool. The local qbee agent on the device queries the centralized server with a set interval and checks if new configuration information is available. If not it just checks if it still complies to the current state and goes to sleep again. Otherwise it downloads the new configuration and then converges against the new configuration. This concept has many advantages as the agent can operate and maintain configuration even if the network is down. Also devices that come later into a group (spare parts, replacements) will always get the latest configuration with latest ssh keys and user definitions. Some practical implications are:

  • configuration is only becoming valid if you commit it

  • it will take at least until the next connect interval of the agent until it receives the configuration

  • the qbee agent will report success or an error if it cannot reach its state

  • the qbee agent maintains its defined state. If you manage password or firewall settings and you change it manually qbee will revert it in the next run (and send you a log message about this)

  • in very few cases the agent cannot converge to its defined state. Then an error message is displayed (for example you play out a file where you do not have correct access rights, or a process should be started that does not exist)

In the configure menu the top selector defines for which device or group of devices configuration is applied. "All devices" will enable this for the main branch of the tree. All subsequent groups and devices that do not have their inheritance disabled will inherit this configuration.
You can do multiple configuration changes at once, but it is recommended to always do one after the other.

Any time new configuration is committed the system generates a list of all devices that are impacted by this. Please check if this is as expected before you continue.

inheritance

In the image above the group "hardware devices" has inherited its settings from group "All devices". The menus are greyed out but show the configuration. By pressing the orange "Edit settings" button this inheritance can be broken. Then hardware devices (and all its group below in the branch of the tree) will inherit the new configuration. Breaking inheritance and defining new settings will cause a green "save" button to appear. New configurations need to be saved. Then an information box pops up showing all impacted groups and devices.

configure

Here you can cancel or save. Then the configuration is written. At this point in time the configuration is not yet committed.

commit

First when you press the orange commit button the configuration change gets active on the server and can be picked up by the relevant devices. Now it will propagate through the targeted devices with the defined connect interval of the devices. Progress can be followed in the log window, in the log tab for specific devices or in the audit menu. This configuration change has an entry that will track this specific change throughout the device park.

Follow the progress of your configuration change either in "Logs" or through the "Audit" view for the specific configuration change.

This was the general procedure how to configure and commit a configuration. Below the different configuration options are explained.

Settings

The settings menu is very simple. Selections that can be made include metric collection, report collection, remote console and process inventory. Each item can individually be selected as on or off. Metrics, reports and process information are transmitted every time the agent connects. How often the agent connects is defined in "agent scheduling interval". The number represents minutes. Choosing a fast update rate (e.g. 5 minutes) enables the system to respond very rapidly to configuration changes. However, on a mobile connection it might make sense to increase the time interval to save bandwidth. The software and user inventory is not updated every time the system reports back. So for these topics some delay might be experienced.

Select the correct scheduling interval. If data traffic is of no issue we recommend 5 minutes for fast response time. If you want lower overhead from the tool please reduce this rate. If minimum bandwidth is the ultimate goal it is possible to disable metrics and reports as well as remote login.
Gathering process information gives valuable information but also at a cost of increased data usage.

System

Users

Here users can be added or removed. The password for users can be set in the security tab under passwords.

If a new user is added and the user is assigned a password during the same commit it takes two runs of the qbee agent to converge and set the password.
If a user is removed the home directory of that user will be deleted without any further warning!!!
SSH keys

In ssh keys all valid ssh keys for the users can be defined. By pressing the plus button additional users or ssh keys can be defined. If one or more ssh key is assigned to a user the previous keys will be overwritten.

NTP

Here one or more NTP servers can be defined.

Process Watch

This configuration can keep processes running or stopped. A defined process name will be evaluated with regards to if it is running (present) or not (absent). If the condition is not met the command defined in "command to run" will be executed. This could for example restart a process. On the other hand this can also prevent processes to run.

This evaluation is performed with the frequency with which the qbee agent runs. If it runs every 5 minutes then a process would be restarted at the latest 5 minutes after termination. Like all qbee configuration this will also be maintained should the network connection be down.
File Distribution

This is the most complex configuration policy. Please also refer to the question marks behind the form items to get additional information. File Distribution allows you to perform a multitude of different tasks:

  • Ensure that one or more files are distributed to the devices. If the file gets changed on the device it will be replaced with the original one defined as source file.

  • Convert a configuration file into a templating file and do key-value templating.

  • Create any kind of script, play it out through file distribution and run it

In general any file that should be distributed can be uploaded through the upload function in the UI (see below). A file is then played out to the according destination. This file has a checksum. Any changes to the file will be detected by the qbee agent and causes the file to be overwritten with the original version from the source path. This effectively restores any changes to the original version automatically.

Any change to the file on the local device will be reverted by the next run of the qbee agent.

The first time the file is played out or any time there is a change of the file on the device or in the key-value configuration the "command to run" is executed. A checkbox determines if it runs in the background or not.

It is possible to run chained commands like && . Please see this example:
/bin/chmod -x /tmp/loriot-install.sh && /tmp/loriot-install.sh -f

One very useful feature of qbee.io is to expose parts of regular config files for remote configuration through key-value pairs. This allows for many different use cases. Instead of pushing out different configuration files for different customers you can use the same basic config file and then configure some configurations individually. One example could be that different customers need to get directed to different MQTT servers. This can be very easily done. Just take the original configuration file and expose any accessible paramters through mustache notation - two double brackets as dsiplayed here {{server}}.

Original config file configuration.conf:

[broker]
username = "my_username"
password = "James Bond"
server = "customer_a.mqtt.someserver.com"
port = 8883
tls = 0

[LPWAN system]
system-id = 12

Now it is possible to access one or all configuration options of this file remotely. Please copy this file. Let’s call it configuration.conf.tmpl.
Then all configuration parameters that are in scope need to be exposed with {{your_key}} to expose them as key-value in the UI.

[broker]
username = "{{username}}"
password = "{{password}}"
server = "{{server}}"
port = 8883
tls = 0

[LPWAN system]
system-id = {{system-id}}

This file can now be played out. Now it is possible to select a specific group of devices from the group selector and assign that group a different mqtt server or a different user name or password.

Structure your device groups in a way that it fits your configuration needs. For example
RPI → Customers → Customer_A → Test
RPI → Customers → Customer_A → Production
Then you can use different MQTT servers for testing and production.

file_distribution

Please note that the "Command to run" gets executed any time a parameter is changed. Usually this command would restart the application. This in return reloads the configuration file and allows the new configuration to become active. In the following example you see the feedback the system has generated on a successful reconfiguration:

configuration_chnage

Only expose the parameters that you really are interested in. They can be anywhere in a long configuration file.

Upload

Logs

Graphs

FAQ (Frequently Asked Questions)

Additional resources list

Glossary

Appendices

Index