Skip to content

Binary & File Updates

What is file distribution?

File distribution is the tool that can solve basically any configuration management use case. From distributing simple files or scripts to running complex templated docker-compose actions. It is also used to install or update binaries or run remote scripts.

All files are fetched from the platform's file manager. The file manager is the single source of truth for the files. This means that if the hash of the file on the device changes it is reapplied with the version that is in the file manager.

Files can be uploaded manually, through an API or through the CLI tool. This allows the integration into CI/CD tools.

With file distribution you can:

  • Distribute files to devices

  • Re-distribute files automatically if they change in file manager

  • Re-distribute files if the checksum changes on the device

  • Run a "command to run" any time a file is distributed again or a template value changes

  • Trigger run-once commands

  • Run remote scripts

  • Use key-value templating

The File Distribution UI

The basic UI is described in the configuration UI. There you can also learn about how "enabled" and "extend" works as well as the inheritance concept. Any file set can contain many different file distributions. The name for a file set can be defined in the label box. Pressing the "Add file" sign adds more file distribution dialogues or template parameters. Source files can be selected from file manager through a dialogue by pressing the folder symbol. If a file contains key-value templating the "Template" box needs to be checked. Then keys and values can be defined in the according boxes. The pre-condition is explained here.

File distribution

It is important to note that all files are distributed according to order. This will start with the first file set and go through all files there before proceeding with the second file set.

Command to Run

Command to run is a powerful tool. It allows to run the specific command any time a file is distributed again (independently if it changed in file manager or on the device). It will also trigger any time a key-value templating parameter is changed and therefore a new file is written. This means that it is always run the first time a file is played out and then on any additional change.

Just changing the command to run without any file change does not execute it.

One of the simplest commands would be bash /usr/local/bin/myscript.sh which just runs the script as root. The agent usually executes commands as root if not defined differently. If bash is not used a script needs to be made executable.

Useful commands to run:

It is possible to run chained commands like && :
command A && command B

It is possible to run a command in the background with & :
command A &

It is possible to run complete scripts. They need to be distributed first. Usually they are written as non executable. Therefore you can execute them with bash or you need to do a chmod before being able to execute them.

bash /usr/local/bin/myscript.sh

chmod 755 /usr/local/bin/my-golang-bin && /usr/local/bin/my-golang-bin

Scripts can also be run as user (assuming they have correct file execution and ownership settings):

sudo -u myuser bash /usr/local/bin/myscript.sh

It is possible to run python scripts with python or python3:

python3 /usr/local/bin/run-me.py

If files are played out they might need a chmod or chown command as "command to run" to receive the correct file permissions or ownership or to make them executable. When the agent writes files it tries to determine the current onwership and sets permissions accordingly. So files written into a user's home folder usually belong to that user by default, but are not executable.

chmod 755 myscript.sh

chown pi myscript.sh -> changes ownership for user

chown pi.pi myscript.sh -> changes ownership for user and group to "pi"

Some scripts and executables do not properly close stdin/stderr/stdout. So if you experience that execution of scripts hang then it might be necessary to explicitly close stdin/stderr/stdout. This is done as follows:

bash /usr/local/bin/myscript.sh > /dev/null < /dev/null 2>&1

System binaries already installed on the system can be called directly as they are already set to be executable. Eg. restarting a the ssh service

systemctl restart sshd

The file distribution allows to return 100 lines of output from any operation executed in command to run and show this in the UI. This allows simple debugging or the retrieval of information. The following command to run would return the last 100 lines of the text file. The output can be found in logs:

Configuration logs

Key-Value Templating

One very useful feature of the platform is to be able to use key-value pairs in configuration or other files. One specific use case is to expose certain parts of configuration files in this form. This allows to define certain specific parameters on group, tag or device level while using the same general configuration file. This templating separates logic or general configuration from group or device specific configuration. For more information please refer to this blog post.

Any part of a file can be marked with mustache notation - two double brackets as displayed here {{server}}. Below we use the example of an MQTT server configuration file where different parts of the device fleet should communicate with a different MQTT servers.

Original config file configuration.conf:

[broker]
username = "my_username"
password = "something"
server = "customer_a.mqtt.someserver.com"
port = 8883
tls = 0

Now it is possible to access one or all configuration options of this file remotely by turning it into a key-value enabled file. All configuration parameters that are in scope for change need to be exposed with {{your_key}} to expose them as key-value in the UI.

[broker]
username = "{{username}}"
password = "{{password}}"
server = "{{server}}"
port = 8883
tls = 0

This file can now be distributed. The key-value pairs can now either be specified through the parameter function or through the "Template parameters". Any time a key-value is changed it will trigger a new file distribution. In this case a "command to run" would make sure that for any change the MQTT server gets restarted and will then read again the configuration file

Please note that these key-value pairs can be defined through the UI, the API or the CLI tool. Many users keep the values in their main application database and just transfer the specific parameters to the platform with the API. This gives a very simple, yet powerful mechanism to individualize devices or software.

It is also important to note that in case of any change and an application restart only the devices in scope will do this. This is minimizing risk and optimizing uptime.

Run-once commands

In some cases it is needed to run a command once but at a defined time. One example being a reboot command for parts of the fleet. This is in many ways counter-intuitive in a state based system. Therefore, a trick needs to be applied. In this example we create a reboot script, upload it to file manager and run it with bash. This will only execute one time on all devices. But it is easy to make it controllable. Just add a comment file and use key-value templating. We use epoch time like this {{epoch}} because that is always advancing.

reboot-script.sh

#!/bin/bash

# unique run-once trigger, here epoch time {{epoch}}

shutdown -r now

Now each time epoch time is changed for a device or group a reboot will occur because this will write a new reboot-script.sh file and then invoke the command to run.

Configuration run once

FAQ

What is the default ownership of my distributed file?

The ownership is inherited from the parent folder, where you have uploaded your file into.

The output of my script/program belongs to root. Do I have to run a chmod afterwards?

The reason this happens is, because the qbee agent executes your script/program as root. And therefore the output written to disk belongs to root as well. To change that specify the user as well. E.g. on a Raspberry Pi you might want to use the user pi. Change the "command to run" for example

  • sudo -u pi python3 /path/to/my/script.py for a python script
  • sudo -u pi bash /path/to/my/script.sh for a bash script
  • sudo -u pi exec /path/to/my/custom/executable for a binary file
I piped the command line output of my script to a textfile for debugging but I cannot read it as it belongs to root. Do I have to change permissions?

Running your script in the user context as described above will cause the output written by the program to be written for that user. However, when piping to a text file the owner will be root as the process is still by qbee in the root context. A way to circumvent this issue is to use the tee command as follows:

sudo -u pi bash /path/to/my/script.sh | sudo -u pi tee /path/to/output/output.txt

whereas a simple

sudo -u pi bash /path/to/my/script.sh > /path/to/output/output.txt

will result in output.txt being written for root.

I cannot execute my uploaded file - what should I do?

If you want to execute a script it is recommended to use the corresponding interpreter. This way the file does not need any execution permissions but only read permissions.

See "The output of my script/program belongs to root. Do I have to run a chmod afterwards?" in this FAQ for examples.

However, when executing a binary file, even with the exec command, you need to add execution files. This can be done for the root user via

chmod 755 /usr/local/bin/my-golang-bin && /usr/local/bin/my-golang-bin

or if the file already belongs to a custom user (e.g. pi) you can also do

sudo -u pi chmod 755 /usr/local/bin/my-golang-bin && sudo -u pi /usr/local/bin/my-golang-bin

Why does my file distribution fail or repeat?

Is your file distribution followed by a "command to run"?

  1. No "command to run" specified

    • Have you distributed your file to a temporary folder (such as /tmp/)? Then, if the folder (or its content) is deleted a new downloaded is issued by qbee.
    • Is your file changed by a system process? Then, qbee is working against the system process and the file is downloaded again.
  2. A "command to run" is specified

    • The problems from above still hold but additionally to the download the specified command is run again
    • Does your file have the wrong permissions? If you want to run a script you may have the wrong permissions and the process will return an exit code. Confer "I cannot execute my uploaded file - what should I do?" within this FAQ.
    • Do you have a typo or a wrong path in your "command to run"? This is actually the most common reason for file distribution to fail :)