Skip to content

File distribution

Overview

All qbee.io configuration management options can get accessed through the "Configuration" function. Please see the "Configuration management" section for additional information.

This is one of the most complex configuration policies. Please also refer to the question marks behind the form items to get additional information. File Distribution allows you to perform a multitude of different tasks:

  • Ensure that one or more files are distributed to the devices. If the file gets changed on the device it will be replaced with the original one defined as source file from file manager.

  • Re-distribute one or more files if they change in file manager

  • Convert a configuration file into a templating file and do key-value templating. This allows for example to substitue different parameters in config files depending on which group the devices belong to.

  • Create any kind of script, play it out through file distribution and run it

  • Trigger run-once actions

Basic file distribution

In general any file that should be distributed can be uploaded through the file manager. In the file distribution UI it is important to define the correct source and destination path. This is always the full path. A file is then played out to the according destination. This file has a checksum. Any changes to the file will be detected by the qbee agent and causes the file to be overwritten with the original version from the source path. This effectively restores any changes to the original version automatically as well as that it allows to play out new updated files.

Info

Any change to the file on the local device will be reverted by the next run of the qbee agent.

Warning

Please be aware that qbee agent checks if a file checksum on the target machine is different than the file checksum in the qbee file manager. This can consume a lot of bandwidth if either one of the following cases appears:

  • A file on the target changes between qbee agent runs
    This causes qbee to download the file again and again
  • A file is written to a temporary directory
    If a file is donwloaded to /tmp on the target and /tmp is a temporary directory that is cleaned after every reboot then the qbee agent will download the file again and again after each reboot. So, when using file distribution we recommend to put installation files in a permanent directory to avoid extensive downloads.

The first time a file is played out or any time there is a change of the file on the device or in the key-value configuration the "command to run" is executed.

Info

It is possible to run chained commands like && :
command A && command B

It is possible to run a command in the background with & :
command A &

It is possible to run complete scripts. They need to be distributed first. Usually they are written as non executable. Therefore you can execute them with bash or you need to do a chmod
bash /usr/local/bin/my_script.sh

It is possible to run python scripts with python or python3:
python3 /usr/local/bin/run-me.py

If files are played out they might need a chmod or chown command as "command to run" to receive the correct file permissions or ownership. Please do not use chmod on a templated file:

chmod 755 myfile.txt

chown pi myscript.sh -> changes ownership for user

chown pi.pi myscript.sh -> changes ownership for user and group to "pi"

Sometimes it makes sense to run the script in the background and pipe output messages to /dev/null. This can be done as follows: Use "Command to run" as bash /usr/local/bin/cpu_temp.sh > /dev/null 2>&1 &. This runs the script in the background and the appendix > /dev/null 2>&1 & pipes bash output to /dev/null.

Templating and chmod will NOT work

If templating is used you CANNOT do a chmod on that file. This will cause a recurring execution. So it is not possible to template a script, do a chmod on the same file and then execute it. It is also not possible to template a config file and do chmod. Please use a cpcommand or the technique described here. This is an example for a config file: cp /some/temporary/collection/location/monitrc /etc/monit/monitrc && /bin/chmod 0700 /etc/monit/monitrc

Files are played out in sequential order

Multiple files can be played out. They will be played out in sequential order. So it is possible to play out a tarball and then play out an installation script in the second file distribution and run this with the second "command to run" definition. Only after that the first file has been played out the second file will be downloaded and run.

One-off file distribution for files that will be changed on the target device

Sometimes it is needed to deliver a file to a target device that can be changed on the target without being re-deployed and over-written. We call this a one-off file distribution. The recommended way to solve this is to deliver a file "file1_tmp" to the device. Then deliver a "script.sh" that checks if the file "file1" exists on the device. If so it does not do anything. Otherwise it will copy "file1_tmp" to the correct location as "file1". After that it will never re-distribute the file even if "file1" is changed locally. Just deploying "file1" directly would result in a new file distribution and an overwrite process any time the file is changed on the device. This can also be solved with a command to run:

[ ! -f /home/pi/text.txt ] && cp -p /home/pi/text-base.txt /home/pi/text.txt 
Please see here for an example.

Adjusting file ownership - how to change files from being owned by root

The ownership of distributed files is inherited from the parent folder, where you have uploaded your file into. If this ownership and file permissions is wrong i can be set with chmod and chown commands. Using chown pi.pi myscript.sh sets both user and group ownership to pi. Please do not use chmod on templated files. Below we describe how to solve this.

Running scripts / software in a user context

Sometimes it is important to run a software or script in a user context and not as root. This can be done as follows through the command to run box (here for user "pi"):

chown pi.pi /home/pi/my-application && sudo -u pi /home/pi/my-application

and here as a service

sudo -u pi node-red-restart

Advanced file distribution with templating

One very useful feature of qbee.io is to expose parts of regular config files for remote configuration through key-value pairs. This allows for many different use cases. Instead of pushing out different configuration files for different customers you can use the same basic config file and then configure some configurations individually. One example could be that different customers need to get directed to different MQTT servers. This can be very easily done. Just take the original configuration file and expose any accessible parameters through mustache notation - two double brackets as displayed here {{server}}. A thorough example can be found here

Original config file configuration.conf:

[broker]
username = "my_username"
password = "James Bond"
server = "customer_a.mqtt.someserver.com"
port = 8883
tls = 0

[LPWAN system]
system-id = 12

Now it is possible to access one or all configuration options of this file remotely. Please copy this file. Let’s call it configuration.conf.tmpl. Then all configuration parameters that are in scope need to be exposed with {{your_key}} to expose them as key-value in the UI.

[broker]
username = "{{username}}"
password = "{{password}}"
server = "{{server}}"
port = 8883
tls = 0

[LPWAN system]
system-id = {{system-id}}

This file can now be played out. It is then possible to select a specific group of devices from the group selector and assign that group a different mqtt server or a different user name or password.

Tip

Structure your device groups in a way that it fits your configuration needs. For example
RPI → Customers → Customer_A → Test
RPI → Customers → Customer_A → Production
Then you can use different MQTT servers for testing and production.

qbee-file-distribution-embedded-device2

Please note that the "Command to run" gets executed any time a parameter is changed. Usually this command would restart the application. This in return reloads the configuration file and allows the new configuration to become active. In the following example you see the feedback the system has generated on a successful reconfiguration of a node-red flow file:

qbee-file-distribution-embedded-device3

Tip

Only expose the parameters that you really are interested in. They can be anywhere in a long configuration file. This way you can expose the three or four relevant parameters within a configuration file with many hundred lines.

FAQ

What is the default ownership of my distributed file?

The ownership is inherited from the parent folder, where you have uploaded your file into.

The output of my script/program belongs to root. Do I have to run a chmod afterwards?

The reason this happens is, because the qbee agent executes your script/program as root. And therefore the output written to disk belongs to root as well. To change that specify the user as well. E.g. on a Raspberry Pi you might want to use the user pi. Change the "command to run" for example

  • sudo -u pi python3 /path/to/my/script.py for a python script
  • sudo -u pi bash /path/to/my/script.sh for a bash script
  • sudo -u pi exec /path/to/my/custom/executable for a binary file
I piped the command line output of my script to a textfile but I cannot read it as it belongs to root. Do I have to change permissions?

Running your script in the user context as described above will cause the output written by the program to be written for that user. However, when piping to a text file the owner will be root as the process is still by qbee in the root context. A way to circumvent this issue is to use the tee command as follows:

sudo -u pi bash /path/to/my/script.sh | sudo -u pi tee /path/to/output/output.txt

whereas a simple

sudo -u pi bash /path/to/my/script.sh > /path/to/output/output.txt

will result in output.txt being written for root.

I cannot execute my uploaded file - what should I do?

If you want to execute a script it is recommended to use the corresponding interpreter. This way the file does not need any execution permissions but only read permissions.

See "The output of my script/program belongs to root. Do I have to run a chmod afterwards?" in this FAQ for examples.

However, when executing a binary file, even with the exec command, you need to add execution files. This can be done for the root user via

chmod u+x /path/to/my/binary

or if the file already belongs to a custom user (e.g. pi) you can also do

sudo -u pi chmod u+x /path/to/my/binary

I am using a template file. Is there anything special I need to consider?

Indeed these files are treated differently by qbee than your "regular" files. Since the template file is expanded by qbee on the system its "state" (ownership and permissions) cannot be changed. As indicated before you cannot run a chmod on template files.

If for some reason you want to modify permissions for the template file anyhow, then we suggest the workaround to have "template collection space" where the original template files reside (that cannot be modified) and then to use the install command within "command to run" to place the template file with desired permissions to your target directory. E.g.

install -C -m 700 -o pi -g pi /template/collection/file.conf /target/directory/file.conf

such that /target/directory/file.conf belongs to the user and group pi with permissions -rwx------.

Why does my file distribution fail or repeat?

Is your file distribution followed by a "command to run"?

  1. No "command to run" specified

    • Have you distributed your file to a temporary folder (such as /tmp/)? Then, if the folder (or its content) is deleted a new downloaded is issued by qbee.
    • Is your file changed by a system process? Then, qbee is working against the system process and the file is downloaded again.
  2. A "command to run" is specified

    • The problems from above still hold but additionally to the download the specified command is run again
    • Have you performed a chmod or a chown on a template file? You cannot do that, c.f. "I am using a template file. Is there anything special I need to consider?" within this FAQ for advice.
    • Does your file have the wrong permissions? If you want to run a script you may have the wrong permissions and the process will return an exit code. Confer "I cannot execute my uploaded file - what should I do?" within this FAQ.
    • Do you have a typo or a wrong path in your "command to run"? This is actually the most common reason for file distribution to fail :)