Run-once scripts / commands

How to send a reboot command to a group of remote edge devices?

This example shows how qbee can be used to issue commands to a large fleet of remote embedded Linux devices. It will explain a simple way to distribute the shell script and then execute it remotely. While doing so we explain how qbee works and give valuable background information.

With qbee "Configure -> File distribution" it is possible to distribute and run script files to one or many groups or devices. qbee is a state based tool. This means that any configuration is autonomously handled by the agent. If a new configuration is available it is downloaded and the agent starts to converge towards this state. The theory behind this is "promise theory". A huge advantage is that a spare device that was offline for a few years still will get and converge to the last state (for example the current ssh keys) and the agent can also continue to maintain the state when the network is down. So if someone tries to manually change the firewall settings that are under qbee watch the agent will restore the settings again with its next run. The disadvantage is that it is slightly more complicated to do a "run-once" configuration but it also introduces the advantage that even the run-once script will be able to run months after it was initiated if a device was offline. Removing or disabling it will obviously cancel the action.

Handling run-once actions with qbee.io

qbee is state based and the concept of run-once needs to be created first. But then it works extremely well and also reaches devices that are not online at the time the command is issued or if they get moved into a new group later on.

This small demo creates a script that reboots a group of devices. This will explain a lot of the concept.

reboot-script.sh

#!/usr/bin/env bash

shutdown -R now

The "command to run" that is used is

    bash /usr/local/bin/reboot-script.sh  &
You could also run a python script by using "python" or "python3".

Explaining the "command to run"

The script is defined as a bash script and started by calling bash /usr/local/bin/reboot-script.sh &. This runs the script in the background in root context. This command will run any time the file is changed.

File distribution will now look like this:

!qbee-reboot-script-simple

This only works for one run!

This will work. But only one time. The first time the script is not on the target device. Therefore, it gets distributed and run. But as long as it does not change on the target or in the file manager this will never run again. So this does not really solve the problem.

A simple trick to enable the same script to run multiple times on demand

This is a very simple trick. Just introduce another file that uses the qbee Mustache key-value templating possibilities. This means that we create another file in the same file distribution only containing a run-once trigger with two curly brackets {{key}}. The simplest way is create another file that only contains a comment that can work as a trigger. The value we use is epoch time as this always advances and always is new and unique. So the proposed file only contains one line: # unique run-once trigger, here epoch time {{epoch}}. Please make sure that it is in the same "Files to distribute" section and that you enable the "Template" checkbox and add one checkbox with key = epoch and value = 1598530165 (see below). This reboot-trigger.tmpl file is now expanded into a new reboot-trigger.txt file any time we change the epoch variable in the UI, save it and commit it. Thus any device in scope (here all) devices in group "Production" will receive a new .txt file and the "command-to-run" runs the reboot-script.sh again.

This is the content of the reboot-trigger.tmpl file that gets expanded to reboot-trigger.txt

reboot-trigger.tmpl

# unique run-once trigger, here epoch time {{epoch}}

Both files need to be in the same file distribution

Both files need to be in the same file distribution and the template option needs to be checked. The first time this executes immediately. The next time only when the number changes. This gives you full control.

!qbee-reboot-run-once

When will the command (in this case a reboot) happen?

The qbee agent has a run interval. As default this is 5 minutes. The run-once command will be executed the next time the device asks for its configuration. So if 5000 devices should reboot and the qbee interval is 5 minutes then all online devices will do so within 5 minutes. If the run interval is 6 hours it could take 6 hours until the command is issued on all devices. If a device is offline and comes online again in a year from now it would then reboot once before doing its duty. But it will only reboot once even if the reboot has been issued multiple times in between.

It is possible to do much more with templating. Please see also the distribution and running scripts section. There we show an example how you manage services allowing you to stop, start,restart,enable and disable.

In general, if you want to control that you can fire off scripts when you want just upload and distribute the script and then you add another template file only with a comment that can be changed. We will eventually simplify this. If you have problems with this concept please do not hesitate to reach out to us and we can explain this with an example.