Why is centralized logging important?
All organizations involved in some form of IT operations are dependent on observability tools that gather metrics and system logs in order to understand whether or not the fleet of devices is operating correctly. In a cloud environment with short lived infrastructure centralized logging has become a must-have, but aggregating logs also has a clear use case in an operating environment that depends on more static infrastructure. IoT devices are usually intended to be in the field for a significant amount of time, but might change over time as they would need to be updated with security patches, business software, configuration settings etc. Log aggregation therefore plays a crucial role in the lifecycle management of IoT devices. Some use cases include:
- Detection of bugs in business software
- Potential break in attempts
- Auditing of outages
This blog post will show how you can manage a log aggregation in a simple way using the qbee.io platform in conjunction with AWS and the Cloudwatch observability agent on Debian based devices. However, the same concepts apply to any other cenralized logging solution.
All files used in this demo are available on Github under this link.
Configuring your AWS account
In order to authenticate the devices against AWS and also allow them to push logs, you would need to set up a user with a permissions profile. In the AWS console navigate to IAM -> Users and click “Add users” in the upper right corner. In this example we use cloudwatch-agent-iot as username and attach the CloudWatchAgentServerPolicy to the user. Last screen should look similar to the following:
Once the user has been created, you need to attach some credentials to it. We do that by going to IAM -> Users, select the user we just created and “Create access key” under “Security credentials”. We choose “Application running outside AWS” and click “Next”.
At the end of the process with note down the AWS_ACCESS_KEY_ID and the corresponding AWS_SECRET_ACCESS_KEY. These we need to use in our qbee configuration.
Preparing application files
We will need a couple of application files in order for cloudwatch agent to work correctly, so we create the following credentials file.
[default]
region = eu-west-1
output = json
aws_access_key_id = <AWS_ACCESS_KEY_ID>
aws_secret_access_key = <AWS_SECRET_ACCESS_KEY>
Here we need to change replace the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY with the values recorded previously.
The application would need to know where to find its credentials file, so we also need a common-config.toml holding that information.
[credentials]
shared_credential_profile = "default"
shared_credential_file = "/opt/aws/amazon-cloudwatch-agent/etc/credentials"
In addition, we would need a file describing which log files track and ingest into AWS. This example will we therefore create a amazon-cloudwatch-agent.json.
{
"agent": {
"run_as_user": "root"
},
"logs": {
"logs_collected": {
"files": {
"collect_list": [
{
"file_path": "/var/log/syslog",
"log_group_name": "iot-syslog",
"log_stream_name": "{local_hostname}",
"retention_in_days": 7
}
]
}
}
}
}
Here we are sending entries from /var/log/syslog to the log group iot-syslog with the hostname as log stream name.
We also need to fetch the application itself from https://s3.amazonaws.com/amazoncloudwatch-agent/ubuntu/amd64/latest/amazon-cloudwatch-agent.deb.
We upload these files to the qbee file manager and set up our configuration using “Software management”.
NB! Software management in qbee relies on strict debian naming conventions. We therefore need to change the package name to include the version number and the architecture. This is done by renaming the package file to match the package metadata.
$ dpkg -I amazon-cloudwatch-agent.deb
new Debian package, version 2.0.
size 39704626 bytes: control archive=706 bytes.
56 bytes, 1 lines conffiles
289 bytes, 10 lines control
799 bytes, 19 lines * preinst #!/bin/sh
222 bytes, 7 lines * prerm #!/bin/sh
Package: amazon-cloudwatch-agent
Architecture: amd64
Version: 1.247359.1b252618-1
Section: admin
Depends: libc6
Priority: optional
Copyright: MIT License. Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
Maintainer: Amazon.com, Inc.
Description: Amazon CloudWatch Agent
$ mv amazon-cloudwatch-agent.deb amazon-cloudwatch-agent_1.247359.1b252618-1_amd64.deb
We upload the four files we now have into the qbee file manager under the /cloudwatch directory.
Setting up the qbee-agent policies
We now move on to setting up the qbee-agent configuration policies by navigating to Configuration -> Software management in the qbee UI and selecting the relevant tag/group that are target for the configuration. We point to the package file we just uploaded under package “Debian software package”, leave service name empty (as it is the same as the package name in this case). Then we set up the three configuration files necessary for the application to work.
We would also need to set up a dummy parameter as the qbee templating engine expects at least one to be set even though there are no qbee parameters in the files that would need expanding.
The configuration screen then looks something like this:
Configuration then needs to be saved and committed before it’s active for the devices. We now wait for the qbee-agent to check in and apply its configuration. Once it runs we’ll see some messages related to the configuration policies in the qbee device report view.
We should now be able to see the syslog log entries in the AWS cloudwatch management console.
Changing the qbee-agent configuration
A key strength of qbee is that it attempts continuously to converge the target device to a configured state. For software management that means that it will monitor the configured service. If it crashes it is restarted and a log message is sent to the qbee log. If a new version of the package is uploaded to the file manager it will automatically upgrade it and also revert back configuration if that changed locally on the disk.
We can also do configuration changes in the UI. A typical example is a change of configuration files or parameters. This will result in a rewrite of the config files and a restart of the service. Again, this will be logged by qbee.
Let’s assume we want the log stream name to be static. We therefore change the amazon-cloudwatch-agent.json to this:
{
"agent": {
"run_as_user": "root"
},
"logs": {
"logs_collected": {
"files": {
"collect_list": [
{
"file_path": "/var/log/syslog",
"log_group_name": "iot-syslog",
"log_stream_name": "my-log-stream",
"retention_in_days": 7
}
]
}
}
}
}
We upload the new file to the qbee file manager and wait for the new configuration to be picked up. We now see this in the qbee report logs for the device:
New configuration has been applied and the service has been restarted. We should now see syslog lines under a different log stream name in the cloudwatch console:
Entries from /var/log/syslog are now appended to the new log stream.
Conclusion
Configuration management is a must have for most IT based operations. Qbee offers a very agile agent with a very low resources footprint in order to enable organizations to confidently and securely manage their IT infrastructure.
