Solve old Let’s Encrypt Root Certificate Expiration problem

During the last few days many companies around the globe saw certificate verification failures. This was caused by an unfortunate chain of events. First of all the DST Root CA X3 certificate from Let’s Encrypt expired on September 30th, 2021. If your device has a ISRG Root X1 certificate installed this should not pose a problem since it will start to use this instead.

However, there are two special cases you need to be aware of:

• if clients of your API use OpenSSL they need to use OpenSSL 1.1.0 or later. Then the ISRG Root X1 is already in the trust store.
• if you use OpenSSL 1.0.0 to 1.0.2 and you have updated the trust store manually with ISRG Root X1 it will still not work. These devices unfortunately will default to an untrusted certificate chain thus rendering all certificates not valid.

So how can you solve the Let’s Encrypt root certificate expiry? There are basically four ways available how to fix the certificate problem. What is the correct one for you depends on your Linux distribution and use case:

• update the ca-certificates package
• run update-ca-certificates on the devices
• update OpenSSL to a version that fixes this (for Ubuntu LTS 16.04 you can use 1.0.2g-1ubuntu4.20)
• Blacklist the certificate with the id pkcs11:id=%c4%a7%b1%a4%7b%2c%71%fa%db%e1%4b%90%75%ff%c4%15%60%85%89%10 that causes the problem

How qbee can solve the Let’s Encrypt root certificate issue on your whole infrastructure

The first thing you can do is to update the ca-certificates package. With qbee you can just go into the configuration engine and select auto update of this package. Then all devices in scope will automatically deploy the new version of this package. Using advanced device automation also on embedded devices is a huge advantage as you can update and re-configure fleets with thousands of devices.

The second way to fix this is to run update-ca-certificates on all relevant devices through remote shell script execution. This can be achieved by distributing and executing a remote shell script on all these devices. The shell script just calls the command in verbose mode. The output will be picked up by the qbee agent and is reported back via the output logs. The script is very simple and will be distributed through qbee file distribution:

#!/usr/bin/env bash

update-ca-certificates -v

exit 0

The file distribution is shown below and the command to run looks as follows:
bash /home/pi/update-certificates.sh

In this case all devices show no change since they were already up-to-date. If the certificates were not up to date new ones would be flagged as added here.

Another way to check that is to use the analysis tab. Here you can for example explore which OpenSSL version the devices run. In our case all are above 1.0.2 and thus not impacted.

Above you see that some devices like the RPI5 have updates for OpenSSL. These can very easily be deployed with package management such that all devices will be brought to the same library version. This is the nice thing with automation and over-the-air package updates.

The last resort to solve the old Let’s Encrypt Root Certificate Expiration problem is to blacklist the certificate with the id
pkcs11:id=%c4%a7%b1%a4%7b%2c%71%fa%db%e1%4b%90%75%ff%c4%15%60%85%89%10
This can be done in various ways and we would like to ask you to find the correct way for your Linux distribution.

If you are interested to learn more about automated package management, over-the-air software updates and all the other features qbee can offer please feel free to reach out to us by chat or mail and book a meeting. We are happy to discuss your project and demo qbee.io.