We removed the previously deprecated stable APT repository:
deb [arch=your-arch] https://downloads.mender.io/repos/debian stable mainIf you see errors of the form:
Err:3 https://downloads.mender.io/repos/debian stable InRelease
403 Forbidden [IP: 52.222.214.71 443]it means you are using the old repository, please update to the current one (see Set up the APT repository section).
In order to remove the obsolete repository deb [arch=your-arch] https://downloads.mender.io/repos/debian stable main
you can use the following command:
add-apt-repository -r "deb [arch=$(dpkg --print-architecture)] https://downloads.mender.io/repos/debian stable main"and then re-add the new one with:
add-apt-repository "deb [arch=$(dpkg --print-architecture)] https://downloads.mender.io/repos/debian debian/buster/stable main"In case you see the following errors during apt-get update:
E: Repository 'http://raspbian.raspberrypi.org/raspbian buster InRelease' changed its 'Suite' value from 'stable' to 'oldstable'
N: This must be accepted explicitly before updates for this repository can be applied. See apt-secure(8) manpage for details.In order to accept the change of the suite name, you need to run:
apt-get update --allow-releaseinfo-changeThe mender-client version 3.2.0 Debian package is deprecated. If you are getting installation errors, with a missing libffi6 dependency, then please install the new Debian package, as per the installation instructions in downloads
Logs are usually needed in order to diagnose an issue.
The Mender client by default logs to the system log using systemd, so the easiest way to retrieve logs
is to run the following command:
journalctl -u mender-clientPlease note that the default log level is Info. It is possible to increase the
verbosity by editing the Mender systemd unit file and add the --log-level debug option:
ExecStart=/usr/bin/mender --log-level debug daemonIn addition to system logging, Mender also writes debug logs directly to a file when a deployment starts. This file in turn gets uploaded to the server if the deployment fails.
By default, log files for the past 5 deployments are kept.
They are stored in /var/lib/mender/, named by the deployment id,
for example deployments.0001.fcd8bca2-6dae-488e-969e-23559c674ba5.log.
In order to see what the Mender client is doing currently, follow the log as it is being written with this command:
journalctl -u mender-client -fTo stop it use Ctrl+C.
The Mender client can not connect to the server, typically the first time it tries, and emits messages like the following to syslog at the device:
... level=info msg="Mender state: authorize-wait -> bootstrapped" module=mender
... level=error msg="authorize failed: transient error: authorization request failed: failed to execute authorization request:
Post https://<SERVER-URI>/api/devices/v1/authentication/auth_requests: x509: certificate has expired or is not yet valid" module=stateThis could occur in several places, and the distinguishing message is x509: certificate has expired or is not yet valid. Each TLS certificate has a validity period, Not Before and Not After, and this message means that the Mender client concludes that the current time is outside this range.
Most commonly this is caused by incorrect time setting on the device which runs the Mender client. Check this by
running date on the device, and make sure it is correct. Consult the section on Correct clock
for a more detailed discussion.
To determine the status of your time synchronization, execute the following:
# timedatectl statusNote that it can take some time after boot before the time synchronization is completed. If after 5-10 minutes, the time still has not synchronized, consult with your local network administrator for further troubleshooting.
If this is not the problem, you need to verify that the certificates you are using are valid. Replace the hostname with the one for your Mender API Gateway below and run the following command:
echo | openssl s_client -connect mender.example.com:443 2>/dev/null | openssl x509 -noout -dates
notBefore=Dec 14 19:52:46 2016 GMT
notAfter=Dec 12 19:52:46 2026 GMT
Also note that the storage proxy has its own certificate, and it runs on the same host as the API Gateway on port 9000 by default. Adjust the hostname and verify the validity of its certificate with the following command:
echo | openssl s_client -connect s3.example.com:9000 2>/dev/null | openssl x509 -noout -dates
notBefore=Dec 14 19:52:46 2016 GMT
notAfter=Dec 12 19:52:46 2026 GMT
We can see that both these certificates are currently valid. Also see the documentation on certificates for an overview and description on how to generate new certificates.
The Mender client can not connect to the server, typically the first time it tries, and emits messages like the following to syslog at the device:
... level=info msg="Mender state: authorize-wait -> bootstrapped" module=mender
... level=error msg="authorize failed :transient error :authorisation request failed: failed do execute authorisation request:
Post https://<SERVER-URI>/api/devices/v1/authentication/auth_requests: x509: certificate signed by unknown authority" module=stateThis could occur in several places, and the distinguishing message is x509: certificate signed by unknown authority. The message shows that the Mender client rejects the Mender server's certificate because it does not trust the certificate authority (CA).
If your server is using a certificate that is signed by an official Certificate Authority, then you likely need to update your client's root certificate store. For example, hosted Mender uses an official CA so the only reason your client would reject this is if it does not have updated root certificates in its system store.
On the other hand, if you set up the Mender server yourself as described in
Production installation and generated certificates as part of it,
your need to make sure that the server certificates are in /etc/mender/server.crt on your device.
To test that they match, run cat /etc/mender/server.crt on your device, and compare that to the output
of the following command, adjusting the hostnames mender.example.com / s3.example.com (ideally run on device, but can be run from elsewhere as well):
openssl s_client -showcerts -connect mender.example.com:443 < /dev/null 2>/dev/null | openssl x509 && openssl s_client -showcerts -connect s3.example.com:9000 < /dev/null 2>/dev/null | openssl x509If these mismatch, then you need to update /etc/mender/server.crt on your client.
You can do this manually for testing purposes, and you should
include the certificates in your Yocto Project build.
The Mender Client detected a self-signed certificate that is the only one in the chain and the same certificate can't be located in the trusted store. That means the OpenSSL is unable to verify the server identity. You have to either use another (not self-signed) certificate or include the certificate in the local trust store.
The key length for the end entity in the certificate chain is too short.
This can happen in conjunction with the security level setting in the OpenSSL
configuration file, and the actual key length. First check if the security level
setting is what you want it to be. Start by locating the openssl.cnf file,
by running:
# openssl version -d
OPENSSLDIR: "/opt/local/etc/openssl"In the above example you can find the configuration file
at /opt/local/etc/openssl/openssl.cnf and check the security level:
# cat `openssl version -d | sed -e 's/.*"\([^"]*\)".*/\1/'`/openssl.cnf | grep SECLEVEL
CipherString = DEFAULT@SECLEVEL=2A security level of 2 is the default one on many installations nowadays.
Starting with the Mender Client 2.4.0, the client uses OpenSSL and it is possible that you see this error with shorter keys and certain values of security level. You have two choices: make the keys longer, or decrease the security level.
_INCONSISTENT appended to itThe _INCONSISTENT suffix is appended to the software name on a device when the last Artifact deployment failed, and either the rollback also failed, or the particular Update Module being used has no rollback capability. As the name implies, in this case the device is in an inconsistent state, somewhere between two known states. In this case the deployment log of the last deployment may provide more information about what went wrong, and whether there is cause for concern.
When deploying an update with the Mender client, you see a log message similar to the following:
ERRO[0001] update install failed: failed to read and install update: reader: unsupported version: 2 module=stateThe problem here is most likely that you have built a new version of the Artifact format that your Mender Client does not support. It could also be that you are building a very old version of the Artifact format that your new version of the Mender Client does not support.
In either case the solution is to build a different version of the Artifact format that your Mender Client supports until you have upgraded all Mender Clients and can use the corresponding latest version of the Mender Artifact format.
You have the Mender binary on your device and try to trigger a rootfs update but you get output similar to the following:
mender install /media/rootfs-image-mydevice.mender
ERRO[0000] exit status 1 module=partitions
ERRO[0000] No match between boot and root partitions. module=mainThe problem here is most likely that the device does not have the partition layout Mender expects. This could have happened if you just placed the Mender binary into your rootfs, but did not reflash the entire storage device with the .sdimg. file output from the Yocto Project build. When this happens, output from mount and fw_printenv can confirm that this is the problem you are seeing. The solution is to flash your entire storage device with the .sdimg output from the Yocto Project build process.
If you are using the Mender client in demo mode, either by selecting it when running mender setup, or set up with the demo layer, the Mender client has more aggressive polling intervals to simplify testing.
See the documentation on building for production and polling intervals to reduce the network bandwidth usage.
For more specific troubleshooting issue please look at the troubleshooting section for the delta update module.
The delta mechanism makes use of the Provides and Depends.
The block below shows 3 example artifacts.
+-------------------------------+
|Type: rootfs-image |
|Version: v1 |
|Checksum: 5bb84175 |
| |
|Provides |
|rootfs-image.checksum: 5bb84175| -> matches the Depends for the delta
+-------------------------------+
+--------------------------------+ +--------------------------------+
|Type: mender-binary-delta | |Type: rootfs-image |
|Version: v2 | |Version: v2 |
|Checksum: ff532419 | |Checksum: b9147deb5 |
| | | |
|Provides | |Provides |
|rootfs-image.checksum: b9147deb5| |rootfs-image.checksum: b9147deb5|
| | +--------------------------------+
|Depends: |
|rootfs-image.checksum: 5bb841755|
+--------------------------------+v1
rootfs-image type artifact - contains the entire partition contentchecksum and rootfs-image.checksum
v2 mender-binary-delta
mender-binary-delta type artifact - contains only a delta (binary difference between two payloads)rootfs-image.checksum: 5bb841755 defines the checksumchecksum and rootfs-image.checksum differ
checksum - checksum of the delta payloadrootfs-image.checksum - checksum of the payload once it's running on the devicev2 rootfs-image
rootfs-image type artifact - it contains the entire partition contentrootfs-image.checksum: b9147deb5 - same as the v2 mender-binary-deltamender-artifact read <mender-artifact.mender>mender show-providesReleases -> Select Release -> Expand the artifact info by clicking it -> Expand Provides and Depends© 2024 Northern.tech AS