Requires the Mender Monitor add-on package. See the Mender features page for an overview of all Mender plans and features.
This tutorial will walk you through how to monitor your device and its applications with Mender. We will be using the Monitor Add-on, allowing you to monitor various system parts.
To follow this tutorial and perform the examples, you will need to install the Monitor Add-on package and the Demo monitors package on your device. If you have followed the get started tutorial to prepare your device, the Monitor Add-on, and Demo packages should already be installed.
Verify installed dependencies on your device with:
sudo mender --version && sudo mender-monitorctl --version
The output should give no errors and print the version of both tools.
Check installed monitoring subsystems with:
ls -lAh /etc/mender-monitor/monitor.d | grep -v '^d'
-rwxr-xr-x 1 root root 2669 Aug 17 20:09 connectivity.sh
-rwxr-xr-x 1 root root 2298 Mar 10 09:37 dbus.sh
-rwxr-xr-x 1 root root 2498 Aug 18 01:14 diskusage.sh
-rwxr-xr-x 1 root root 3738 Mar 10 09:37 log.sh
-rwxr-xr-x 1 root root 3016 Mar 10 09:37 service.sh
Verify existence of check definitions with:
ls -lAh /etc/mender-monitor/monitor.d/available/ | grep -v '^d'
-rwxr-xr-x 1 root root 176 Aug 17 20:09 connectivity_example.sh
-rwxr-xr-x 1 root root 173 Aug 18 01:14 diskusage_root_space.sh
-rwxr-xr-x 1 root root 201 Aug 17 20:09 log_mender_client.sh
-rwxr-xr-x 1 root root 203 Aug 17 20:09 log_mender_connect.sh
-rwxr-xr-x 1 root root 297 Aug 17 20:09 log_usb_disconnect.sh
Demo Checks for the Mender Monitor Add-on, found in the Demo monitors package (Debian package) or in the
examples directory for Yocto serve as a starting point for quick evaluation.
Once ready, you can customize and define your own Checks.
By default both mail notifications and UI alerts occur once an monitored event happens. While going through the examples in this tutorial, watch the email inbox of your Mender user to see that you get notified about Alerts triggered and cleared on the device.
The default configuration for
mender-monitorctl command requires read-write access to the
/etc/mender-monitor directory, which on most systems means switching to super user or running with
For a read-only filesystem, it is essential to establish a symbolic link to a writable directory. This symlink is required to create, modify, and enable alert checks.
This demo shows detecting USB disconnects, alerting you to potential device tampering and preventing unauthorized access
Enable the check by running:
sudo mender-monitorctl enable log usb_disconnect
Now remove a USB device from the device (you can insert it first, e.g., a thumb drive or mouse, if you don't have any USB devices inserted). Once you remove the USB device, the log subsystem triggers an alert, which you can inspect in the device details in the Mender UI:
Note: This Alert will remain unless a manual alert cleaning is performed:
monitor_send_alert OK "Log file contains \[.*\] +usb [\w\-\.]+: USB disconnect" "\[.*\] +usb [\w\-\.]+: USB disconnect present in /var/log/kern.log" "log_usb_disconnect" LOGCONTAINS "log" "\[.*\] +usb [\w\-\.]+: USB disconnect" "/var/log/kern.log"
Running low on essential device resources like disk space, often due to growing log files, can disrupt the product's functionality. This example shows you how to monitor for high disk usage, allowing timely action to avoid downtime.
diskusage check for the root space partition called
root_space by running the following command:
sudo mender-monitorctl enable diskusage root_space
Check the current disk usage.
df -h /
# Filesystem Size Used Avail Use% Mounted on
# /dev/root 3.4G 1.4G 1.9G 42% /
With the check enabled an alert will be triggered when the root partition exceeds 75% (predefined threshold). To trigger the Alert, fill up the filesystem with a large file:
fallocate -l 10G ~/large-file
as the disk usage goes above the 75% threshold
df -h /
# Filesystem Size Used Avail Use% Mounted on
# /dev/root 3.4G 3.4G 0 100% /
an alert shows up in the UI.
An OK alert will be sent by the
diskusage monitoring subsystem if this
file is removed, and the disk space is less than the threshold.
Ongoing connectivity issues may cause the device application to hang or malfunction, disrupting the user experience or function of the product. This Alert is one example of how to detect connectivity issues.
Mender stores triggered Checks on the device. Therefore, even if Mender cannot send the Checks to the server immediately, you will be notified about triggered Checks once the device regains connectivity. This means that even during offline periods, Alterts are triggered.
connectivity check called
sudo mender-monitorctl enable connectivity example
This enabled a Check using the demo
connectivity monitoring subsystem, which sends HTTP HEAD
example.com, ensuring it is responding.
To trigger the Alert, let us stop the traffic to
redirecting the DNS resolver to localhost in
echo 127.0.0.1 example.com | sudo tee -a /etc/hosts
Which then triggers the Alert:
And when re-enabling the route to
sudo sed -i '/example/d' /etc/hosts
After connection to
example.com is restored, an OK Alert will show up in the UI:
Assume you want to monitor the state of a systemd service, and you wish to receive CRITICAL alerts if the service is not running, and OK alerts when it is back up.
Create a service that will count some seconds and then fail.
cat > countdown.sh << "EOF"
while [ $i -gt 0 ]; do
echo "INFO: $i seconds remaining"
echo "ERROR: Exiting with return 1"
chmod +x $PATH_TO_SCRIPT
cat > /etc/systemd/system/countdown.service << EOF
systemctl start countdown.service
You can confirm the countdown with
journalctl -fu countdown.service.
Now create and enable the check for the systemd
service monitoring subsystem:
sudo mender-monitorctl create service countdown systemd
sudo mender-monitorctl enable service countdown
Once the service exits, an alert is sent.
Restarting the service will send an ok Alert.
systemctl restart countdown
To stop the monitoring, disable and delete the check
sudo mender-monitorctl disable service countdown
sudo mender-monitorctl delete service countdown systemd
If you want to examine the low-level architecture of mender-monitor using this example follow this link.
The root account has complete control over the server, making it a target for unauthorized access. Monitoring these sessions helps in the early detection of security breaches, ensuring that any unauthorized access is quickly identified and addressed.
First, create a check for the
log monitoring subsystem named
auth_root_session by running
the following command:
sudo mender-monitorctl create log auth_root_session "Started User Manager for UID 0" /var/log/auth.log
sudo mender-monitorctl enable log auth_root_session
Not that the check is in place, change your user to
this will trigger an alert notifying you of the event that took place.
© 2024 Northern.tech AS