In Linux environments, especially those using systemd, some services create unit files instead of traditional services. Unit files describe systemd units, which can be services, timers, or other types of resources that systemd manages. Monitoring these unit files is crucial for ensuring the stability and functionality of services, but it works a bit differently compared to monitoring standard services.
Why Unit Files May Show as "Exited" or "Dead"
A common issue arises when certain systemd services, like WireGuard, report their status as Active (exited) or Dead, even though they are functioning as expected. This occurs because these services are designed to exit after initialization and run in the background without staying active as typical services do. As a result, a service monitor might mistakenly detect the service as "down."
For example, if you are monitoring wg-quick (the WireGuard service), systemd might show it as Exited, but that doesn’t necessarily mean the service has failed. In such cases, the systemd SubState should be checked for a more accurate status representation.
Monitoring Unit Files with Scripts
To monitor these types of unit files effectively, it's better to use script-based monitoring instead of a traditional service monitor. Script-based monitors allow you to customize how the unit’s state is checked. This way, you can account for services that behave like WireGuard, where the service shows as exited but is still running properly.
Here’s an example of a command that can help identify the actual state of a systemd unit file:
systemctl show -p SubState --value <unit_name>
This command provides the SubState, which gives a finer-grained detail about the unit's status. For instance, it may show "exited," which could still indicate the service is functioning correctly.
Implementing a Script-Based Monitor
You can set up a script-based monitor in Level to check the status of unit files more reliably:
Create a custom script that runs the
systemctlcommand mentioned above.Configure the monitor in Level to execute this script periodically, ensuring that you’re monitoring the actual state of the unit, not just the high-level service status.
Here’s a basic script example for monitoring a unit file:
#!/bin/bash
UNIT_NAME="wg-quick@wg0"
STATUS=$(systemctl show -p SubState --value $UNIT_NAME)
if [ "$STATUS" = "exited" ]; then
echo "Service is running correctly."
else
echo "Service is not running."
fi
By using a script like this, Level can report the real-time status of a unit file, ensuring that services like WireGuard are monitored accurately, even when they don't behave like traditional services.
Best Practices for Monitoring Unit Files
Avoid monitoring using service monitors for unit files that frequently show
Active (exited)or similar states. Use script-based monitors instead.Check both the ActiveState and SubState to get a full picture of the service's status.
Customize your monitor configuration in Level to ensure you're catching the actual state of these services and not triggering false alerts.
By following these practices, you can ensure that your Linux-based services, especially those managed by systemd unit files, are correctly monitored and functioning as expected.