Level has powerful built-in monitor types, however there are hundreds and thousands of device characteristics that an IT team might want to monitor. In response to this, we have created a custom monitoring type called "Run Script". When this type is selected, a script can be chosen for Level to run on the devices and evaluate if there is a metric outside of a healthy threshold.
A script can be a one-liner, or a complex series of health checks. Regardless of the size of the script, a script-based monitor simply reads the console output of the script and will evaluate if the device is in an errored state. The error state is determined by which values are checked on the monitor and can be a numerical comparison, or a check whether a string is present.
Video Walkthrough
First Create a Script
Before jumping into creating a script-based monitor, first create the script that will query for the information in question. Any of the supported scripting languages can be used (PowerShell, Bash, Zsh, Osquery, etc).
Pro Tip: Osquery works on Windows, Mac, and Linux, so a single script might be able to cover all operating systems! Check if there is an Osquery table that contains what you're looking for.
Ideally, the script will query for a specific value or state that Level can use to determine if the monitor item is in a healthy state or not. For example, if we want to query the state of the Windows firewall, we can run the following PowerShell command.
PS C:\> get-netfirewallprofile | select Enabled Enabled ------- True True True
In this case we see each of the 3 Windows Firewall profiles (Domain, Private, and Public) are enabled and display "True". If any value was "False" then this would indicate the firewall is disabled on a profile. Now we can paste this command into a Level script.
Here is the command in a script
Create the Monitor
Either create a monitor policy or edit an existing monitor policy. Add a new monitor to the policy and give it a useful name (this name will appear in the alert). In our firewall example we will use "Windows Firewall is disabled". Choose the type "Run Script" and set the desired severity for your alerts. Select the script that was created, in our example, it's called "Check Windows Firewall".
In the "Script output" dropdown there are several options that can be used for matching the script output.
In our case we want to match if the script output contains "False", so we chose "Contains" from the dropdown and entered "False" as the value. Greater than and less than are useful for comparing a numeric value to the threshold value.
The "Run frequency" will determine how frequently the script is run (in minutes or hours). We caution that if a script is run too frequently it could cause performance issues on the device.
The trigger count determines how many consecutive runs the matched value is present.
All remaining settings are the same as what can be found in the standard monitor types.
The Results
Now that the monitor is in place, the script will begin running at the set frequency on all targeted devices. If the value is ever matched on a device, then an alert will be generated. If you would like to test the monitor on the targeted devices in order to see the returned output of the script, then open the monitor and select "Test Script". This will open a new window with a new job that already has the script and devices selected. Simply press "Execute Script" and it will run. Press the expand icon at the far right of the device list to see the live output.
An alert with the warning threshold
Best practices and recommendations
Attempt to keep the logic that determines if a script is in a healthy or unhealthy state in the monitor. For example query for a numerical value or a state (true/false, yes/no, enabled/disabled) and then choose what to match in the monitor.
In complex monitors where many things need to be checked, then the logic can reside in the script. Consider outputting a message to the console like "Alert" and then exit the script with a failure (exit 1). In the monitor simply choose Contains "Alert" and that will trigger the alert.
A Script-based monitor evaluates all the output of a script, not just the first or last line. Keep this in mind when considering what to output to the console.