Watchdog

[jvde-github]

AIS-catcher has long included an option to terminate the program if no messages are received for a specified period of time:

 AIS-catcher -T 10 nomsg_only

When combined with automatic restarting of the AIS-catcher service, this helps resolve certain issues. For example, if the USB device becomes stale or unresponsive, simply restarting the program can restore normal operation.

With the latest version, an additional option has been introduced: the system can automatically reboot the entire device if AIS-catcher needs to restart three times within 30 minutes.

To enable this feature, run:

sudo bash -c "$(curl -fsSL https://raw.githubusercontent.com/jvde-github/AIS-catcher/main/scripts/aiscatcher-install) --set-reboot-on-failure"

To disable it again:

sudo bash -c "$(curl -fsSL https://raw.githubusercontent.com/jvde-github/AIS-catcher/main/scripts/aiscatcher-install) --unset-reboot-on-failure"

This functionality is also available in the AIS-catcher-control tool.

On the General page, you can configure when AIS-catcher should restart if no messages are received.

On the System page, under System Actions, you can enable the automatic reboot feature.

This page will also show the watchdog status.

Behind the scenes, the script modifies the systemd service configuration to define an escalation path. When the feature is enabled, the configuration will look like this:

[Unit]
Description=AIS-catcher Service
After=network.target
OnFailure=ais-catcher-reboot.service
StartLimitIntervalSec=1800
StartLimitBurst=3

[Service]
User=aiscatcher
Group=aiscatcher
SupplementaryGroups=plugdev dialout
ExecStart=/bin/bash -c '/usr/bin/AIS-catcher -G system on -o 0 -C /etc/AIS-catcher/config.json $(/bin/grep -v "^#" /etc/AIS-catcher/config.cmd | /bin/grep -v "^[[:space:]]*$" | tr -d "\r" | /usr/bin/tr "\n" " ")'
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

[jvde-github]

I added a 5 minute wait for the actual reboot to give time for manual intervention. Note that after 3 restarts in 30 minutes systemd stops retrying and marks the service as failed — it will not attempt to restart again until either manually reset or the system reboot

In that period to stop the pending reboot: systemctl stop ais-catcher-reboot.service
To start AIS-catcher after failing and before the reboot, first reset it from failed state: sudo systemctl reset-failed.

Reboot script:

#!/bin/bash
# Wait up to 5 minutes for AIS-catcher to recover before rebooting.
# Checks every 10 seconds; aborts if the service comes back active
# (e.g. after manual "systemctl reset-failed + start" intervention).
# To cancel a pending reboot: systemctl stop ais-catcher-reboot.service
wall "AIS-catcher failed repeatedly - system rebooting in 5 minutes. To cancel: systemctl stop ais-catcher-reboot.service"
for i in $(seq 30); do
    sleep 10
    if systemctl is-active --quiet ais-catcher.service; then
        exit 0
    fi
    if [[ $i -eq 27 ]]; then
        wall "System rebooting in 30 seconds due to AIS-catcher failures. To cancel: systemctl stop ais-catcher-reboot.service"
    fi
done

[kit-powell]

Love this, I building a station for a friend that isn't particularly technical and somewhat remote, that will be nicer than telling them to unplug it and plug it back in over the phone