How it works
backpulse is a dead-man’s switch. Instead of watching logs or exit codes, it watches for the check-in each scheduled job owes it — and raises the alarm when the check-in doesn’t arrive or reports failure.
The check-in contract
Section titled “The check-in contract”Each check has a schedule (a cron expression plus timezone and a grace
window). The job — directly or through backupctl — tells backpulse three
things:
/start: the run began./success: the run finished and the result verified./fail: the run finished but something was wrong.
From those three signals plus the schedule, the detector derives everything:
| Signal | Meaning |
|---|---|
No /start before the expected time | the scheduler or host failed |
/start with no completion | stuck, or the host disappeared |
/fail | the command or its result verification failed |
| Completion past the grace window | late |
Two /starts with no completion between them | overlapping or duplicate runs |
Verify, then report success
Section titled “Verify, then report success”The critical ordering: backupctl verifies the result before it reports
/success. A restic adapter demands a fresh snapshot; a file adapter demands
a recent, non-empty artifact. If verification fails, the run reports /fail
even though the wrapped command exited zero. And the command stays
authoritative — if backpulse itself is unreachable, pings only warn; a
monitoring outage never fails your backup.
Delivery is at-least-once
Section titled “Delivery is at-least-once”Detection and delivery are separate slices that never import each other. Detection enqueues one notification per channel; the delivery worker claims due rows under a lease, sends them, and records the outcome guarded so a stale worker can’t overwrite a recorded result. A crash between send and record can therefore deliver twice — receivers must tolerate duplicates. Failed sends retry with backoff, then dead-letter after the configured attempts so a broken destination stays visible instead of silently retrying forever.