Self-hosted monitor for Slurm jobs and training runs, with push alerts.
2.0K
Every Slurm job on every cluster you use, live on your phone. Self-hosted monitoring for HPC jobs and training runs, with push alerts when something needs you.
Quick start · Deploy · Connect a cluster · Configuration · Self-hosting guide
If you train models on HPC clusters, you know the loop: ssh, squeue, tail -f,
repeat, on every cluster where you have an allocation. You find out a job died hours after
it did, and the loss curve lives in a TensorBoard you can't open from your phone.
Pulse does that loop for you. It watches every job on every account you link, reads the training metrics your jobs already write, and sends a push notification when a job starts, finishes, fails, stalls, runs out of time, or its loss goes NaN. It installs on your phone's home screen like an app.
It's built to be easy to run and kind to clusters:
loss=… lines your code already produces.Jobs, live
Training metrics, without code changes
*metric*.jsonl and *metric*.csv logs;step=… loss=…-style lines in the job's output;hparams.yaml and config.* (W&B's config.yaml included).train/…, val/…), plus loss sparklines in the job list.Runs and sweeps
Alerts that matter
| Alert | When |
|---|---|
| Started, finished, failed, cancelled | The job's Slurm state changes |
| Near time limit | It has used 90% of its time limit (adjustable) |
| Stalled | Its output log hasn't changed for 15 minutes (adjustable) |
| Loss is NaN | The loss becomes NaN or infinite |
| Diverging | The smoothed loss climbs 50% above its best value (adjustable) |
| Plateau | The loss hasn't improved for 60 minutes (adjustable) |
Clusters
authorized_keys. It pins each server's host key on first connect.ssh -J, ProxyJump): paste the same command you'd type, and Pulse
tunnels through the gateway.Sign-in and security
On your phone
Try it with a simulated cluster, no HPC access needed:
docker run --rm -p 8000:8000 -e PULSE_SLURM_BACKEND=mock neecat/pulse
Open http://localhost:8000, enter the setup code printed in the container's output,
and create your account. Then add any account under Accounts (for example
[email protected]): the simulated cluster fills it with jobs, training curves and
sweeps within a minute.
You need a machine that can reach your clusters over SSH, with Docker and Compose, and a domain name pointing at it.
git clone https://github.com/neecatt/pulse.git && cd pulse
cp .env.example .env # set PULSE_DOMAIN and PULSE_ADMIN_PASSWORD
docker compose up -d # Pulse plus Caddy, which gets the HTTPS certificate
Open https://<your domain>, sign in, and add your clusters.
Important
**HTTPS is required** for push notifications, passkeys and installing on a phone. Compose sets it up with Caddy. No domain? The [self-hosting guide](https://github.com/neecatt/pulse/blob/main/docs/deployment.md) covers a private setup with Tailscale, running behind your own proxy, and running without Docker. Behind a proxy, set `PULSE_TRUSTED_PROXIES` so Pulse sees the real scheme.
Warning
**Keep `/data` on a persistent volume.** It holds the database, _Pulse's SSH key_ and its push keys. Lose it and you'll have to add the new SSH key on every cluster again, and turn notifications back on on every device.
Push notifications work out of the box: Pulse creates its push keys on first start. Open Settings → Notifications and allow them. On an iPhone, add Pulse to the home screen first (Share → Add to Home Screen) and enable notifications from the installed app.
Upgrade with docker compose pull && docker compose up -d. The database schema updates
itself.
Open Accounts → Add account and paste the login you already use, the way you'd type it
after ssh:
[email protected]
ssh -p 2222 [email protected]
ssh -J gateway.example.org [email protected]
Then choose how Pulse connects:
| How it works | Use it when | |
|---|---|---|
| Direct | Pulse signs in over SSH with its own key (through a jump host if you gave one) and runs read-only Slurm commands. | You can ssh in with a key and no 2FA prompt |
| Via agent | You paste one command on the login node. A small script then reports to Pulse over HTTPS. | The cluster requires 2FA or short-lived SSH certificates |
Pulse generates an Ed25519 key in /data on first start. Settings → Connection shows the
public key with a ready-to-paste command:
echo 'ssh-ed25519 AAAA… pulse' >> ~/.ssh/authorized_keys
Run it on the cluster, or upload the key wherever your centre manages SSH keys (some use a web portal, which usually covers the jump host too). Then press Connect: Pulse tests the login straight away and pins the host key. With a jump host, Pulse's key must be accepted on the gateway as well as the login node.
Pulse uses its own key, never yours. To use an existing key instead, set
PULSE_SSH_KEY_PATH.
Pulse gives you one command to run on the login node. It installs ~/.pulse-agent.py and a
cron entry that reports every minute. The agent only sends raw sacct/scontrol output, log
tails and metric-file chunks; the server does all the parsing.
python3 ~/.pulse-agent.py status, and read errors in
~/.config/pulse-agent/agent.log.python3 ~/.pulse-agent.py uninstall.What Pulse runs on the cluster:
sacct --json for your jobs, every poll;scontrol and stat/tail on your jobs' output and metric files, reading only new bytes;srun step that samples CPU, memory and
nvidia-smi on each node (Slurm 20.11 or newer).It's always read-only, and all of it runs over one SSH connection that stays open.
Everything is set with PULSE_* environment variables, from .env or from Docker secrets
in /run/secrets. Every setting is optional.
| Variable | What it does | Default |
|---|---|---|
PULSE_ADMIN_USERNAME, PULSE_ADMIN_PASSWORD | Admin account. The password follows the variable, so change it there and restart. Unset: create the admin in the browser with the logged setup code | (unset) |
PULSE_DOMAIN | Public hostname Caddy gets a certificate for (compose only) | — |
PULSE_PUBLIC_URL | Pins the address passkeys belong to | the request's host |
PULSE_TRUSTED_PROXIES | Proxies allowed to set X-Forwarded-*; * behind a platform proxy | 127.0.0.1 |
PULSE_VAPID_PUBLIC_KEY, PULSE_VAPID_PRIVATE_KEY | Your own push keys (from pulse gen-vapid); only needed to move them between servers | generated in /data |
PULSE_VAPID_SUBJECT | Contact sent with push messages | PULSE_PUBLIC_URL |
PULSE_ALLOW_SIGNUP | Let anyone with the URL create an account | false |
PULSE_SESSION_DAYS | How long a sign-in lasts | 30 |
PULSE_SLURM_BACKEND | ssh for real clusters, mock for a simulated one | ssh in the image |
PULSE_SSH_KEY_PATH | Use an existing private key instead of the generated one | /data/pulse_ed25519 |
PULSE_SSH_KNOWN_HOSTS | Strict host-key checking against a known_hosts file | pin on first use |
PULSE_POLL_INTERVAL_LIVE_SECONDS | Poll interval while someone has Pulse open | 10 |
PULSE_POLL_INTERVAL_IDLE_SECONDS | Poll interval otherwise | 60 |
PULSE_NODE_STATS_INTERVAL_SECONDS | Per-node usage sampling (each sample starts a job step) | 60 |
PULSE_STALL_MINUTES, PULSE_TIME_LIMIT_WARNING_RATIO, PULSE_DIVERGE_PERCENT, PULSE_PLATEAU_MINUTES | Default alert thresholds; users can change their own in the app | 15, 0.9, 50, 60 |
PULSE_PORT (or PORT) | Port Pulse listens on | 8000 |
The full list, with SSH timeouts and cookie settings, is in
backend/.env.example.
Command line: pulse serve, pulse create-user <name>, pulse gen-vapid and
pulse reset-password <name>, which also signs that user out everywhere. In Docker, use
docker compose exec pulse pulse ….
flowchart LR
phone["Phone / browser<br/>(PWA)"] -- "HTTPS · REST + live updates (SSE)" --> pulse
push["Push services<br/>(Apple, Google, Mozilla)"] -- "notifications" --> phone
subgraph host ["Your server · one container"]
pulse["Pulse<br/>FastAPI + web app"] --- db[("SQLite<br/>/data")]
end
pulse -- "Web Push" --> push
pulse -- "SSH · sacct, log tails" --> login["Login node"]
pulse -- "SSH via jump host" --> gw["Gateway"] --> login2["Login node"]
agent["Agent on a<br/>2FA login node"] -- "HTTPS reports" --> pulse
backend/: Python 3.12, FastAPI and async SQLAlchemy on SQLite. A poller visits each
account, turns state changes into events, sends Web Push, and streams one live update per
cycle to every open app. pulse/slurm/ is the only code that knows about Slurm.frontend/: React 19, Vite, TanStack Router and Query, and Tailwind. The charts are
plain SVG with no chart library. It's built into the image and served by the same process,
so the whole app sits on one origin.backend/src/pulse/agent/: the single-file agent for clusters Pulse can't reach directly.Everything lives in /data (a Docker volume):
| File | Contents |
|---|---|
pulse.db | Users, linked accounts, jobs, metrics, events, push subscriptions, passkey public keys |
pulse_ed25519, pulse_ed25519.pub | Pulse's SSH key. Keep it private, and back it up |
vapid.json | Pulse's push keys. Every device's notifications depend on them |
Back up /data and you've backed up everything. Passkey private keys never leave your
devices. Pulse stores only hashes of passwords and session tokens.
Needs Python 3.12+ and bun.
make install # dependencies for both packages
make dev # API on :8000 and the web app with hot reload on :5173, on a simulated cluster
make check # lint, type checks and tests, as CI runs them
make demo # build and run the Docker image locally
backend/ and frontend/ are independent packages that talk only over the HTTP API.
AGENTS.md describes the architecture and conventions, and
CONTRIBUTING.md explains how to propose a change and how releases work.
neecat/pulse on Docker Hub, mirrored to
ghcr.io/neecatt/pulse, for amd64 and arm64:
| Tag | Contents |
|---|---|
latest, 0.1, 0.1.0 | Releases. Pin a version for predictable upgrades |
edge | The newest code on main |
Pulse is developed with Claude Code, Anthropic's coding
agent. Much of the code, tests and documentation were drafted in Claude Code sessions, and
AGENTS.md holds the project context those sessions start from.
Pulse is free and stays free: MIT licensed, no paid tier, nothing held back. If it saves you
a few ssh … squeue round trips and you'd like to say thanks, you can buy me a coffee. A
star, a bug report or a pull request helps just as much.
Bug reports and pull requests are welcome; start with CONTRIBUTING.md. Please report security issues privately, as described in SECURITY.md.
MIT. Use it, change it and share it; keep the copyright notice.
Content type
Image
Digest
sha256:8d4bb8cb4…
Size
80.8 MB
Last updated
about 5 hours ago
docker pull neecat/pulse