Zero-downtime upgrades (systemd)¶

ShinyHub hands new connections to a successor during a compatible upgrade. Existing app sessions keep using the retiring proxy until they close or its drain deadline expires; sessions still open at that deadline are disconnected. On SIGHUP the running control plane re-execs the new binary, hands off its listening sockets to the successor, drains in-flight WebSocket sessions, releases the ownership lease, and exits. App processes keep running throughout (server.shutdown_apps: adopt, the default).

One-time setup¶

1. Install the binary at /usr/local/bin/shinyhub (a real built binary; go run cannot perform graceful reloads).
2. Set a PID file in shinyhub.yaml:

   server:
     pid_file: /run/shinyhub/shinyhub.pid
   

3. Install deploy/systemd/shinyhub.service, then systemctl daemon-reload and systemctl enable --now shinyhub.

Upgrading¶

Special procedure: 0.12.x to 0.13.0¶

The schedule-provenance upgrade is an intentional exception to the normal zero-downtime procedure. Use a maintenance window and stop the 0.12.x control plane; do not use systemctl reload for this version boundary. Version 0.13.0 installs a database admission fence that prevents a still-running 0.12.x server from launching another unattributed schedule process.

1. While 0.12.x is still running, inventory every enabled schedule with on_success = "roll". Version 0.12.x allowed these schedules on local Docker; 0.13.0 requires every data-producing schedule to use effective worker_isolation = "multiplex" and a local native runtime. Move each app to that topology, or disable/remove the producer policy before upgrading.
2. Replace every manifest run_on_register declaration with an explicit deploy_trigger: normally bundle_change for a cache that must match its bundle, or first_deploy for a genuine one-time bootstrap.
3. Let all in-flight schedule runs finish, verify their terminal state with shinyhub schedule runs, then stop the old control plane with systemctl stop shinyhub. Confirm no scheduled-job process or descendant is left on the host; reboot if that is the only reliable way to establish the process fence.
4. Install 0.13.0 and start the service. Startup validates every enabled stored producer before serving. An error names the app, schedule, isolation, tier, or runtime that still needs correction. Stop the service again before correcting a startup error so Restart=on-failure cannot retry around your maintenance work.
5. If startup reports legacy schedule writer fence, keep all old servers and the new service stopped, verify the listed process trees are gone, and run the new binary's offline resolver:

systemctl stop shinyhub
shinyhub resolve-legacy-schedule-writers
shinyhub resolve-legacy-schedule-writers --acknowledge-processes-stopped
systemctl start shinyhub

The first invocation is read-only and lists the affected app, schedule, and run IDs. The acknowledged invocation records conservative data uncertainty; it does not declare the cache safe. 6. Reapply the updated manifests after startup. Migrated schedules default to deploy_trigger = "never" because 0.12.x did not persist run_on_register. If the resolver marked a producer as requiring repair, rerun that exact schedule under the new fence and wait for success:

shinyhub schedule run APP SCHEDULE --follow

Check shinyhub apps show APP and shinyhub schedule ls APP; do not return the app to service while compatibility_quarantined or producer_repair_required remains true.

For later compatible releases, use the normal handoff:

# 1. Replace the binary in place with the new version.
install -m 0755 ./shinyhub /usr/local/bin/shinyhub
# 2. Trigger the zero-downtime handoff.
systemctl reload shinyhub

A continuous client sees no connection-refused gap; in-flight sessions drain on the old process for up to server.reload_drain_timeout (defaults to server.drain_timeout, normally 60s) before any straggler is force-closed.

For a longer upgrade drain without extending app-version retirement:

server:
  drain_timeout: 10m
  reload_drain_timeout: 8h

This setting is read by the retiring process at its own startup. Editing the file immediately before reload changes the successor's future drain budget, not the already-running parent's budget.

This postpones the upgrade disconnect; it does not transfer existing sessions to the successor. The retiring proxy continues checking session authorization throughout its drain. A SIGINT/SIGTERM shortens a reload drain to at most the ordinary drain_timeout; it does not reset an earlier reload deadline.

For stable single-host native multiplex apps, session relays can remove the reload deadline entirely:

server:
  drain_timeout: 10m
  reload_sessions: preserve
  reload_relay_limit: 4

This is opt-in. Configure it before starting the process whose sessions you want to retain; the already-running parent's settings govern that handoff. The old process keeps its existing WebSocket tunnels, closes its database and lifecycle control plane, and retires its session lease when its final session closes. If it launched native backends, it keeps only their stdout/stderr readers and process reapers alive until those backends exit, preserving logs without database access. Adopted backends do not create another output reader on each reload. A ready successor reads sealed leases for the exact backend processes, counts them for admission and hibernation, and performs authorization and usage persistence for the relays. Later reloads can proceed while earlier relays remain alive. Login expiration, access revocation and ordinary service stop still close sessions as usual.

A preserving reload is refused while an app operation, candidate or previous generation is still draining, or when any pool uses elastic isolation. The current process keeps serving; inspect the upgrade log and retry after the transition finishes. At the relay limit, another preserving reload is refused instead of disconnecting older sessions, reserving capacity even if a new session arrives during handoff. Close old tabs to reclaim capacity. The limit bounds retained session relays, not their age or output-only processes still needed by native backends. Retained tunnels continue running the old proxy code, so use an ordinary service restart when a security fix must replace every running proxy immediately.

Compatible app deployments and rolling restarts may run after the control plane has retired. Their own drain_timeout still applies to retained backend sessions; preserving a hub reload does not preserve a session across replacement of its app process. Stop-first changes remain fenced while relayed sessions exist. Reloading during an app-generation drain is intentionally unsupported. This mode preserves WebSocket transport; XHR fallback transport and recovery after an owner crash are outside its continuity guarantee.

Relay IPC uses the private app-data registry when its path fits the Unix socket limit, then a short private RUNTIME_DIRECTORY if available. For unusually long paths it falls back to private /tmp/shinyhub-relay-* directories; exclude those live directories from age-based temporary-file cleanup. Removing a live socket interrupts authorization checks and closes retained sessions after the verification grace period.

Do not replace the binary with a legacy build while relays are alive. The side-effect-free relay-protocol preflight refuses a handoff to a binary lacking relay support. Auth-secret rotation invalidates relay RPC authentication and ends retained sessions after the fail-closed authorization grace (at least 90s). Runtime leases live in the private .reload-relays directory beneath app-data; a private, uid-scoped local Unix socket carries owner RPCs. Relay protocols use frozen versioned identity and usage fields, so future upgrades must retain that compatibility while older relays remain alive.

Run the real Shiny acceptance test with SHINYHUB_E2E_SESSION_RELAYS=1 node scripts/browser-lifecycle-e2e.mjs. It holds reactive sessions across repeated reloads, probes page loads, checks the relay limit, deploys a replacement, and revokes access to a relayed session.

Tunables (shinyhub.yaml)¶

Notes & limits¶

• Repeated upgrades in drain mode. The handoff refuses another upgrade while the successor's parent is still alive. A long reload drain therefore delays the next upgrade; wait for the retiring proxy to exit before reloading again. It also retains that process's memory and database connection.
• Backend protection during handoff. The successor cannot count sessions still flowing through its parent. While that parent is alive, idle hibernation and lifecycle mutations such as deploy, stop, resize and non-rolling restart are fenced until the parent exits and recovery cleanup finishes. Rolling restarts are accepted into the durable queue and wait. This deliberately trades lifecycle availability for session continuity in drain mode. Session-preserving relays supply cross-process accounting after their control plane retires.
• Multiplex scope. Multiplex workers can be re-adopted. Elastic workers (grouped and per_session) have process-local client bindings and are rebuilt during recovery; a longer proxy drain does not preserve their backend sessions.
• Service stop budget. Set systemd TimeoutStopSec above your ordinary drain_timeout plus HTTP and ownership shutdown (allow at least another minute). For a ten-minute drain, a twelve-minute stop budget is appropriate. The supervisor's termination policy can close sessions sooner than ShinyHub.
• App changes remain separate. A deploy or schedule roll may retire the backend an old proxy connection uses before the reload drain ends. The longer reload budget does not extend that backend's generation deadline.
• Failure is safe. If the successor fails to start within upgrade_timeout, the old process keeps serving - the upgrade simply does not happen.
• Startup preflight. The handoff re-execs the process's own argv[0], so at boot the server checks that argv[0] resolves to an executable (the same PATH lookup the handoff uses) and logs a warning when it does not. With the warning present every SIGHUP fails safe with the current process still serving; fix how the service launches the binary (an absolute ExecStart path, or a name resolvable on the unit's PATH) before relying on reloads.
• Database migrations must be backward-compatible and non-blocking. During the handoff window both versions briefly run against the same SQLite database. The successor applies any new migrations at startup while the previous version is still serving on the old schema. An upgrade that adds migrations must therefore use additive / expand-contract changes (the previous version keeps working against the new schema) - never a destructive rename or column drop in the same release - and must avoid long-running/locking migrations (SQLite runs them in a transaction that can block the old process); split large backfills or table rewrites out of the upgrade and run them separately afterward. Same-version restarts apply no migrations and are always safe.
• A rollback point is taken automatically. Before applying pending migrations the successor copies the SQLite database aside as <dsn>.pre-migration-v<version>-<timestamp>.sqlite, and logs the path. It is written only when migrations are pending, and older snapshots are pruned down to a configurable retention count afterward. See Schema migrations on startup for the opt-out and the Postgres equivalent.
• Rolling back to an older binary needs the snapshot. An older build refuses to serve a database a newer one migrated, exiting 7 (schema_incompatible); the unit sets RestartPreventExitStatus=7 so systemd stops rather than restart-loops. Put back the pre-migration snapshot along with the old binary, following Getting back to the older build
• a snapshot is moved into place, not fed to shinyhub restore, and its -wal sidecar has to go or the rollback silently undoes itself.
• systemd MAINPID. With Type=notify, ShinyHub sends READY=1 plus MAINPID=<own pid> on startup and after each handoff, so systemd retargets the main PID to the successor. The unit sets Restart=on-failure (not Restart=always): a genuine crash is restarted, but the original PID's deliberate exit 0 after a handoff is a clean success, so systemd does not fight it. After your first systemctl reload, verify with systemctl show -p MainPID shinyhub that it matches the new PID file.
• Scope. This is the systemd/VM path and covers all app runtimes (native, Docker, Fargate, remote-worker). Multi-pod Kubernetes rolling upgrades are part of the separate high-availability project.
• Platform. Linux and macOS only (tableflip does not support Windows).