Load Testing ShinyHub¶

ShinyHub ships a k6-based load-test harness under loadtest/. Two scenarios provide the evidence for the platform's key performance claims:

• cold-start: real wake latency from a hibernated app (< 15 s p95 claim)
• sessions: concurrent WebSocket sessions at scale (1 000-session claim)

All load tests are make targets; no pipeline-specific tooling is needed. They are run manually against a live server (they need a running instance and the k6 binary), not in CI. Run the same make load-test invocation locally, and add ASSERT=1 to fail the run when a threshold is missed.

Prerequisites¶

Install k6 (v2.0+, confirmed with v2.0.0):

brew install k6           # macOS
# or: https://k6.io/docs/get-started/installation/

A running ShinyHub server and a deployed app are required. Quick local setup:

make dev                  # starts server on :8080
shinyhub deploy /tmp/demo-shiny-app --slug demo

Running the tests¶

Sessions scenario (default)¶

Ramps to LT_SESSIONS concurrent WebSocket connections, holds each for LT_HOLD seconds, then reports established counts and connect latencies.

make load-test LT_SLUG=demo

With explicit parameters:

make load-test LT_SLUG=demo LT_SESSIONS=200 LT_RAMP=30s LT_HOLD=20s

Reproduce the 1 000-session evaluation claim (needs adequate hardware):

make load-test LT_SLUG=demo LT_SESSIONS=1000 LT_RAMP=60s LT_HOLD=30s

Cold-start scenario¶

Triggers hibernation wake and measures elapsed time until the app serves real content (not the loading page), using the /.shinyhub/ready probe as the primary signal.

make load-test LT_SCENARIO=cold-start LT_SLUG=demo

First hibernate the app to make the cold-start meaningful:

shinyhub apps set --slug demo --hibernate-timeout 1  # 1-minute idle timeout
# wait for the app to hibernate, then:
make load-test LT_SCENARIO=cold-start LT_SLUG=demo

Both scenarios¶

make load-test LT_SCENARIO=both LT_SLUG=demo

CI usage with ASSERT=1¶

Pass ASSERT=1 to turn on k6 thresholds. The run exits non-zero if the thresholds are not met, gating CI on the performance claims.

# cold-start p95 < 15 s
make load-test LT_SCENARIO=cold-start LT_SLUG=demo ASSERT=1

# session established rate >= 99%
make load-test LT_SLUG=demo LT_SESSIONS=100 ASSERT=1

# both thresholds in one run
make load-test LT_SCENARIO=both LT_SLUG=demo ASSERT=1

Parameter reference¶

All parameters are optional except LT_SLUG.

WebSocket paths by framework¶

The sessions scenario was designed and evaluated against Python Shiny (server- sends-first). The framework caveat is also noted in the script header of loadtest/sessions.js.

Private apps¶

For apps with access: private or access: shared, obtain a session cookie by logging in through the UI or via POST /api/auth/login, then pass the full Cookie header value:

make load-test LT_SLUG=demo \
  LT_AUTH_COOKIE="shinyhub_session=<your-session-token>"

The cookie is forwarded on both the initial HTTP GET (which collects the sticky-routing cookie) and the WebSocket upgrade.

How to read the results¶

Headline output¶

Each scenario prints a single headline line to stdout:

COLD START: http=3.42s, session=3.45s (slug=demo, host=http://127.0.0.1:8080)
SESSIONS: established 198/200 (99.0%), established p95=4ms

The cold-start scenario reports two numbers: - http: time from first GET (wake trigger) until the app root returns a real 200 response whose body is NOT the ShinyHub loading page. This is the point at which a browser user would see actual app content. - session: time from first GET through a completed WebSocket handshake and first server frame received. This is time-to-usable-session - the evaluation claim threshold applies here.

Why the ready probe is not used for polling: /.shinyhub/ready returns {"ready":true} only after a completed WebSocket handshake (IsWSReady in internal/proxy/proxy.go). A freshly woken app serving HTTP content but without any WS handshake will return 503 indefinitely if nothing opens a WS, so polling the ready probe alone cannot work. The probe IS used once at startup as a fast existence check - it correctly returns 404 for unknown slugs regardless of WS state.

For a warm app (min_warm_replicas >= 1), stage 1 completes in one round trip (sub-100ms) and stage 2 is one WS connect - both numbers give the warm-floor baseline.

The sessions line shows how many VUs received the first server WebSocket frame within LT_FIRST_MSG_TIMEOUT. A mere TCP connection is not counted as established - the first server message must arrive. This matches the real user experience: a Shiny session is not active until the server sends its init frame.

Two timing metrics are recorded per VU: - ws_connect_ms: time from connectStart to the 101 Upgrade (open event) - ws_established_ms: time from connectStart to the first server frame

The headline prints established p95 from ws_established_ms - the more meaningful number for real users. Both metrics use k6/ws (blocking callback API) which drives the event loop inline, giving accurate sub-millisecond local timings. The newer k6/websockets async module fires callbacks only after a blocking sleep returns, making connect timings unreliable.

JSON results¶

Full k6 metric data is written to loadtest/results/<ISO-ts>-<scenario>.json after each run. The directory is gitignored. Use these files for trend analysis or to share results without re-running.

Evaluation claims¶

For the 1 000-session run, ensure the OS open-file limit is high enough:

ulimit -n 65536
make load-test LT_SLUG=demo LT_SESSIONS=1000 LT_RAMP=60s LT_HOLD=30s

Results directory¶

loadtest/results/ is gitignored (all files except .gitignore). Results are local to the machine that ran the test. Copy them out before deleting the working tree if you want to keep them.