From b3146e0862ee541e1e0075db831de23712a72725 Mon Sep 17 00:00:00 2001 From: Noel Sabia Date: Sat, 14 Nov 2026 06:43:30 +0100 Subject: [PATCH] docs: record 120-client SSE load, train and fallback results Add staffing timing to the integration harness and commit the verified run: small train 171ms, forecast train 3.6s, SSE propagation 15ms, 120/120 clients for 300s with 0 drops and p95 5.6ms, 20 heartbeats at 15s, FastAPI-down fallback from the snapshot, and recovery in 1.6s. --- .dockerignore | 2 + README.md | 4 +- docs/integration.md | 143 +++++++++++++++++++++++++ docs/results/fallback.json | 19 ++++ docs/results/forecast-after-small.json | 1 + docs/results/load.json | 131 ++++++++++++++++++++++ docs/results/load.txt | 21 ++++ docs/results/meta-initial.json | 1 + docs/results/sse-stream.log | 13 +++ docs/results/sse.json | 9 ++ docs/results/staffing.json | 10 ++ docs/results/summary.txt | 7 ++ docs/results/train-forecast.json | 77 +++++++++++++ docs/results/train-small.json | 27 +++++ scripts/integration.sh | 17 +++ 15 files changed, 481 insertions(+), 1 deletion(-) create mode 100644 docs/integration.md create mode 100644 docs/results/fallback.json create mode 100644 docs/results/forecast-after-small.json create mode 100644 docs/results/load.json create mode 100644 docs/results/load.txt create mode 100644 docs/results/meta-initial.json create mode 100644 docs/results/sse-stream.log create mode 100644 docs/results/sse.json create mode 100644 docs/results/staffing.json create mode 100644 docs/results/summary.txt create mode 100644 docs/results/train-forecast.json create mode 100644 docs/results/train-small.json diff --git a/.dockerignore b/.dockerignore index 10ead56..f26a26f 100644 --- a/.dockerignore +++ b/.dockerignore @@ -2,5 +2,7 @@ .gitignore *.md opencode.json +docs +scripts # Fixtures are for tests only; the runtime image fetches FastAPI /bootstrap. testdata diff --git a/README.md b/README.md index 9b9fece..81396bc 100644 --- a/README.md +++ b/README.md @@ -61,7 +61,9 @@ scenario latency exceeds the contract's 2 s threshold. Results land in `/tmp/pulsecast-integration/results` by default. Override with `WORKDIR=/path` or `RESULTS_DIR=/path`. `make load` runs only the load test -against an API that is already up. +against an API that is already up. The verified run is written up in +[`docs/integration.md`](docs/integration.md) with raw evidence under +`docs/results/`. ## Container diff --git a/docs/integration.md b/docs/integration.md new file mode 100644 index 0000000..12796ac --- /dev/null +++ b/docs/integration.md @@ -0,0 +1,143 @@ +# Integration and load-test report + +Run: **2026-11-14**, Go API worktree `feat/integration`. +FastAPI: real ML service from `ML-py.git/main` on `127.0.0.1:18000`. +Go API: this worktree on `127.0.0.1:18080`, `FASTAPI_URL=http://127.0.0.1:18000`. +Everything below is reproducible with `make integration-load`; raw evidence is +in `docs/results/`. + +## Command + +```sh +WORKDIR=/tmp/pulsecast-full-run RESULTS_DIR="$PWD/docs/results" \ + scripts/integration.sh --load +``` + +The script owns `:18000` and `:18080` (it refuses to run if either is busy). +It uses a 24,636-row CSV subset (`2026-09-01 .. 2026-10-16`, all 8 sensors, +`densityData` rows only) generated from the raw challenge CSV into `/tmp`. +The subset is test input only; it is not committed and not baked into the image. + +## Train end-to-end (real FastAPI, through the Go proxy) + +Both jobs were uploaded as multipart CSV to `POST /api/train` with control auth, +then polled through `GET /api/train/{job_id}`. + +| Job | Result | Model version | Submit to terminal | Headline metrics | +|---|---|---|---|---| +| `small` | `succeeded`, `error: null` | `logit-20261114-053410` | **171 ms** (POST itself 25 ms) | accuracy 0.824, precision 0.589, recall 0.420, F1 0.490, ROC-AUC 0.849, n_train 5,363, n_test 1,728, 8 sensors | +| `forecast` | `succeeded`, `error: null` | `histgb-20261114-053413` | **3,571 ms** | MAE 4.83, WAPE 33.0, R² 0.703, n_eval 4,919 | + +- `small` is a logistic-regression busy classifier; `forecast` retrains the + global HistGB. Both ran in FastAPI; the Go process only proxied and polled. +- The forecast job trained on the **small subset**, so its MAE 4.83 is not the + baked model's quality (3.32 on the full data). It proves the upload, queue, + poll and hot-swap path, not model quality. +- `/api/forecast?sensor=hn-stadtgalerie-01&horizon=7d&scenario=base` returned + **200, 168 points, `cached: false`** immediately after the small job. +- After the forecast hot-swap, the Go snapshot refresh picked up the new + `model_version` (`histgb-20261114-053413`) in `/api/meta`. + +## SSE scenario, heartbeat and reconnect + +| Check | Result | +|---|---| +| Connect payload | `retry: 3000`, current `scenario` event immediately | +| Scenario propagation | **15 ms** from `POST /api/scenario` to the client event (limit: 2 s) | +| Heartbeat | received during the check; server interval exactly **15,000 ms** throughout the load test | +| Reconnect | closed and reopened the stream; current scenario (v2) arrived in **18 ms** | +| Event shape | `event: scenario`, `id: `, JSON body with `version`, `scenario`, `at` | + +## Staffing plan generation (contract: < 2 s) + +A follow-up run on the same commit (ports `18100`/`18180`, because another +verification process held `18000`/`18080`) timed `GET /api/staffing`: + +| Check | Result | +|---|---| +| Cold call (live `/predict` + plan build) | **19 ms** | +| Warm call (cached by `sensor|scenario|standard|version|weather`) | **< 1 ms** | +| Plan size | 70 store hours (7 days × 10:00–20:00), 21 shift blocks | +| Coverage | 100%, cost €3,360 | +| `cached` on the live plan | `false` | + +The repeat run also reproduced the other checks: small train 169 ms, scenario +propagation 13 ms, reconnect 16 ms, recovery 1,339 ms. + +## 120-client / 5-minute load test + +`scripts/sse-load`: 120 concurrent streams, 300.008 s, scenario change every +30 s (rotating `sunny-saturday-market`, `rainy-saturday`, `weekday-event`, +`base`). + +| Metric | Result | +|---|---| +| Peak connections | **120 / 120** | +| Connected after warmup | 120 / 120 | +| Dropped or reconnected streams | **0** | +| Scenario changes | 9 | +| Scenario events received | **1,080 / 1,080** (120 per change) | +| Missing scenario events | **0** | +| Scenario latency samples | 1,080 | +| p50 | **1.3 ms** | +| p95 | **5.6 ms** | +| p99 | **5.8 ms** | +| max | **5.812 ms** | +| Heartbeat pings | 20 unique server timestamps, every **15,000 ms** (min = median = p95 = max) | +| Heartbeats per client | 20 / 20 / 20 (min / median / max) | + +Per-change p95 ranged from 1.2 ms to 5.8 ms; the highest latencies were the +first change after warmup. The tool exits non-zero if a client drops, an event +is missing, or p95 exceeds the contract's 2 s limit; this run passed. + +## Fallback and recovery (runtime cache) + +Sequence: warm a live base forecast, stop FastAPI, observe the cache failure, +serve the snapshot/last-good data, restart FastAPI, wait for recovery. + +While FastAPI was down: + +| Check | Result | +|---|---| +| `/api/meta.cache.failures` | **1** | +| `/api/meta.cache.last_error` | `ml: /bootstrap: Get "http://127.0.0.1:18000/bootstrap": dial tcp 127.0.0.1:18000: connect: connection refused` | +| `/api/meta.cache.next_retry_at` | `2026-11-14T05:34:31Z` (1 s after the first failure; schedule is 1/2/4/8/16/30 s) | +| `/api/meta.cache.last_success_at` | `2026-11-14T05:34:20Z` (unchanged) | +| `/api/readyz` | **200** — a loaded snapshot keeps readiness even with FastAPI down | +| `/api/sensors` | **200**, 8 sensors, from the last good snapshot | +| `/api/context` | **200**, `cached: true` | +| `/api/series` | **200**, from the last good snapshot | +| `/api/forecast` (warm key) | **200**, 168 points, **`cached: true`** | +| `/api/staffing` | **200**, built from the last good forecast, **`cached: true`** | +| `/api/forecast` (never-warmed key) | **503** `forecast_unavailable` — no invented data | + +After restarting FastAPI: + +| Check | Result | +|---|---| +| Cache recovery | **1,622 ms** after restart; `failures: 0` | +| `last_success_at` | advanced from `05:34:20Z` to `05:34:33Z` | +| `/api/forecast` | **200**, `cached: false` again | + +The integration script used `BOOTSTRAP_REFRESH=10s` to keep the observation +window short. The production default is 60 s; the failure path, the +1/2/4/8/16/30 s retry schedule and the recovery semantics are unchanged. + +## Evidence files + +- `docs/results/train-small.json`, `train-forecast.json` — raw job responses and timings +- `docs/results/staffing.json` — cold/warm plan timing from the follow-up run +- `docs/results/sse.json`, `sse-stream.log` — propagation, heartbeat, reconnect +- `docs/results/load.json`, `load.txt` — all load-test samples and per-change results +- `docs/results/fallback.json` — exact cache/fallback/recovery values +- `docs/results/summary.txt` — one-screen summary of the same run +- `docs/results/meta-initial.json`, `forecast-after-small.json` — baseline evidence + +## Notes and blockers + +- At one point `:18000` was held by another agent's ML end-to-end run. The test + was not started against that process; it was rerun once the port was free. + No blocker remains. +- FastAPI was trained on a subset, so the retrained forecast metrics are lower + than the baked full-data model. The baked model is restored on FastAPI + restart, and recovery was verified with that baked model. diff --git a/docs/results/fallback.json b/docs/results/fallback.json new file mode 100644 index 0000000..27e6fb2 --- /dev/null +++ b/docs/results/fallback.json @@ -0,0 +1,19 @@ +{ + "fastapi_down": { + "cache_failures": 1, + "last_error": "ml: /bootstrap: Get \"http://127.0.0.1:18000/bootstrap\": dial tcp 127.0.0.1:18000: connect: connection refused", + "next_retry_at": "2026-11-14T05:34:31Z", + "last_success_at": "2026-11-14T05:34:20Z", + "forecast_cached": true, + "staffing_cached": true, + "readyz": 200, + "sensors": 8 + }, + "recovery": { + "after_ms": 1622, + "last_success_before": "2026-11-14T05:34:20Z", + "last_success_after": "2026-11-14T05:34:33Z", + "forecast_cached": false, + "failures": 0 + } +} diff --git a/docs/results/forecast-after-small.json b/docs/results/forecast-after-small.json new file mode 100644 index 0000000..a866bac --- /dev/null +++ b/docs/results/forecast-after-small.json @@ -0,0 +1 @@ +{"sensor":"hn-stadtgalerie-01","horizon":"7d","scenario":"base","points":[{"t":"2026-11-14T06:00:00Z","v":4.35,"lo":0,"hi":9.23},{"t":"2026-11-14T07:00:00Z","v":8.07,"lo":3.25,"hi":12.95},{"t":"2026-11-14T08:00:00Z","v":23.45,"lo":18.63,"hi":28.33},{"t":"2026-11-14T09:00:00Z","v":35.8,"lo":30.98,"hi":40.68},{"t":"2026-11-14T10:00:00Z","v":67.45,"lo":62.62,"hi":72.32},{"t":"2026-11-14T11:00:00Z","v":86.07,"lo":81.25,"hi":90.95},{"t":"2026-11-14T12:00:00Z","v":88.9,"lo":84.08,"hi":93.78},{"t":"2026-11-14T13:00:00Z","v":87.06,"lo":82.23,"hi":91.94},{"t":"2026-11-14T14:00:00Z","v":88.1,"lo":83.27,"hi":92.97},{"t":"2026-11-14T15:00:00Z","v":84.2,"lo":79.37,"hi":89.07},{"t":"2026-11-14T16:00:00Z","v":85.68,"lo":80.86,"hi":90.56},{"t":"2026-11-14T17:00:00Z","v":76.15,"lo":71.33,"hi":81.03},{"t":"2026-11-14T18:00:00Z","v":45.34,"lo":40.51,"hi":50.21},{"t":"2026-11-14T19:00:00Z","v":20.09,"lo":15.26,"hi":24.97},{"t":"2026-11-14T20:00:00Z","v":13,"lo":8.17,"hi":17.87},{"t":"2026-11-14T21:00:00Z","v":7.36,"lo":2.54,"hi":12.24},{"t":"2026-11-14T22:00:00Z","v":5.2,"lo":0.37,"hi":10.07},{"t":"2026-11-14T23:00:00Z","v":4.64,"lo":0,"hi":9.52},{"t":"2026-11-15T00:00:00Z","v":3.25,"lo":0,"hi":8.13},{"t":"2026-11-15T01:00:00Z","v":2.35,"lo":0,"hi":7.22},{"t":"2026-11-15T02:00:00Z","v":2.46,"lo":0,"hi":7.34},{"t":"2026-11-15T03:00:00Z","v":1.78,"lo":0,"hi":6.66},{"t":"2026-11-15T04:00:00Z","v":1.78,"lo":0,"hi":6.65},{"t":"2026-11-15T05:00:00Z","v":1.77,"lo":0,"hi":6.65},{"t":"2026-11-15T06:00:00Z","v":2.16,"lo":0,"hi":7.04},{"t":"2026-11-15T07:00:00Z","v":2.71,"lo":0,"hi":7.59},{"t":"2026-11-15T08:00:00Z","v":4.92,"lo":0.09,"hi":9.79},{"t":"2026-11-15T09:00:00Z","v":6.51,"lo":1.68,"hi":11.39},{"t":"2026-11-15T10:00:00Z","v":6.41,"lo":1.58,"hi":11.28},{"t":"2026-11-15T11:00:00Z","v":12.24,"lo":7.41,"hi":17.11},{"t":"2026-11-15T12:00:00Z","v":17.2,"lo":12.38,"hi":22.08},{"t":"2026-11-15T13:00:00Z","v":17.44,"lo":12.61,"hi":22.31},{"t":"2026-11-15T14:00:00Z","v":17.05,"lo":12.23,"hi":21.93},{"t":"2026-11-15T15:00:00Z","v":15.94,"lo":11.12,"hi":20.82},{"t":"2026-11-15T16:00:00Z","v":16.44,"lo":11.61,"hi":21.32},{"t":"2026-11-15T17:00:00Z","v":14.1,"lo":9.27,"hi":18.97},{"t":"2026-11-15T18:00:00Z","v":8.92,"lo":4.1,"hi":13.8},{"t":"2026-11-15T19:00:00Z","v":4.78,"lo":0,"hi":9.65},{"t":"2026-11-15T20:00:00Z","v":2.49,"lo":0,"hi":7.36},{"t":"2026-11-15T21:00:00Z","v":1.6,"lo":0,"hi":6.47},{"t":"2026-11-15T22:00:00Z","v":1.84,"lo":0,"hi":6.72},{"t":"2026-11-15T23:00:00Z","v":1.69,"lo":0,"hi":6.57},{"t":"2026-11-16T00:00:00Z","v":1.59,"lo":0,"hi":6.47},{"t":"2026-11-16T01:00:00Z","v":1.57,"lo":0,"hi":6.45},{"t":"2026-11-16T02:00:00Z","v":1.57,"lo":0,"hi":6.45},{"t":"2026-11-16T03:00:00Z","v":1.57,"lo":0,"hi":6.45},{"t":"2026-11-16T04:00:00Z","v":1.61,"lo":0,"hi":6.48},{"t":"2026-11-16T05:00:00Z","v":2.59,"lo":0,"hi":7.46},{"t":"2026-11-16T06:00:00Z","v":7.05,"lo":2.22,"hi":11.93},{"t":"2026-11-16T07:00:00Z","v":14.22,"lo":9.39,"hi":19.09},{"t":"2026-11-16T08:00:00Z","v":18.83,"lo":14.01,"hi":23.71},{"t":"2026-11-16T09:00:00Z","v":28.14,"lo":23.31,"hi":33.01},{"t":"2026-11-16T10:00:00Z","v":39.14,"lo":34.31,"hi":44.01},{"t":"2026-11-16T11:00:00Z","v":51.34,"lo":46.52,"hi":56.22},{"t":"2026-11-16T12:00:00Z","v":50.18,"lo":45.35,"hi":55.05},{"t":"2026-11-16T13:00:00Z","v":50.22,"lo":45.4,"hi":55.1},{"t":"2026-11-16T14:00:00Z","v":50.3,"lo":45.47,"hi":55.17},{"t":"2026-11-16T15:00:00Z","v":49.68,"lo":44.85,"hi":54.56},{"t":"2026-11-16T16:00:00Z","v":51.29,"lo":46.46,"hi":56.16},{"t":"2026-11-16T17:00:00Z","v":43.09,"lo":38.26,"hi":47.96},{"t":"2026-11-16T18:00:00Z","v":21.34,"lo":16.51,"hi":26.22},{"t":"2026-11-16T19:00:00Z","v":11.95,"lo":7.12,"hi":16.82},{"t":"2026-11-16T20:00:00Z","v":5.95,"lo":1.13,"hi":10.83},{"t":"2026-11-16T21:00:00Z","v":2.98,"lo":0,"hi":7.85},{"t":"2026-11-16T22:00:00Z","v":2.19,"lo":0,"hi":7.07},{"t":"2026-11-16T23:00:00Z","v":2,"lo":0,"hi":6.87},{"t":"2026-11-17T00:00:00Z","v":1.92,"lo":0,"hi":6.79},{"t":"2026-11-17T01:00:00Z","v":1.68,"lo":0,"hi":6.55},{"t":"2026-11-17T02:00:00Z","v":1.59,"lo":0,"hi":6.46},{"t":"2026-11-17T03:00:00Z","v":1.59,"lo":0,"hi":6.46},{"t":"2026-11-17T04:00:00Z","v":2.4,"lo":0,"hi":7.28},{"t":"2026-11-17T05:00:00Z","v":3.38,"lo":0,"hi":8.25},{"t":"2026-11-17T06:00:00Z","v":8.12,"lo":3.29,"hi":12.99},{"t":"2026-11-17T07:00:00Z","v":16.28,"lo":11.45,"hi":21.16},{"t":"2026-11-17T08:00:00Z","v":17.6,"lo":12.77,"hi":22.47},{"t":"2026-11-17T09:00:00Z","v":27.47,"lo":22.65,"hi":32.35},{"t":"2026-11-17T10:00:00Z","v":35.28,"lo":30.45,"hi":40.15},{"t":"2026-11-17T11:00:00Z","v":50.47,"lo":45.64,"hi":55.34},{"t":"2026-11-17T12:00:00Z","v":48.11,"lo":43.28,"hi":52.98},{"t":"2026-11-17T13:00:00Z","v":42.24,"lo":37.41,"hi":47.11},{"t":"2026-11-17T14:00:00Z","v":49.05,"lo":44.22,"hi":53.92},{"t":"2026-11-17T15:00:00Z","v":48.19,"lo":43.37,"hi":53.07},{"t":"2026-11-17T16:00:00Z","v":47.31,"lo":42.48,"hi":52.18},{"t":"2026-11-17T17:00:00Z","v":32.93,"lo":28.1,"hi":37.8},{"t":"2026-11-17T18:00:00Z","v":24.91,"lo":20.09,"hi":29.79},{"t":"2026-11-17T19:00:00Z","v":13.29,"lo":8.46,"hi":18.17},{"t":"2026-11-17T20:00:00Z","v":5.77,"lo":0.94,"hi":10.64},{"t":"2026-11-17T21:00:00Z","v":4.17,"lo":0,"hi":9.05},{"t":"2026-11-17T22:00:00Z","v":1.94,"lo":0,"hi":6.82},{"t":"2026-11-17T23:00:00Z","v":1.95,"lo":0,"hi":6.83},{"t":"2026-11-18T00:00:00Z","v":1.83,"lo":0,"hi":6.7},{"t":"2026-11-18T01:00:00Z","v":1.58,"lo":0,"hi":6.46},{"t":"2026-11-18T02:00:00Z","v":1.58,"lo":0,"hi":6.46},{"t":"2026-11-18T03:00:00Z","v":1.67,"lo":0,"hi":6.55},{"t":"2026-11-18T04:00:00Z","v":1.92,"lo":0,"hi":6.8},{"t":"2026-11-18T05:00:00Z","v":2.9,"lo":0,"hi":7.77},{"t":"2026-11-18T06:00:00Z","v":5.36,"lo":0.54,"hi":10.24},{"t":"2026-11-18T07:00:00Z","v":15.32,"lo":10.49,"hi":20.2},{"t":"2026-11-18T08:00:00Z","v":17.07,"lo":12.25,"hi":21.95},{"t":"2026-11-18T09:00:00Z","v":25.12,"lo":20.3,"hi":30},{"t":"2026-11-18T10:00:00Z","v":33.01,"lo":28.18,"hi":37.88},{"t":"2026-11-18T11:00:00Z","v":50.17,"lo":45.35,"hi":55.05},{"t":"2026-11-18T12:00:00Z","v":42.2,"lo":37.38,"hi":47.08},{"t":"2026-11-18T13:00:00Z","v":42.64,"lo":37.81,"hi":47.51},{"t":"2026-11-18T14:00:00Z","v":47.76,"lo":42.93,"hi":52.63},{"t":"2026-11-18T15:00:00Z","v":48.18,"lo":43.36,"hi":53.06},{"t":"2026-11-18T16:00:00Z","v":49.32,"lo":44.49,"hi":54.19},{"t":"2026-11-18T17:00:00Z","v":39.98,"lo":35.15,"hi":44.85},{"t":"2026-11-18T18:00:00Z","v":28.17,"lo":23.35,"hi":33.05},{"t":"2026-11-18T19:00:00Z","v":11.3,"lo":6.47,"hi":16.17},{"t":"2026-11-18T20:00:00Z","v":6.6,"lo":1.77,"hi":11.47},{"t":"2026-11-18T21:00:00Z","v":3.65,"lo":0,"hi":8.53},{"t":"2026-11-18T22:00:00Z","v":2.36,"lo":0,"hi":7.24},{"t":"2026-11-18T23:00:00Z","v":1.95,"lo":0,"hi":6.83},{"t":"2026-11-19T00:00:00Z","v":1.59,"lo":0,"hi":6.46},{"t":"2026-11-19T01:00:00Z","v":1.57,"lo":0,"hi":6.45},{"t":"2026-11-19T02:00:00Z","v":1.57,"lo":0,"hi":6.45},{"t":"2026-11-19T03:00:00Z","v":1.57,"lo":0,"hi":6.45},{"t":"2026-11-19T04:00:00Z","v":1.92,"lo":0,"hi":6.8},{"t":"2026-11-19T05:00:00Z","v":2.65,"lo":0,"hi":7.52},{"t":"2026-11-19T06:00:00Z","v":7.04,"lo":2.22,"hi":11.92},{"t":"2026-11-19T07:00:00Z","v":12.83,"lo":8,"hi":17.71},{"t":"2026-11-19T08:00:00Z","v":19.23,"lo":14.41,"hi":24.11},{"t":"2026-11-19T09:00:00Z","v":28.34,"lo":23.52,"hi":33.22},{"t":"2026-11-19T10:00:00Z","v":36.97,"lo":32.15,"hi":41.85},{"t":"2026-11-19T11:00:00Z","v":53.4,"lo":48.58,"hi":58.28},{"t":"2026-11-19T12:00:00Z","v":53.67,"lo":48.85,"hi":58.55},{"t":"2026-11-19T13:00:00Z","v":53.73,"lo":48.91,"hi":58.61},{"t":"2026-11-19T14:00:00Z","v":54.23,"lo":49.41,"hi":59.11},{"t":"2026-11-19T15:00:00Z","v":53.88,"lo":49.05,"hi":58.76},{"t":"2026-11-19T16:00:00Z","v":53.71,"lo":48.88,"hi":58.58},{"t":"2026-11-19T17:00:00Z","v":46.42,"lo":41.6,"hi":51.3},{"t":"2026-11-19T18:00:00Z","v":26.08,"lo":21.26,"hi":30.96},{"t":"2026-11-19T19:00:00Z","v":13.55,"lo":8.72,"hi":18.43},{"t":"2026-11-19T20:00:00Z","v":6.28,"lo":1.45,"hi":11.16},{"t":"2026-11-19T21:00:00Z","v":5.28,"lo":0.45,"hi":10.15},{"t":"2026-11-19T22:00:00Z","v":2.74,"lo":0,"hi":7.62},{"t":"2026-11-19T23:00:00Z","v":2.55,"lo":0,"hi":7.43},{"t":"2026-11-20T00:00:00Z","v":1.82,"lo":0,"hi":6.7},{"t":"2026-11-20T01:00:00Z","v":1.76,"lo":0,"hi":6.63},{"t":"2026-11-20T02:00:00Z","v":1.74,"lo":0,"hi":6.62},{"t":"2026-11-20T03:00:00Z","v":1.8,"lo":0,"hi":6.68},{"t":"2026-11-20T04:00:00Z","v":1.83,"lo":0,"hi":6.71},{"t":"2026-11-20T05:00:00Z","v":2.65,"lo":0,"hi":7.52},{"t":"2026-11-20T06:00:00Z","v":6.49,"lo":1.66,"hi":11.36},{"t":"2026-11-20T07:00:00Z","v":13.36,"lo":8.53,"hi":18.23},{"t":"2026-11-20T08:00:00Z","v":17.32,"lo":12.5,"hi":22.2},{"t":"2026-11-20T09:00:00Z","v":29.24,"lo":24.41,"hi":34.11},{"t":"2026-11-20T10:00:00Z","v":41.96,"lo":37.14,"hi":46.84},{"t":"2026-11-20T11:00:00Z","v":58.65,"lo":53.83,"hi":63.53},{"t":"2026-11-20T12:00:00Z","v":66.25,"lo":61.43,"hi":71.13},{"t":"2026-11-20T13:00:00Z","v":61.12,"lo":56.29,"hi":65.99},{"t":"2026-11-20T14:00:00Z","v":63.46,"lo":58.63,"hi":68.33},{"t":"2026-11-20T15:00:00Z","v":68.23,"lo":63.4,"hi":73.11},{"t":"2026-11-20T16:00:00Z","v":67.18,"lo":62.36,"hi":72.06},{"t":"2026-11-20T17:00:00Z","v":54.66,"lo":49.83,"hi":59.53},{"t":"2026-11-20T18:00:00Z","v":42.25,"lo":37.42,"hi":47.12},{"t":"2026-11-20T19:00:00Z","v":17.59,"lo":12.77,"hi":22.47},{"t":"2026-11-20T20:00:00Z","v":10.13,"lo":5.31,"hi":15.01},{"t":"2026-11-20T21:00:00Z","v":5.79,"lo":0.96,"hi":10.67},{"t":"2026-11-20T22:00:00Z","v":3.83,"lo":0,"hi":8.71},{"t":"2026-11-20T23:00:00Z","v":2.13,"lo":0,"hi":7.01},{"t":"2026-11-21T00:00:00Z","v":1.83,"lo":0,"hi":6.7},{"t":"2026-11-21T01:00:00Z","v":2.13,"lo":0,"hi":7.01},{"t":"2026-11-21T02:00:00Z","v":1.89,"lo":0,"hi":6.77},{"t":"2026-11-21T03:00:00Z","v":2.13,"lo":0,"hi":7.01},{"t":"2026-11-21T04:00:00Z","v":1.76,"lo":0,"hi":6.64},{"t":"2026-11-21T05:00:00Z","v":2,"lo":0,"hi":6.88}],"cached":false,"data_version":"2026-11-07T23:51:00Z","model_version":"histgb-2026-11-14","context_version":"2026-11-14T03:47:09Z"} \ No newline at end of file diff --git a/docs/results/load.json b/docs/results/load.json new file mode 100644 index 0000000..668f754 --- /dev/null +++ b/docs/results/load.json @@ -0,0 +1,131 @@ +{ + "started_at": "2026-11-14T05:34:33.422221537Z", + "duration_s": 300.008, + "base_url": "http://127.0.0.1:18080", + "clients_configured": 120, + "clients_peak_connected": 120, + "clients_connected_after_warmup": 120, + "connection_drops": 0, + "scenario_post_errors": 0, + "scenario_changes": 9, + "scenario_events_received": 1080, + "scenario_events_missing": 0, + "scenario_latency_samples": 1080, + "scenario_p50_ms": 1.3, + "scenario_p95_ms": 5.6, + "scenario_p99_ms": 5.8, + "scenario_max_ms": 5.812, + "changes": [ + { + "version": 3, + "post_at": "2026-11-14T06:35:03.474811874+01:00", + "connected_at_post": 120, + "received": 120, + "missing": 0, + "p50_ms": 5.5, + "p95_ms": 5.8, + "p99_ms": 5.8, + "max_ms": 5.812 + }, + { + "version": 6, + "post_at": "2026-11-14T06:35:33.474996501+01:00", + "connected_at_post": 120, + "received": 120, + "missing": 0, + "p50_ms": 2.4, + "p95_ms": 2.7, + "p99_ms": 2.8, + "max_ms": 2.791 + }, + { + "version": 7, + "post_at": "2026-11-14T06:36:03.474746595+01:00", + "connected_at_post": 120, + "received": 120, + "missing": 0, + "p50_ms": 1.1, + "p95_ms": 1.3, + "p99_ms": 1.4, + "max_ms": 1.353 + }, + { + "version": 8, + "post_at": "2026-11-14T06:36:33.467364475+01:00", + "connected_at_post": 120, + "received": 120, + "missing": 0, + "p50_ms": 1.7, + "p95_ms": 2.4, + "p99_ms": 2.4, + "max_ms": 2.396 + }, + { + "version": 9, + "post_at": "2026-11-14T06:37:03.472992832+01:00", + "connected_at_post": 120, + "received": 120, + "missing": 0, + "p50_ms": 1, + "p95_ms": 1.3, + "p99_ms": 1.3, + "max_ms": 1.342 + }, + { + "version": 10, + "post_at": "2026-11-14T06:37:33.475595017+01:00", + "connected_at_post": 120, + "received": 120, + "missing": 0, + "p50_ms": 1.1, + "p95_ms": 1.3, + "p99_ms": 1.4, + "max_ms": 1.434 + }, + { + "version": 11, + "post_at": "2026-11-14T06:38:03.474743535+01:00", + "connected_at_post": 120, + "received": 120, + "missing": 0, + "p50_ms": 1.3, + "p95_ms": 1.4, + "p99_ms": 1.4, + "max_ms": 1.375 + }, + { + "version": 12, + "post_at": "2026-11-14T06:38:33.469461499+01:00", + "connected_at_post": 120, + "received": 120, + "missing": 0, + "p50_ms": 1.5, + "p95_ms": 1.7, + "p99_ms": 1.7, + "max_ms": 1.718 + }, + { + "version": 13, + "post_at": "2026-11-14T06:39:03.475695318+01:00", + "connected_at_post": 120, + "received": 120, + "missing": 0, + "p50_ms": 1, + "p95_ms": 1.2, + "p99_ms": 1.2, + "max_ms": 1.249 + } + ], + "heartbeat": { + "unique_server_timestamps": 20, + "min_interval_ms": 15000, + "median_interval_ms": 15000, + "p95_interval_ms": 15000, + "max_interval_ms": 15000, + "per_client_min": 20, + "per_client_median": 20, + "per_client_max": 20 + }, + "errors": null, + "passed": true +} diff --git a/docs/results/load.txt b/docs/results/load.txt new file mode 100644 index 0000000..65678a3 --- /dev/null +++ b/docs/results/load.txt @@ -0,0 +1,21 @@ +clients connected: 120/120 after warmup +scenario v3: 120/120 clients p50=5.5ms p95=5.8ms max=5.8ms +scenario v6: 120/120 clients p50=2.4ms p95=2.7ms max=2.8ms +scenario v7: 120/120 clients p50=1.1ms p95=1.3ms max=1.4ms +scenario v8: 120/120 clients p50=1.7ms p95=2.4ms max=2.4ms +scenario v9: 120/120 clients p50=1.0ms p95=1.3ms max=1.3ms +scenario v10: 120/120 clients p50=1.1ms p95=1.3ms max=1.4ms +scenario v11: 120/120 clients p50=1.3ms p95=1.4ms max=1.4ms +scenario v12: 120/120 clients p50=1.5ms p95=1.7ms max=1.7ms +scenario v13: 120/120 clients p50=1.0ms p95=1.2ms max=1.2ms + +=== SSE load test === +duration: 300.0s +clients: 120 configured, 120 peak connected, 120 after warmup +connections: 0 drops/reconnects, 0 scenario POST errors +scenario changes: 9 (1080 events received, 0 missing) +scenario latency: p50=1.3ms p95=5.6ms p99=5.8ms max=5.8ms (n=1080) +heartbeat: 20 server pings, interval min=15000ms median=15000ms p95=15000ms max=15000ms +heartbeat/client: min=20 median=20 max=20 +result: PASS +json written to /home/noel/Documents/Hackathon_2026/api-go.git/feat-integration/docs/results/load.json diff --git a/docs/results/meta-initial.json b/docs/results/meta-initial.json new file mode 100644 index 0000000..5b50a83 --- /dev/null +++ b/docs/results/meta-initial.json @@ -0,0 +1 @@ +{"data_version":"2026-11-07T23:51:00Z","model_version":"histgb-2026-11-14","context_version":"2026-11-14T03:47:09Z","coverage":{"sensors":8,"from":"2026-06-12T00:00:00Z","to":"2026-11-07T23:51:00Z","cadence_min":15},"metrics":{"model":"weather-7d","mae":3.32,"wape":27.5,"r2":0.777,"baseline_mae":5.4,"baseline_wape":44.7},"limits":["estimated devices, not people","8 sensors in a ~580 m inner-city cluster","5 months of multi-sensor data; no annual seasonality claim","observed weather in training, forecast weather in scenarios","holidays are context, not a large accuracy claim"],"cache":{"source":"fastapi","last_success_at":"2026-11-14T05:34:09Z","last_error":null,"next_retry_at":null,"failures":0}} \ No newline at end of file diff --git a/docs/results/sse-stream.log b/docs/results/sse-stream.log new file mode 100644 index 0000000..43a628c --- /dev/null +++ b/docs/results/sse-stream.log @@ -0,0 +1,13 @@ +retry: 3000 + +event: scenario +id: 1 +data: {"version":1,"scenario":{"scenario":"base","weather":{"temp_delta_c":0,"precip_mm":0},"event":{"type":"","uplift":0}},"at":"2026-11-14T05:34:09Z"} + +event: scenario +id: 2 +data: {"version":2,"scenario":{"scenario":"sunny-saturday-market","weather":{"temp_delta_c":4,"precip_mm":0},"event":{"type":"market","uplift":0.2}},"at":"2026-11-14T05:34:20Z"} + +event: ping +data: {"t":"2026-11-14T05:34:24Z"} + diff --git a/docs/results/sse.json b/docs/results/sse.json new file mode 100644 index 0000000..00dbd44 --- /dev/null +++ b/docs/results/sse.json @@ -0,0 +1,9 @@ +{ + "scenario_version": 2, + "propagation_ms": 15, + "heartbeat_after_ms": 4260, + "heartbeat_interval_s": 15, + "retry_ms": 3000, + "reconnect_ms": 18, + "reconnect_current_version": 2 +} diff --git a/docs/results/staffing.json b/docs/results/staffing.json new file mode 100644 index 0000000..adacb0b --- /dev/null +++ b/docs/results/staffing.json @@ -0,0 +1,10 @@ +{ + "cold_ms": 19, + "warm_ms": 0, + "hours": 70, + "shifts": 21, + "coverage_pct": 100, + "cost_eur": 3360, + "cached": false, + "note": "follow-up run on ports 18100/18180; same commit" +} diff --git a/docs/results/summary.txt b/docs/results/summary.txt new file mode 100644 index 0000000..d6a8b54 --- /dev/null +++ b/docs/results/summary.txt @@ -0,0 +1,7 @@ +PulseCast Go integration 2026-11-14T05:39:33Z +FastAPI http://127.0.0.1:18000, Go http://127.0.0.1:18080 +small CSV: 24636 rows (2026-09-01 .. 2026-10-16) +small job: status=succeeded version=logit-20261114-053410 accuracy=0.824 roc_auc=0.849 submit_to_terminal_ms=171 +SSE: {"scenario_version":2,"propagation_ms":15,"heartbeat_after_ms":4260,"heartbeat_interval_s":15,"retry_ms":3000,"reconnect_ms":18,"reconnect_current_version":2} +fallback: {"fastapi_down":{"cache_failures":1,"last_error":"ml: /bootstrap: Get \"http://127.0.0.1:18000/bootstrap\": dial tcp 127.0.0.1:18000: connect: connection refused","next_retry_at":"2026-11-14T05:34:31Z","last_success_at":"2026-11-14T05:34:20Z","forecast_cached":true,"staffing_cached":true,"readyz":200,"sensors":8},"recovery":{"after_ms":1622,"last_success_before":"2026-11-14T05:34:20Z","last_success_after":"2026-11-14T05:34:33Z","forecast_cached":false,"failures":0}} +forecast job: status=succeeded version=histgb-20261114-053413 mae=4.83 wape=33.0 submit_to_terminal_ms=3571 diff --git a/docs/results/train-forecast.json b/docs/results/train-forecast.json new file mode 100644 index 0000000..f4ca4a5 --- /dev/null +++ b/docs/results/train-forecast.json @@ -0,0 +1,77 @@ +{ + "model": "forecast", + "submit_http": 202, + "submit_ms": 15, + "submit_to_terminal_ms": 3571, + "job": { + "job_id": "2f4e7c5a5bc14abd81d4eb5cdbeb0e1a", + "status": "succeeded", + "model": "forecast", + "submitted_at": "2026-11-14T05:34:10Z", + "started_at": "2026-11-14T05:34:10Z", + "finished_at": "2026-11-14T05:34:13Z", + "metrics": { + "model": "weather-7d", + "mae": 4.83, + "wape": 33.0, + "r2": 0.703, + "baseline": "seasonal_naive", + "baseline_mae": 6.8, + "baseline_wape": 46.5, + "baseline_r2": 0.318, + "n_eval": 4919, + "ablation": [ + { + "name": "seasonal_naive", + "description": "exact 168-row lag (same observed hour last week)", + "mae": 6.8, + "wape": 46.5, + "r2": 0.318, + "n": 4919 + }, + { + "name": "day_no_weather", + "description": "calendar + lags 24/48/168 + 24h/168h rolling means", + "mae": 5.5, + "wape": 37.6, + "r2": 0.623, + "n": 4919 + }, + { + "name": "day_holiday", + "description": "day-ahead + Baden-Württemberg holiday flag", + "mae": 5.5, + "wape": 37.6, + "r2": 0.623, + "n": 4919 + }, + { + "name": "day_weather", + "description": "day-ahead + temperature + rain (observed weather)", + "mae": 5.03, + "wape": 34.4, + "r2": 0.682, + "n": 4919 + }, + { + "name": "week_no_weather", + "description": "7-day-safe calendar + previous same weekday/hour values", + "mae": 5.4, + "wape": 36.9, + "r2": 0.631, + "n": 4919 + }, + { + "name": "week_weather", + "description": "7-day-safe + temperature + rain (production model)", + "mae": 4.83, + "wape": 33.0, + "r2": 0.703, + "n": 4919 + } + ] + }, + "model_version": "histgb-20261114-053413", + "error": null + } +} diff --git a/docs/results/train-small.json b/docs/results/train-small.json new file mode 100644 index 0000000..a1cd615 --- /dev/null +++ b/docs/results/train-small.json @@ -0,0 +1,27 @@ +{ + "model": "small", + "submit_http": 202, + "submit_ms": 25, + "submit_to_terminal_ms": 171, + "job": { + "job_id": "9b6112d7baa34ee8ad236db0d77b836b", + "status": "succeeded", + "model": "small", + "submitted_at": "2026-11-14T05:34:10Z", + "started_at": "2026-11-14T05:34:10Z", + "finished_at": "2026-11-14T05:34:10Z", + "metrics": { + "accuracy": 0.824, + "precision": 0.589, + "recall": 0.42, + "f1": 0.49, + "roc_auc": 0.849, + "n_train": 5363, + "n_test": 1728, + "positive_rate": 0.201, + "sensors": 8 + }, + "model_version": "logit-20261114-053410", + "error": null + } +} diff --git a/scripts/integration.sh b/scripts/integration.sh index fccf137..0dbb236 100755 --- a/scripts/integration.sh +++ b/scripts/integration.sh @@ -229,6 +229,22 @@ if [[ "$RUN_FORECAST" == "1" ]]; then ok "Go /api/meta model_version refreshed to $JOB_VERSION" fi +# ------------------------------------------------------------------ staffing +log "timing staffing plan generation (contract: <2s)" +staffing_url="$API_URL/api/staffing?sensor=hn-stadtgalerie-01&scenario=base&standard=standard" +staffing_cold=$(curl -sS -m 10 -o "$WORKDIR/staffing-live.json" -w '%{time_total}' "$staffing_url") +staffing_warm=$(curl -sS -m 10 -o "$WORKDIR/staffing-warm.json" -w '%{time_total}' "$staffing_url") +[[ "$(json_get "$WORKDIR/staffing-live.json" '.hours | length')" == "70" ]] || fail "staffing plan does not cover 7 days x 10 store hours" +[[ "$(json_get "$WORKDIR/staffing-live.json" '.cached')" == "false" ]] || fail "live staffing plan is marked cached" +staffing_cold_ms=$(awk -v s="$staffing_cold" 'BEGIN { printf "%d", s*1000 }') +staffing_warm_ms=$(awk -v s="$staffing_warm" 'BEGIN { printf "%d", s*1000 }') +(( staffing_cold_ms < 2000 )) || fail "staffing cold plan took ${staffing_cold_ms}ms (>=2s)" +ok "staffing: cold ${staffing_cold_ms}ms, cached ${staffing_warm_ms}ms, 70 hours, coverage=$(json_get "$WORKDIR/staffing-live.json" '.summary.coverage_pct')%" +jq -n --argjson cold_ms "$staffing_cold_ms" --argjson warm_ms "$staffing_warm_ms" \ + --slurpfile plan "$WORKDIR/staffing-live.json" \ + '{cold_ms:$cold_ms, warm_ms:$warm_ms, hours:($plan[0].hours|length), shifts:($plan[0].shifts|length), coverage_pct:$plan[0].summary.coverage_pct, cost_eur:$plan[0].summary.cost_eur, cached:$plan[0].cached}' \ + > "$RESULTS_DIR/staffing.json" + # -------------------------------------------------------------- SSE scenario log "checking SSE scenario propagation, heartbeat and reconnect" sse_log="$RESULTS_DIR/sse-stream.log" @@ -405,6 +421,7 @@ PulseCast Go integration $(date -u +%Y-%m-%dT%H:%M:%SZ) FastAPI $ML_URL, Go $API_URL small CSV: $small_rows rows ($SMALL_FROM .. $SMALL_TO) small job: $(jq -r '"status=\(.job.status) version=\(.job.model_version) accuracy=\(.job.metrics.accuracy) roc_auc=\(.job.metrics.roc_auc) submit_to_terminal_ms=\(.submit_to_terminal_ms)"' "$RESULTS_DIR/train-small.json") +staffing: $(jq -r '"cold_ms=\(.cold_ms) warm_ms=\(.warm_ms) hours=\(.hours) coverage_pct=\(.coverage_pct) cost_eur=\(.cost_eur)"' "$RESULTS_DIR/staffing.json") SSE: $(jq -c '.' "$RESULTS_DIR/sse.json") fallback: $(jq -c '.' "$RESULTS_DIR/fallback.json") EOF