Merge pull request 'feat: ci docs' (#3) from feat/ci-docs into main
ci / check (push) Successful in 56s

Reviewed-on: #3
This commit was merged in pull request #3.
This commit is contained in:
Noel committed 2026-11-14 08:34:33 +00:00
commit a0ad60ec53
38 files changed
+3210 -400

No files matched your search

+4
View File
@@ -2,3 +2,7 @@
.gitignore
*.md
opencode.json
docs
scripts
# Fixtures are for tests only; the runtime image fetches FastAPI /bootstrap.
testdata
+4
View File
@@ -10,6 +10,7 @@ permissions:
jobs:
check:
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Check out
uses: actions/checkout@v4
@@ -18,6 +19,9 @@ jobs:
uses: actions/setup-go@v5
with:
go-version-file: go.mod
# The module is stdlib-only, so there is no go.sum for the action's
# default cache key. Remove this once the module has dependencies.
cache: false
- name: Check
run: make check
+3 -4
View File
@@ -17,11 +17,10 @@ FROM gcr.io/distroless/static-debian12:nonroot
WORKDIR /app
COPY --from=build /out/api /app/api
# Baked fallback data lives in the image; there is no runtime data fetch.
COPY --from=build /src/testdata/data /app/data
ENV DATA_DIR=/app/data \
PORT=8080
# No data is baked into the image. The API fetches FastAPI /bootstrap at
# startup, caches the snapshot in memory, and refreshes it in the background.
ENV PORT=8080
EXPOSE 8080
# Numeric UID/GID: the kubelet cannot verify named users for runAsNonRoot.
+14 -1
View File
@@ -1,6 +1,6 @@
GO ?= go
.PHONY: check vet test build
.PHONY: check vet test build integration integration-load load
# Local equivalent of the Gitea Actions CI job.
check: vet test build
@@ -13,3 +13,16 @@ test:
build:
$(GO) build ./...
# End-to-end: real FastAPI on :18000, Go on :18080, train small+forecast,
# SSE scenario/heartbeat/reconnect, FastAPI kill + cache fallback + recovery.
integration:
scripts/integration.sh
# The same run plus the 120-client, 5-minute SSE load test.
integration-load:
scripts/integration.sh --load
# Load-only against an already-running API (CONTROL_USER/CONTROL_PASS from env).
load:
$(GO) run ./scripts/sse-load -base http://127.0.0.1:18080 -clients 120 -duration 5m
+92 -3
View File
@@ -1,6 +1,39 @@
# api-go
# PulseCast API
Golang API that is the backbone for daily usage, while python works in batches and is heavy in compute while Golang handles concurrent usage cheap
Go service for PulseCast. It serves sensor series, live forecasts, staffing
plans and scenario state over SSE, and proxies training to FastAPI. It keeps the
FastAPI `/bootstrap` snapshot in memory, so reads survive FastAPI outages.
## Quick start
```sh
go run ./cmd/api
curl -s localhost:8080/api/healthz # {"ok":true}
curl -s localhost:8080/api/readyz # 503 until the first /bootstrap snapshot
```
The API listens on `:8080` and expects FastAPI at `http://localhost:8000`
(`FASTAPI_URL`). It starts without FastAPI; readiness stays `503` until the
first snapshot arrives. Control endpoints return `503` until `CONTROL_USER` and
`CONTROL_PASS` are set.
## Endpoints
| Endpoint | Purpose |
|---|---|
| `GET /api/healthz` | Liveness. |
| `GET /api/readyz` | Data and model readiness; `503` until the first snapshot. |
| `GET /api/meta` | Versions, coverage, metrics and cache status. |
| `GET /api/sensors` | Sensors sorted by id. |
| `GET /api/series` | Observed series; query `sensor`, `from`, `to`, `step=15m\|1h`. |
| `GET /api/context` | Weather and holidays. |
| `GET /api/forecast` | Live forecast or last-good fallback; query `sensor`, `horizon`, `scenario`. |
| `GET /api/staffing` | Shift plan; query `sensor`, `scenario`, `standard`. |
| `GET /api/scenario` | Current scenario. |
| `POST /api/scenario` | Set scenario (control auth) and broadcast it over SSE. |
| `GET /api/stream` | SSE scenario events and the 15 s heartbeat. |
| `POST /api/train` | Proxy training to FastAPI (control auth). |
| `GET /api/train/{job_id}` | Training job status (control auth). |
## Checks
@@ -10,7 +43,63 @@ make check
Runs `go vet ./...`, `go test -count=1 ./...`, and `go build ./...`. The same
target runs in Gitea Actions on every push and pull request
(`.gitea/workflows/ci.yml`).
(`.gitea/workflows/ci.yml`). CI does not run the integration or load tests;
those need the ML checkout and raw CSV.
## Runtime data
The API image contains **no baked data**. At startup it fetches
`GET {FASTAPI_URL}/bootstrap` and keeps `sensors`, `series`, `context` and
`meta` in memory. A background loop refreshes the snapshot; on failure it keeps
the last good snapshot and retries with exponential backoff
**1, 2, 4, 8, 16, 30 s (cap)**, resetting on success.
- `GET /api/readyz` returns `503` until the first valid snapshot is loaded.
After that, FastAPI being down does not fail readiness.
- `GET /api/meta` reports the cache state under `cache`:
`source`, `last_success_at`, `last_error`, `next_retry_at`, `failures`.
- `GET /api/forecast` calls FastAPI `/predict`, remembers the last good points
per `sensor|horizon|scenario`, and serves them with `"cached": true` when
FastAPI is unavailable.
- `testdata/` is used by tests only and is excluded from the image by
`.dockerignore`.
| Env | Default | Purpose |
|---|---|---|
| `PORT` | `8080` | HTTP listen port |
| `FASTAPI_URL` | `http://localhost:8000` | FastAPI base URL |
| `BOOTSTRAP_REFRESH` | `60s` | Healthy snapshot refresh interval |
| `BOOTSTRAP_TIMEOUT` | `20s` | Per-request `/bootstrap` timeout |
| `ML_TIMEOUT` | `1500ms` | Per-request `/predict` timeout |
| `SSE_HEARTBEAT` | `15s` | SSE ping interval |
| `CONTROL_USER` / `CONTROL_PASS` | unset | Basic auth for `POST /api/scenario` and `/api/train*` |
| `CONTROL_TOKEN` | unset | Optional Bearer token forwarded to FastAPI `/train*` |
## Integration and load test
`make integration` runs the real stack (it refuses to start if `:18000` or
`:18080` is busy):
1. starts FastAPI from the ML checkout on `:18000` and the Go API on `:18080`;
2. uploads a date-range CSV subset to `POST /api/train` and polls the job
(`small`, then `forecast` by default);
3. verifies the live forecast still answers, scenario propagation, the 15 s
heartbeat and reconnect behaviour;
4. stops FastAPI, verifies `/api/meta.cache`, snapshot-served sensors, the
last-good forecast (`cached: true`) and staffing fallback;
5. restarts FastAPI and waits for the cache to recover.
`make integration-load` additionally runs `scripts/sse-load`: 120 concurrent
SSE clients for 5 minutes with a scenario change every 30 s. It records peak
connections, drops, per-change and overall p95 latency, and heartbeat cadence.
The tool exits non-zero when a client drops, an event is missing, or p95
scenario latency exceeds the contract's 2 s threshold.
Results land in `/tmp/pulsecast-integration/results` by default. Override with
`WORKDIR=/path` or `RESULTS_DIR=/path`. `make load` runs only the load test
against an API that is already up. The verified run is written up in
[`docs/integration.md`](docs/integration.md) with raw evidence under
`docs/results/`.
## Container
+14 -11
View File
@@ -1,4 +1,4 @@
// Command api serves the PulseCast Go API, SSE stream and cached fallback.
// Command api serves the PulseCast Go API, SSE stream and runtime snapshot cache.
package main
import (
@@ -12,38 +12,41 @@ import (
"time"
"pulsecast/api/internal/api"
"pulsecast/api/internal/cache"
"pulsecast/api/internal/config"
"pulsecast/api/internal/ml"
"pulsecast/api/internal/sse"
"pulsecast/api/internal/store"
)
func main() {
cfg := config.Load()
// A partial dataset is not fatal: the process starts and /api/readyz reports
// 503 until the baked artifacts are complete, instead of crash-looping.
st, err := store.Load(cfg.DataDir)
if err != nil {
slog.Error("baked data is incomplete", "dir", cfg.DataDir, "error", err)
}
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
client := ml.NewWithToken(cfg.FastAPIURL, cfg.MLTimeout, cfg.ControlToken)
snapshots := cache.New(client, cache.Options{
RefreshInterval: cfg.RefreshInterval,
Timeout: cfg.BootstrapTimeout,
})
// Loading happens in the background so the process can answer /api/readyz
// with 503 immediately instead of crash-looping while FastAPI starts.
go snapshots.Run(ctx)
hub := sse.NewHub(cfg.Heartbeat)
go hub.Run(ctx)
server := &http.Server{
Addr: cfg.Addr,
Handler: api.New(cfg, st, ml.NewWithToken(cfg.FastAPIURL, cfg.MLTimeout, cfg.ControlToken), hub).Handler(),
Handler: api.New(cfg, snapshots, client, hub).Handler(),
ReadHeaderTimeout: 5 * time.Second,
IdleTimeout: 120 * time.Second,
}
serveErr := make(chan error, 1)
go func() {
slog.Info("api listening", "addr", cfg.Addr, "data_dir", cfg.DataDir, "fastapi", cfg.FastAPIURL)
slog.Info("api listening", "addr", cfg.Addr, "fastapi", cfg.FastAPIURL,
"refresh", cfg.RefreshInterval)
serveErr <- server.ListenAndServe()
}()
+143
View File
@@ -0,0 +1,143 @@
# Integration and load-test report
Run: **2026-11-14**, Go API worktree `feat/integration`.
FastAPI: real ML service from `ML-py.git/main` on `127.0.0.1:18000`.
Go API: this worktree on `127.0.0.1:18080`, `FASTAPI_URL=http://127.0.0.1:18000`.
Everything below is reproducible with `make integration-load`; raw evidence is
in `docs/results/`.
## Command
```sh
WORKDIR=/tmp/pulsecast-full-run RESULTS_DIR="$PWD/docs/results" \
scripts/integration.sh --load
```
The script owns `:18000` and `:18080` (it refuses to run if either is busy).
It uses a 24,636-row CSV subset (`2026-09-01 .. 2026-10-16`, all 8 sensors,
`densityData` rows only) generated from the raw challenge CSV into `/tmp`.
The subset is test input only; it is not committed and not baked into the image.
## Train end-to-end (real FastAPI, through the Go proxy)
Both jobs were uploaded as multipart CSV to `POST /api/train` with control auth,
then polled through `GET /api/train/{job_id}`.
| Job | Result | Model version | Submit to terminal | Headline metrics |
|---|---|---|---|---|
| `small` | `succeeded`, `error: null` | `logit-20261114-053410` | **171 ms** (POST itself 25 ms) | accuracy 0.824, precision 0.589, recall 0.420, F1 0.490, ROC-AUC 0.849, n_train 5,363, n_test 1,728, 8 sensors |
| `forecast` | `succeeded`, `error: null` | `histgb-20261114-053413` | **3,571 ms** | MAE 4.83, WAPE 33.0, R² 0.703, n_eval 4,919 |
- `small` is a logistic-regression busy classifier; `forecast` retrains the
global HistGB. Both ran in FastAPI; the Go process only proxied and polled.
- The forecast job trained on the **small subset**, so its MAE 4.83 is not the
baked model's quality (3.32 on the full data). It proves the upload, queue,
poll and hot-swap path, not model quality.
- `/api/forecast?sensor=hn-stadtgalerie-01&horizon=7d&scenario=base` returned
**200, 168 points, `cached: false`** immediately after the small job.
- After the forecast hot-swap, the Go snapshot refresh picked up the new
`model_version` (`histgb-20261114-053413`) in `/api/meta`.
## SSE scenario, heartbeat and reconnect
| Check | Result |
|---|---|
| Connect payload | `retry: 3000`, current `scenario` event immediately |
| Scenario propagation | **15 ms** from `POST /api/scenario` to the client event (limit: 2 s) |
| Heartbeat | received during the check; server interval exactly **15,000 ms** throughout the load test |
| Reconnect | closed and reopened the stream; current scenario (v2) arrived in **18 ms** |
| Event shape | `event: scenario`, `id: <version>`, JSON body with `version`, `scenario`, `at` |
## Staffing plan generation (contract: < 2 s)
A follow-up run on the same commit (ports `18100`/`18180`, because another
verification process held `18000`/`18080`) timed `GET /api/staffing`:
| Check | Result |
|---|---|
| Cold call (live `/predict` + plan build) | **19 ms** |
| Warm call (same plan key) | **< 1 ms** |
| Plan size | 70 store hours (7 days × 10:00–20:00), 21 shift blocks |
| Coverage | 100%, cost €3,360 |
| `cached` on the live plan | `false` |
The repeat run also reproduced the other checks: small train 169 ms, scenario
propagation 13 ms, reconnect 16 ms, recovery 1,339 ms.
## 120-client / 5-minute load test
`scripts/sse-load`: 120 concurrent streams, 300.008 s, scenario change every
30 s (rotating `sunny-saturday-market`, `rainy-saturday`, `weekday-event`,
`base`).
| Metric | Result |
|---|---|
| Peak connections | **120 / 120** |
| Connected after warmup | 120 / 120 |
| Dropped or reconnected streams | **0** |
| Scenario changes | 9 |
| Scenario events received | **1,080 / 1,080** (120 per change) |
| Missing scenario events | **0** |
| Scenario latency samples | 1,080 |
| p50 | **1.3 ms** |
| p95 | **5.6 ms** |
| p99 | **5.8 ms** |
| max | **5.812 ms** |
| Heartbeat pings | 20 unique server timestamps, every **15,000 ms** (min = median = p95 = max) |
| Heartbeats per client | 20 / 20 / 20 (min / median / max) |
Per-change p95 ranged from 1.2 ms to 5.8 ms; the highest latencies were the
first change after warmup. The tool exits non-zero if a client drops, an event
is missing, or p95 exceeds the contract's 2 s limit; this run passed.
## Fallback and recovery (runtime cache)
Sequence: warm a live base forecast, stop FastAPI, observe the cache failure,
serve the snapshot/last-good data, restart FastAPI, wait for recovery.
While FastAPI was down:
| Check | Result |
|---|---|
| `/api/meta.cache.failures` | **1** |
| `/api/meta.cache.last_error` | `ml: /bootstrap: Get "http://127.0.0.1:18000/bootstrap": dial tcp 127.0.0.1:18000: connect: connection refused` |
| `/api/meta.cache.next_retry_at` | `2026-11-14T05:34:31Z` (1 s after the first failure; schedule is 1/2/4/8/16/30 s) |
| `/api/meta.cache.last_success_at` | `2026-11-14T05:34:20Z` (unchanged) |
| `/api/readyz` | **200** — a loaded snapshot keeps readiness even with FastAPI down |
| `/api/sensors` | **200**, 8 sensors, from the last good snapshot |
| `/api/context` | **200**, `cached: true` |
| `/api/series` | **200**, from the last good snapshot |
| `/api/forecast` (warm key) | **200**, 168 points, **`cached: true`** |
| `/api/staffing` | **200**, built from the last good forecast, **`cached: true`** |
| `/api/forecast` (never-warmed key) | **503** `forecast_unavailable` — no invented data |
After restarting FastAPI:
| Check | Result |
|---|---|
| Cache recovery | **1,622 ms** after restart; `failures: 0` |
| `last_success_at` | advanced from `05:34:20Z` to `05:34:33Z` |
| `/api/forecast` | **200**, `cached: false` again |
The integration script used `BOOTSTRAP_REFRESH=10s` to keep the observation
window short. The production default is 60 s; the failure path, the
1/2/4/8/16/30 s retry schedule and the recovery semantics are unchanged.
## Evidence files
- `docs/results/train-small.json`, `train-forecast.json` — raw job responses and timings
- `docs/results/staffing.json` — cold/warm plan timing from the follow-up run
- `docs/results/sse.json`, `sse-stream.log` — propagation, heartbeat, reconnect
- `docs/results/load.json`, `load.txt` — all load-test samples and per-change results
- `docs/results/fallback.json` — exact cache/fallback/recovery values
- `docs/results/summary.txt` — one-screen summary of the same run
- `docs/results/meta-initial.json`, `forecast-after-small.json` — baseline evidence
## Notes and blockers
- At one point `:18000` was held by another agent's ML end-to-end run. The test
was not started against that process; it was rerun once the port was free.
No blocker remains.
- FastAPI was trained on a subset, so the retrained forecast metrics are lower
than the baked full-data model. The baked model is restored on FastAPI
restart, and recovery was verified with that baked model.
+19
View File
@@ -0,0 +1,19 @@
{
"fastapi_down": {
"cache_failures": 1,
"last_error": "ml: /bootstrap: Get \"http://127.0.0.1:18000/bootstrap\": dial tcp 127.0.0.1:18000: connect: connection refused",
"next_retry_at": "2026-11-14T05:34:31Z",
"last_success_at": "2026-11-14T05:34:20Z",
"forecast_cached": true,
"staffing_cached": true,
"readyz": 200,
"sensors": 8
},
"recovery": {
"after_ms": 1622,
"last_success_before": "2026-11-14T05:34:20Z",
"last_success_after": "2026-11-14T05:34:33Z",
"forecast_cached": false,
"failures": 0
}
}
+1
View File
@@ -0,0 +1 @@
{"sensor":"hn-stadtgalerie-01","horizon":"7d","scenario":"base","points":[{"t":"2026-11-14T06:00:00Z","v":4.35,"lo":0,"hi":9.23},{"t":"2026-11-14T07:00:00Z","v":8.07,"lo":3.25,"hi":12.95},{"t":"2026-11-14T08:00:00Z","v":23.45,"lo":18.63,"hi":28.33},{"t":"2026-11-14T09:00:00Z","v":35.8,"lo":30.98,"hi":40.68},{"t":"2026-11-14T10:00:00Z","v":67.45,"lo":62.62,"hi":72.32},{"t":"2026-11-14T11:00:00Z","v":86.07,"lo":81.25,"hi":90.95},{"t":"2026-11-14T12:00:00Z","v":88.9,"lo":84.08,"hi":93.78},{"t":"2026-11-14T13:00:00Z","v":87.06,"lo":82.23,"hi":91.94},{"t":"2026-11-14T14:00:00Z","v":88.1,"lo":83.27,"hi":92.97},{"t":"2026-11-14T15:00:00Z","v":84.2,"lo":79.37,"hi":89.07},{"t":"2026-11-14T16:00:00Z","v":85.68,"lo":80.86,"hi":90.56},{"t":"2026-11-14T17:00:00Z","v":76.15,"lo":71.33,"hi":81.03},{"t":"2026-11-14T18:00:00Z","v":45.34,"lo":40.51,"hi":50.21},{"t":"2026-11-14T19:00:00Z","v":20.09,"lo":15.26,"hi":24.97},{"t":"2026-11-14T20:00:00Z","v":13,"lo":8.17,"hi":17.87},{"t":"2026-11-14T21:00:00Z","v":7.36,"lo":2.54,"hi":12.24},{"t":"2026-11-14T22:00:00Z","v":5.2,"lo":0.37,"hi":10.07},{"t":"2026-11-14T23:00:00Z","v":4.64,"lo":0,"hi":9.52},{"t":"2026-11-15T00:00:00Z","v":3.25,"lo":0,"hi":8.13},{"t":"2026-11-15T01:00:00Z","v":2.35,"lo":0,"hi":7.22},{"t":"2026-11-15T02:00:00Z","v":2.46,"lo":0,"hi":7.34},{"t":"2026-11-15T03:00:00Z","v":1.78,"lo":0,"hi":6.66},{"t":"2026-11-15T04:00:00Z","v":1.78,"lo":0,"hi":6.65},{"t":"2026-11-15T05:00:00Z","v":1.77,"lo":0,"hi":6.65},{"t":"2026-11-15T06:00:00Z","v":2.16,"lo":0,"hi":7.04},{"t":"2026-11-15T07:00:00Z","v":2.71,"lo":0,"hi":7.59},{"t":"2026-11-15T08:00:00Z","v":4.92,"lo":0.09,"hi":9.79},{"t":"2026-11-15T09:00:00Z","v":6.51,"lo":1.68,"hi":11.39},{"t":"2026-11-15T10:00:00Z","v":6.41,"lo":1.58,"hi":11.28},{"t":"2026-11-15T11:00:00Z","v":12.24,"lo":7.41,"hi":17.11},{"t":"2026-11-15T12:00:00Z","v":17.2,"lo":12.38,"hi":22.08},{"t":"2026-11-15T13:00:00Z","v":17.44,"lo":12.61,"hi":22.31},{"t":"2026-11-15T14:00:00Z","v":17.05,"lo":12.23,"hi":21.93},{"t":"2026-11-15T15:00:00Z","v":15.94,"lo":11.12,"hi":20.82},{"t":"2026-11-15T16:00:00Z","v":16.44,"lo":11.61,"hi":21.32},{"t":"2026-11-15T17:00:00Z","v":14.1,"lo":9.27,"hi":18.97},{"t":"2026-11-15T18:00:00Z","v":8.92,"lo":4.1,"hi":13.8},{"t":"2026-11-15T19:00:00Z","v":4.78,"lo":0,"hi":9.65},{"t":"2026-11-15T20:00:00Z","v":2.49,"lo":0,"hi":7.36},{"t":"2026-11-15T21:00:00Z","v":1.6,"lo":0,"hi":6.47},{"t":"2026-11-15T22:00:00Z","v":1.84,"lo":0,"hi":6.72},{"t":"2026-11-15T23:00:00Z","v":1.69,"lo":0,"hi":6.57},{"t":"2026-11-16T00:00:00Z","v":1.59,"lo":0,"hi":6.47},{"t":"2026-11-16T01:00:00Z","v":1.57,"lo":0,"hi":6.45},{"t":"2026-11-16T02:00:00Z","v":1.57,"lo":0,"hi":6.45},{"t":"2026-11-16T03:00:00Z","v":1.57,"lo":0,"hi":6.45},{"t":"2026-11-16T04:00:00Z","v":1.61,"lo":0,"hi":6.48},{"t":"2026-11-16T05:00:00Z","v":2.59,"lo":0,"hi":7.46},{"t":"2026-11-16T06:00:00Z","v":7.05,"lo":2.22,"hi":11.93},{"t":"2026-11-16T07:00:00Z","v":14.22,"lo":9.39,"hi":19.09},{"t":"2026-11-16T08:00:00Z","v":18.83,"lo":14.01,"hi":23.71},{"t":"2026-11-16T09:00:00Z","v":28.14,"lo":23.31,"hi":33.01},{"t":"2026-11-16T10:00:00Z","v":39.14,"lo":34.31,"hi":44.01},{"t":"2026-11-16T11:00:00Z","v":51.34,"lo":46.52,"hi":56.22},{"t":"2026-11-16T12:00:00Z","v":50.18,"lo":45.35,"hi":55.05},{"t":"2026-11-16T13:00:00Z","v":50.22,"lo":45.4,"hi":55.1},{"t":"2026-11-16T14:00:00Z","v":50.3,"lo":45.47,"hi":55.17},{"t":"2026-11-16T15:00:00Z","v":49.68,"lo":44.85,"hi":54.56},{"t":"2026-11-16T16:00:00Z","v":51.29,"lo":46.46,"hi":56.16},{"t":"2026-11-16T17:00:00Z","v":43.09,"lo":38.26,"hi":47.96},{"t":"2026-11-16T18:00:00Z","v":21.34,"lo":16.51,"hi":26.22},{"t":"2026-11-16T19:00:00Z","v":11.95,"lo":7.12,"hi":16.82},{"t":"2026-11-16T20:00:00Z","v":5.95,"lo":1.13,"hi":10.83},{"t":"2026-11-16T21:00:00Z","v":2.98,"lo":0,"hi":7.85},{"t":"2026-11-16T22:00:00Z","v":2.19,"lo":0,"hi":7.07},{"t":"2026-11-16T23:00:00Z","v":2,"lo":0,"hi":6.87},{"t":"2026-11-17T00:00:00Z","v":1.92,"lo":0,"hi":6.79},{"t":"2026-11-17T01:00:00Z","v":1.68,"lo":0,"hi":6.55},{"t":"2026-11-17T02:00:00Z","v":1.59,"lo":0,"hi":6.46},{"t":"2026-11-17T03:00:00Z","v":1.59,"lo":0,"hi":6.46},{"t":"2026-11-17T04:00:00Z","v":2.4,"lo":0,"hi":7.28},{"t":"2026-11-17T05:00:00Z","v":3.38,"lo":0,"hi":8.25},{"t":"2026-11-17T06:00:00Z","v":8.12,"lo":3.29,"hi":12.99},{"t":"2026-11-17T07:00:00Z","v":16.28,"lo":11.45,"hi":21.16},{"t":"2026-11-17T08:00:00Z","v":17.6,"lo":12.77,"hi":22.47},{"t":"2026-11-17T09:00:00Z","v":27.47,"lo":22.65,"hi":32.35},{"t":"2026-11-17T10:00:00Z","v":35.28,"lo":30.45,"hi":40.15},{"t":"2026-11-17T11:00:00Z","v":50.47,"lo":45.64,"hi":55.34},{"t":"2026-11-17T12:00:00Z","v":48.11,"lo":43.28,"hi":52.98},{"t":"2026-11-17T13:00:00Z","v":42.24,"lo":37.41,"hi":47.11},{"t":"2026-11-17T14:00:00Z","v":49.05,"lo":44.22,"hi":53.92},{"t":"2026-11-17T15:00:00Z","v":48.19,"lo":43.37,"hi":53.07},{"t":"2026-11-17T16:00:00Z","v":47.31,"lo":42.48,"hi":52.18},{"t":"2026-11-17T17:00:00Z","v":32.93,"lo":28.1,"hi":37.8},{"t":"2026-11-17T18:00:00Z","vLine truncated
+131
View File
@@ -0,0 +1,131 @@
{
"started_at": "2026-11-14T05:34:33.422221537Z",
"duration_s": 300.008,
"base_url": "http://127.0.0.1:18080",
"clients_configured": 120,
"clients_peak_connected": 120,
"clients_connected_after_warmup": 120,
"connection_drops": 0,
"scenario_post_errors": 0,
"scenario_changes": 9,
"scenario_events_received": 1080,
"scenario_events_missing": 0,
"scenario_latency_samples": 1080,
"scenario_p50_ms": 1.3,
"scenario_p95_ms": 5.6,
"scenario_p99_ms": 5.8,
"scenario_max_ms": 5.812,
"changes": [
{
"version": 3,
"post_at": "2026-11-14T06:35:03.474811874+01:00",
"connected_at_post": 120,
"received": 120,
"missing": 0,
"p50_ms": 5.5,
"p95_ms": 5.8,
"p99_ms": 5.8,
"max_ms": 5.812
},
{
"version": 6,
"post_at": "2026-11-14T06:35:33.474996501+01:00",
"connected_at_post": 120,
"received": 120,
"missing": 0,
"p50_ms": 2.4,
"p95_ms": 2.7,
"p99_ms": 2.8,
"max_ms": 2.791
},
{
"version": 7,
"post_at": "2026-11-14T06:36:03.474746595+01:00",
"connected_at_post": 120,
"received": 120,
"missing": 0,
"p50_ms": 1.1,
"p95_ms": 1.3,
"p99_ms": 1.4,
"max_ms": 1.353
},
{
"version": 8,
"post_at": "2026-11-14T06:36:33.467364475+01:00",
"connected_at_post": 120,
"received": 120,
"missing": 0,
"p50_ms": 1.7,
"p95_ms": 2.4,
"p99_ms": 2.4,
"max_ms": 2.396
},
{
"version": 9,
"post_at": "2026-11-14T06:37:03.472992832+01:00",
"connected_at_post": 120,
"received": 120,
"missing": 0,
"p50_ms": 1,
"p95_ms": 1.3,
"p99_ms": 1.3,
"max_ms": 1.342
},
{
"version": 10,
"post_at": "2026-11-14T06:37:33.475595017+01:00",
"connected_at_post": 120,
"received": 120,
"missing": 0,
"p50_ms": 1.1,
"p95_ms": 1.3,
"p99_ms": 1.4,
"max_ms": 1.434
},
{
"version": 11,
"post_at": "2026-11-14T06:38:03.474743535+01:00",
"connected_at_post": 120,
"received": 120,
"missing": 0,
"p50_ms": 1.3,
"p95_ms": 1.4,
"p99_ms": 1.4,
"max_ms": 1.375
},
{
"version": 12,
"post_at": "2026-11-14T06:38:33.469461499+01:00",
"connected_at_post": 120,
"received": 120,
"missing": 0,
"p50_ms": 1.5,
"p95_ms": 1.7,
"p99_ms": 1.7,
"max_ms": 1.718
},
{
"version": 13,
"post_at": "2026-11-14T06:39:03.475695318+01:00",
"connected_at_post": 120,
"received": 120,
"missing": 0,
"p50_ms": 1,
"p95_ms": 1.2,
"p99_ms": 1.2,
"max_ms": 1.249
}
],
"heartbeat": {
"unique_server_timestamps": 20,
"min_interval_ms": 15000,
"median_interval_ms": 15000,
"p95_interval_ms": 15000,
"max_interval_ms": 15000,
"per_client_min": 20,
"per_client_median": 20,
"per_client_max": 20
},
"errors": null,
"passed": true
}
+21
View File
@@ -0,0 +1,21 @@
clients connected: 120/120 after warmup
scenario v3: 120/120 clients p50=5.5ms p95=5.8ms max=5.8ms
scenario v6: 120/120 clients p50=2.4ms p95=2.7ms max=2.8ms
scenario v7: 120/120 clients p50=1.1ms p95=1.3ms max=1.4ms
scenario v8: 120/120 clients p50=1.7ms p95=2.4ms max=2.4ms
scenario v9: 120/120 clients p50=1.0ms p95=1.3ms max=1.3ms
scenario v10: 120/120 clients p50=1.1ms p95=1.3ms max=1.4ms
scenario v11: 120/120 clients p50=1.3ms p95=1.4ms max=1.4ms
scenario v12: 120/120 clients p50=1.5ms p95=1.7ms max=1.7ms
scenario v13: 120/120 clients p50=1.0ms p95=1.2ms max=1.2ms
=== SSE load test ===
duration: 300.0s
clients: 120 configured, 120 peak connected, 120 after warmup
connections: 0 drops/reconnects, 0 scenario POST errors
scenario changes: 9 (1080 events received, 0 missing)
scenario latency: p50=1.3ms p95=5.6ms p99=5.8ms max=5.8ms (n=1080)
heartbeat: 20 server pings, interval min=15000ms median=15000ms p95=15000ms max=15000ms
heartbeat/client: min=20 median=20 max=20
result: PASS
json written to /home/noel/Documents/Hackathon_2026/api-go.git/feat-integration/docs/results/load.json
+1
View File
@@ -0,0 +1 @@
{"data_version":"2026-11-07T23:51:00Z","model_version":"histgb-2026-11-14","context_version":"2026-11-14T03:47:09Z","coverage":{"sensors":8,"from":"2026-06-12T00:00:00Z","to":"2026-11-07T23:51:00Z","cadence_min":15},"metrics":{"model":"weather-7d","mae":3.32,"wape":27.5,"r2":0.777,"baseline_mae":5.4,"baseline_wape":44.7},"limits":["estimated devices, not people","8 sensors in a ~580 m inner-city cluster","5 months of multi-sensor data; no annual seasonality claim","observed weather in training, forecast weather in scenarios","holidays are context, not a large accuracy claim"],"cache":{"source":"fastapi","last_success_at":"2026-11-14T05:34:09Z","last_error":null,"next_retry_at":null,"failures":0}}
+13
View File
@@ -0,0 +1,13 @@
retry: 3000
event: scenario
id: 1
data: {"version":1,"scenario":{"scenario":"base","weather":{"temp_delta_c":0,"precip_mm":0},"event":{"type":"","uplift":0}},"at":"2026-11-14T05:34:09Z"}
event: scenario
id: 2
data: {"version":2,"scenario":{"scenario":"sunny-saturday-market","weather":{"temp_delta_c":4,"precip_mm":0},"event":{"type":"market","uplift":0.2}},"at":"2026-11-14T05:34:20Z"}
event: ping
data: {"t":"2026-11-14T05:34:24Z"}
+9
View File
@@ -0,0 +1,9 @@
{
"scenario_version": 2,
"propagation_ms": 15,
"heartbeat_after_ms": 4260,
"heartbeat_interval_s": 15,
"retry_ms": 3000,
"reconnect_ms": 18,
"reconnect_current_version": 2
}
+10
View File
@@ -0,0 +1,10 @@
{
"cold_ms": 19,
"warm_ms": 0,
"hours": 70,
"shifts": 21,
"coverage_pct": 100,
"cost_eur": 3360,
"cached": false,
"note": "follow-up run on ports 18100/18180; same commit"
}
+7
View File
@@ -0,0 +1,7 @@
PulseCast Go integration 2026-11-14T05:39:33Z
FastAPI http://127.0.0.1:18000, Go http://127.0.0.1:18080
small CSV: 24636 rows (2026-09-01 .. 2026-10-16)
small job: status=succeeded version=logit-20261114-053410 accuracy=0.824 roc_auc=0.849 submit_to_terminal_ms=171
SSE: {"scenario_version":2,"propagation_ms":15,"heartbeat_after_ms":4260,"heartbeat_interval_s":15,"retry_ms":3000,"reconnect_ms":18,"reconnect_current_version":2}
fallback: {"fastapi_down":{"cache_failures":1,"last_error":"ml: /bootstrap: Get \"http://127.0.0.1:18000/bootstrap\": dial tcp 127.0.0.1:18000: connect: connection refused","next_retry_at":"2026-11-14T05:34:31Z","last_success_at":"2026-11-14T05:34:20Z","forecast_cached":true,"staffing_cached":true,"readyz":200,"sensors":8},"recovery":{"after_ms":1622,"last_success_before":"2026-11-14T05:34:20Z","last_success_after":"2026-11-14T05:34:33Z","forecast_cached":false,"failures":0}}
forecast job: status=succeeded version=histgb-20261114-053413 mae=4.83 wape=33.0 submit_to_terminal_ms=3571
+77
View File
@@ -0,0 +1,77 @@
{
"model": "forecast",
"submit_http": 202,
"submit_ms": 15,
"submit_to_terminal_ms": 3571,
"job": {
"job_id": "2f4e7c5a5bc14abd81d4eb5cdbeb0e1a",
"status": "succeeded",
"model": "forecast",
"submitted_at": "2026-11-14T05:34:10Z",
"started_at": "2026-11-14T05:34:10Z",
"finished_at": "2026-11-14T05:34:13Z",
"metrics": {
"model": "weather-7d",
"mae": 4.83,
"wape": 33.0,
"r2": 0.703,
"baseline": "seasonal_naive",
"baseline_mae": 6.8,
"baseline_wape": 46.5,
"baseline_r2": 0.318,
"n_eval": 4919,
"ablation": [
{
"name": "seasonal_naive",
"description": "exact 168-row lag (same observed hour last week)",
"mae": 6.8,
"wape": 46.5,
"r2": 0.318,
"n": 4919
},
{
"name": "day_no_weather",
"description": "calendar + lags 24/48/168 + 24h/168h rolling means",
"mae": 5.5,
"wape": 37.6,
"r2": 0.623,
"n": 4919
},
{
"name": "day_holiday",
"description": "day-ahead + Baden-Württemberg holiday flag",
"mae": 5.5,
"wape": 37.6,
"r2": 0.623,
"n": 4919
},
{
"name": "day_weather",
"description": "day-ahead + temperature + rain (observed weather)",
"mae": 5.03,
"wape": 34.4,
"r2": 0.682,
"n": 4919
},
{
"name": "week_no_weather",
"description": "7-day-safe calendar + previous same weekday/hour values",
"mae": 5.4,
"wape": 36.9,
"r2": 0.631,
"n": 4919
},
{
"name": "week_weather",
"description": "7-day-safe + temperature + rain (production model)",
"mae": 4.83,
"wape": 33.0,
"r2": 0.703,
"n": 4919
}
]
},
"model_version": "histgb-20261114-053413",
"error": null
}
}
+27
View File
@@ -0,0 +1,27 @@
{
"model": "small",
"submit_http": 202,
"submit_ms": 25,
"submit_to_terminal_ms": 171,
"job": {
"job_id": "9b6112d7baa34ee8ad236db0d77b836b",
"status": "succeeded",
"model": "small",
"submitted_at": "2026-11-14T05:34:10Z",
"started_at": "2026-11-14T05:34:10Z",
"finished_at": "2026-11-14T05:34:10Z",
"metrics": {
"accuracy": 0.824,
"precision": 0.589,
"recall": 0.42,
"f1": 0.49,
"roc_auc": 0.849,
"n_train": 5363,
"n_test": 1728,
"positive_rate": 0.201,
"sensors": 8
},
"model_version": "logit-20261114-053410",
"error": null
}
}
+357 -32
View File
@@ -10,18 +10,77 @@ import (
"io"
"net/http"
"net/http/httptest"
"os"
"strings"
"sync"
"testing"
"time"
"pulsecast/api/internal/cache"
"pulsecast/api/internal/config"
"pulsecast/api/internal/ml"
"pulsecast/api/internal/model"
"pulsecast/api/internal/sse"
"pulsecast/api/internal/store"
)
const fixtureDir = "../../testdata/data"
const fixtureBootstrap = "../../testdata/data/bootstrap.json"
type stubFetcher struct {
mu sync.RWMutex
payload model.Bootstrap
err error
refreshFn func(context.Context) (model.Bootstrap, error)
}
func (s *stubFetcher) Bootstrap(ctx context.Context) (model.Bootstrap, error) {
s.mu.RLock()
fn := s.refreshFn
payload, err := s.payload, s.err
s.mu.RUnlock()
if fn != nil {
return fn(ctx)
}
if err != nil {
return model.Bootstrap{}, err
}
return payload, nil
}
func (s *stubFetcher) fail(err error) {
s.mu.Lock()
s.err = err
s.mu.Unlock()
}
func (s *stubFetcher) recover(payload model.Bootstrap) {
s.mu.Lock()
s.payload, s.err = payload, nil
s.mu.Unlock()
}
func loadFixtureBootstrap(t *testing.T) model.Bootstrap {
t.Helper()
raw, err := os.ReadFile(fixtureBootstrap)
if err != nil {
t.Fatalf("read fixture: %v", err)
}
var payload model.Bootstrap
if err := json.Unmarshal(raw, &payload); err != nil {
t.Fatalf("decode fixture: %v", err)
}
return payload
}
func newFixtureCache(t *testing.T) (*cache.Cache, *stubFetcher) {
t.Helper()
payload := loadFixtureBootstrap(t)
fetcher := &stubFetcher{payload: payload}
snapshots := cache.New(fetcher, cache.Options{RefreshInterval: time.Hour, Timeout: time.Second})
if err := snapshots.Refresh(context.Background()); err != nil {
t.Fatalf("prime cache: %v", err)
}
return snapshots, fetcher
}
type stubML struct {
predict func(context.Context, ml.PredictRequest) ([]model.ForecastPoint, error)
@@ -58,20 +117,16 @@ func (s *stubML) TrainStatus(ctx context.Context, jobID string) (int, []byte, er
return s.trainStatus(ctx, jobID)
}
func loadFixtureStore(t *testing.T) *store.Store {
func newTestServer(t *testing.T, client MLClient, mutate func(*config.Config)) (*Server, *httptest.Server) {
t.Helper()
st, err := store.Load(fixtureDir)
if err != nil {
t.Fatalf("load fixtures: %v", err)
}
return st
snapshots, _ := newFixtureCache(t)
return newTestServerWithCache(t, snapshots, client, mutate)
}
func newTestServer(t *testing.T, client MLClient, mutate func(*config.Config)) (*Server, *httptest.Server) {
func newTestServerWithCache(t *testing.T, snapshots *cache.Cache, client MLClient, mutate func(*config.Config)) (*Server, *httptest.Server) {
t.Helper()
cfg := config.Config{
Addr: ":0",
DataDir: fixtureDir,
FastAPIURL: "http://127.0.0.1:1",
Heartbeat: time.Hour,
MLTimeout: 100 * time.Millisecond,
@@ -85,12 +140,47 @@ func newTestServer(t *testing.T, client MLClient, mutate func(*config.Config)) (
t.Cleanup(cancel)
go hub.Run(ctx)
srv := New(cfg, loadFixtureStore(t), client, hub)
srv := New(cfg, snapshots, client, hub)
ts := httptest.NewServer(srv.Handler())
t.Cleanup(ts.Close)
return srv, ts
}
func forecastPoints(n int) []model.ForecastPoint {
start := time.Date(2026, 11, 16, 9, 0, 0, 0, time.UTC)
points := make([]model.ForecastPoint, n)
for i := range points {
points[i] = model.ForecastPoint{
T: start.Add(time.Duration(i) * time.Hour),
V: float64(10 + i),
Lo: float64(5 + i),
Hi: float64(15 + i),
}
}
return points
}
// weekForecast builds seven complete store-hour days (10:00-20:00 Berlin),
// which is what the staffing planner consumes.
func weekForecast(t *testing.T) []model.ForecastPoint {
t.Helper()
loc, err := time.LoadLocation(model.StoreHours.TZ)
if err != nil {
t.Fatal(err)
}
start := time.Date(2026, 11, 16, model.StoreHours.Open, 0, 0, 0, loc)
var points []model.ForecastPoint
for day := 0; day < 7; day++ {
for hour := 0; hour < model.StoreHours.Close-model.StoreHours.Open; hour++ {
points = append(points, model.ForecastPoint{
T: start.AddDate(0, 0, day).Add(time.Duration(hour) * time.Hour).UTC(),
V: 25,
})
}
}
return points
}
func getJSON(t *testing.T, url string, dst any) *http.Response {
t.Helper()
resp, err := http.Get(url)
@@ -125,13 +215,11 @@ func TestHealthAndReady(t *testing.T) {
}
}
func TestReadyzWithoutData(t *testing.T) {
cfg := config.Config{Addr: ":0", DataDir: t.TempDir(), FastAPIURL: "http://127.0.0.1:1", Heartbeat: time.Hour, MLTimeout: 50 * time.Millisecond}
st, err := store.Load(cfg.DataDir)
if err != nil {
t.Logf("expected load error: %v", err)
}
srv := New(cfg, st, &stubML{}, sse.NewHub(time.Hour))
func TestReadyzWithoutSnapshot(t *testing.T) {
cfg := config.Config{Addr: ":0", FastAPIURL: "http://127.0.0.1:1", Heartbeat: time.Hour, MLTimeout: 50 * time.Millisecond}
fetcher := &stubFetcher{err: errors.New("fastapi unavailable")}
snapshots := cache.New(fetcher, cache.Options{RefreshInterval: time.Hour, Timeout: time.Second})
srv := New(cfg, snapshots, &stubML{}, sse.NewHub(time.Hour))
ts := httptest.NewServer(srv.Handler())
defer ts.Close()
@@ -143,6 +231,70 @@ func TestReadyzWithoutData(t *testing.T) {
if ready["ok"] || ready["data"] || ready["model"] {
t.Fatalf("readyz body=%v, want all false", ready)
}
// One failed bootstrap attempt must not make the cache ready.
if err := snapshots.Refresh(context.Background()); err == nil {
t.Fatal("expected the bootstrap refresh to fail")
}
resp = getJSON(t, ts.URL+"/api/readyz", &ready)
if resp.StatusCode != http.StatusServiceUnavailable || ready["ok"] {
t.Fatalf("readyz after failed refresh: status=%d body=%v", resp.StatusCode, ready)
}
}
func TestReadyzSurvivesFastAPIFailureAfterFirstSnapshot(t *testing.T) {
snapshots, fetcher := newFixtureCache(t)
cfg := config.Config{Addr: ":0", FastAPIURL: "http://127.0.0.1:1", Heartbeat: time.Hour, MLTimeout: 50 * time.Millisecond}
srv := New(cfg, snapshots, &stubML{}, sse.NewHub(time.Hour))
ts := httptest.NewServer(srv.Handler())
defer ts.Close()
var ready map[string]bool
resp := getJSON(t, ts.URL+"/api/readyz", &ready)
if resp.StatusCode != http.StatusOK || !ready["ok"] || !ready["data"] || !ready["model"] {
t.Fatalf("readyz status=%d body=%v", resp.StatusCode, ready)
}
fetcher.fail(errors.New("connection refused"))
if err := snapshots.Refresh(context.Background()); err == nil {
t.Fatal("expected the refresh to fail")
}
resp = getJSON(t, ts.URL+"/api/readyz", &ready)
if resp.StatusCode != http.StatusOK || !ready["ok"] || !ready["data"] || !ready["model"] {
t.Fatalf("readyz after FastAPI failure: status=%d body=%v, want 200 with the last snapshot", resp.StatusCode, ready)
}
var meta model.Meta
getJSON(t, ts.URL+"/api/meta", &meta)
if meta.Cache.Failures != 1 || meta.Cache.LastError == nil ||
!strings.Contains(*meta.Cache.LastError, "connection refused") {
t.Fatalf("cache status=%+v", meta.Cache)
}
if meta.Cache.LastSuccessAt == nil || meta.Cache.NextRetryAt == nil {
t.Fatalf("cache timestamps missing: %+v", meta.Cache)
}
// The context endpoint comes from the snapshot; it is now stale.
var ctx model.ContextResponse
getJSON(t, ts.URL+"/api/context", &ctx)
if !ctx.Cached {
t.Error("context cached=false after a failed refresh, want true")
}
// A later success clears the failure state.
fetcher.recover(loadFixtureBootstrap(t))
if err := snapshots.Refresh(context.Background()); err != nil {
t.Fatalf("recovery refresh: %v", err)
}
getJSON(t, ts.URL+"/api/meta", &meta)
if meta.Cache.Failures != 0 || meta.Cache.LastError != nil || meta.Cache.NextRetryAt != nil {
t.Fatalf("cache status after recovery=%+v", meta.Cache)
}
getJSON(t, ts.URL+"/api/context", &ctx)
if ctx.Cached {
t.Error("context cached=true after a successful refresh, want false")
}
}
func TestMetaSensorsAndVersionHeaders(t *testing.T) {
@@ -156,6 +308,10 @@ func TestMetaSensorsAndVersionHeaders(t *testing.T) {
if meta.Coverage.Sensors != 8 || meta.ModelVersion == "" || meta.ContextVersion == "" {
t.Fatalf("meta=%+v", meta)
}
if meta.Cache.Source != "fastapi" || meta.Cache.LastSuccessAt == nil ||
meta.Cache.LastError != nil || meta.Cache.NextRetryAt != nil || meta.Cache.Failures != 0 {
t.Fatalf("meta cache=%+v", meta.Cache)
}
for _, header := range []string{"X-Data-Version", "X-Model-Version", "X-Context-Version"} {
if resp.Header.Get(header) == "" {
t.Errorf("missing %s header", header)
@@ -180,6 +336,58 @@ func TestMetaSensorsAndVersionHeaders(t *testing.T) {
}
}
func TestDataEndpointsFollowSnapshotRefresh(t *testing.T) {
snapshots, fetcher := newFixtureCache(t)
_, ts := newTestServerWithCache(t, snapshots, &stubML{}, nil)
var sensors []model.Sensor
getJSON(t, ts.URL+"/api/sensors", &sensors)
if len(sensors) != 8 {
t.Fatalf("sensors=%d, want 8", len(sensors))
}
updated := loadFixtureBootstrap(t)
updated.DataVersion = "2026-11-14T05:00:00Z"
updated.ContextVersion = "2026-11-14T05:00:00Z"
updated.Sensors = updated.Sensors[:1]
updated.Series = map[string][]model.Point{
updated.Sensors[0].ID: {{T: time.Date(2026, 11, 1, 10, 0, 0, 0, time.UTC), V: 42}},
}
fetcher.recover(updated)
if err := snapshots.Refresh(context.Background()); err != nil {
t.Fatalf("refresh: %v", err)
}
sensors = nil
resp := getJSON(t, ts.URL+"/api/sensors", &sensors)
if len(sensors) != 1 || sensors[0].ID != updated.Sensors[0].ID {
t.Fatalf("sensors after refresh=%+v", sensors)
}
if got := resp.Header.Get("X-Data-Version"); got != updated.DataVersion {
t.Errorf("X-Data-Version=%q, want %q", got, updated.DataVersion)
}
var series struct {
Points []model.Point `json:"points"`
}
getJSON(t, ts.URL+"/api/series?sensor="+updated.Sensors[0].ID+"&step=15m", &series)
if len(series.Points) != 1 || series.Points[0].V != 42 {
t.Fatalf("series after refresh=%+v", series.Points)
}
var ctx model.ContextResponse
getJSON(t, ts.URL+"/api/context", &ctx)
if ctx.UpdatedAt.IsZero() || ctx.Cached {
t.Fatalf("context after refresh=%+v", ctx)
}
var meta model.Meta
getJSON(t, ts.URL+"/api/meta", &meta)
if meta.DataVersion != updated.DataVersion || meta.Cache.Failures != 0 {
t.Fatalf("meta after refresh=%+v", meta)
}
}
func TestSeriesRangeStepAndErrors(t *testing.T) {
_, ts := newTestServer(t, &stubML{}, nil)
@@ -241,37 +449,96 @@ func TestContextEndpoint(t *testing.T) {
if ctx.UpdatedAt.IsZero() {
t.Error("expected updated_at")
}
if ctx.Cached {
t.Error("cached=true while the last snapshot refresh succeeded, want false")
}
}
func TestForecastFallsBackWhenMLIsDown(t *testing.T) {
_, ts := newTestServer(t, &stubML{}, nil)
func TestForecastServesLastGoodWhenMLIsDown(t *testing.T) {
live := forecastPoints(168)
var down bool
var mu sync.Mutex
client := &stubML{predict: func(context.Context, ml.PredictRequest) ([]model.ForecastPoint, error) {
mu.Lock()
defer mu.Unlock()
if down {
return nil, errors.New("connection refused")
}
return live, nil
}}
_, ts := newTestServer(t, client, nil)
var forecast model.ForecastResponse
resp := getJSON(t, ts.URL+"/api/forecast?sensor=hn-bahnhof-01&horizon=7d", &forecast)
if resp.StatusCode != http.StatusOK {
t.Fatalf("forecast status=%d", resp.StatusCode)
t.Fatalf("live forecast status=%d", resp.StatusCode)
}
if !forecast.Cached {
if forecast.Cached || len(forecast.Points) != 168 {
t.Fatalf("live forecast cached=%v points=%d, want false/168", forecast.Cached, len(forecast.Points))
}
mu.Lock()
down = true
mu.Unlock()
var fallback model.ForecastResponse
resp = getJSON(t, ts.URL+"/api/forecast?sensor=hn-bahnhof-01&horizon=7d", &fallback)
if resp.StatusCode != http.StatusOK {
t.Fatalf("fallback status=%d", resp.StatusCode)
}
if !fallback.Cached {
t.Error("cached=false, want true when FastAPI is down")
}
if len(forecast.Points) != 168 {
t.Fatalf("points=%d, want 168", len(forecast.Points))
if len(fallback.Points) != 168 || fallback.Points[0].T != forecast.Points[0].T ||
fallback.Points[0].V != forecast.Points[0].V {
t.Fatalf("fallback points do not match the last good forecast: %+v", fallback.Points[0])
}
if forecast.Scenario != "base" || forecast.DataVersion == "" || forecast.ModelVersion == "" {
t.Fatalf("forecast=%+v", forecast)
if fallback.Scenario != "base" || fallback.DataVersion == "" || fallback.ModelVersion == "" {
t.Fatalf("forecast=%+v", fallback)
}
}
func TestForecastTimeoutFallsBack(t *testing.T) {
func TestForecastWithoutLastGoodFails503(t *testing.T) {
_, ts := newTestServer(t, &stubML{}, nil)
var errBody model.ErrorBody
resp := getJSON(t, ts.URL+"/api/forecast?sensor=hn-bahnhof-01&horizon=7d", &errBody)
if resp.StatusCode != http.StatusServiceUnavailable {
t.Fatalf("status=%d, want 503", resp.StatusCode)
}
if errBody.Error.Code != "forecast_unavailable" {
t.Fatalf("error=%+v", errBody)
}
}
func TestForecastTimeoutServesLastGood(t *testing.T) {
live := forecastPoints(48)
var mu sync.Mutex
slow := false
client := &stubML{predict: func(ctx context.Context, _ ml.PredictRequest) ([]model.ForecastPoint, error) {
<-ctx.Done()
return nil, ctx.Err()
mu.Lock()
block := slow
mu.Unlock()
if block {
<-ctx.Done()
return nil, ctx.Err()
}
return live, nil
}}
_, ts := newTestServer(t, client, func(cfg *config.Config) { cfg.MLTimeout = 20 * time.Millisecond })
var primed model.ForecastResponse
resp := getJSON(t, ts.URL+"/api/forecast?sensor=hn-stadtgalerie-01&horizon=48h&scenario=sunny-saturday-market", &primed)
if resp.StatusCode != http.StatusOK || primed.Cached {
t.Fatalf("prime forecast status=%d cached=%v", resp.StatusCode, primed.Cached)
}
mu.Lock()
slow = true
mu.Unlock()
start := time.Now()
var forecast model.ForecastResponse
resp := getJSON(t, ts.URL+"/api/forecast?sensor=hn-stadtgalerie-01&horizon=48h&scenario=sunny-saturday-market", &forecast)
resp = getJSON(t, ts.URL+"/api/forecast?sensor=hn-stadtgalerie-01&horizon=48h&scenario=sunny-saturday-market", &forecast)
if resp.StatusCode != http.StatusOK {
t.Fatalf("forecast status=%d", resp.StatusCode)
}
@@ -327,7 +594,11 @@ func TestForecastValidation(t *testing.T) {
}
func TestStaffingPlan(t *testing.T) {
_, ts := newTestServer(t, &stubML{}, nil)
points := weekForecast(t)
client := &stubML{predict: func(context.Context, ml.PredictRequest) ([]model.ForecastPoint, error) {
return points, nil
}}
_, ts := newTestServer(t, client, nil)
start := time.Now()
var plan model.StaffingResponse
@@ -350,9 +621,12 @@ func TestStaffingPlan(t *testing.T) {
if plan.Summary.CoveragePct != 100 {
t.Errorf("coverage=%v, want 100", plan.Summary.CoveragePct)
}
if plan.Summary.CostEUR <= 0 || plan.Summary.FlatScheduleCostEUR <= 0 {
if plan.Summary.CostEUR != 2240 || plan.Summary.FlatScheduleCostEUR != 2240 {
t.Errorf("summary=%+v", plan.Summary)
}
if plan.Cached {
t.Error("cached=true, want false for a live plan")
}
if len(plan.Why) == 0 {
t.Error("expected a non-empty why list")
}
@@ -363,6 +637,57 @@ func TestStaffingPlan(t *testing.T) {
}
}
func TestStaffingFallsBackToLastGoodForecast(t *testing.T) {
points := weekForecast(t)
var mu sync.Mutex
down := false
client := &stubML{predict: func(context.Context, ml.PredictRequest) ([]model.ForecastPoint, error) {
mu.Lock()
defer mu.Unlock()
if down {
return nil, errors.New("connection refused")
}
return points, nil
}}
_, ts := newTestServer(t, client, nil)
var live model.StaffingResponse
resp := getJSON(t, ts.URL+"/api/staffing?sensor=hn-bahnhof-01&scenario=base&standard=standard", &live)
if resp.StatusCode != http.StatusOK || live.Cached {
t.Fatalf("live staffing status=%d cached=%v", resp.StatusCode, live.Cached)
}
mu.Lock()
down = true
mu.Unlock()
// Different standard -> different plan key, so the handler must rebuild
// from the last good 7d forecast.
var fallback model.StaffingResponse
resp = getJSON(t, ts.URL+"/api/staffing?sensor=hn-bahnhof-01&scenario=base&standard=premium", &fallback)
if resp.StatusCode != http.StatusOK {
t.Fatalf("fallback staffing status=%d", resp.StatusCode)
}
if !fallback.Cached {
t.Error("cached=false, want true when built from the last good forecast")
}
if len(fallback.Hours) != 70 || len(fallback.Shifts) != 21 {
t.Fatalf("fallback plan hours=%d shifts=%d", len(fallback.Hours), len(fallback.Shifts))
}
}
func TestStaffingUnavailableWithoutForecast(t *testing.T) {
_, ts := newTestServer(t, &stubML{}, nil)
var errBody model.ErrorBody
resp := getJSON(t, ts.URL+"/api/staffing?sensor=hn-bahnhof-01&scenario=base&standard=standard", &errBody)
if resp.StatusCode != http.StatusServiceUnavailable {
t.Fatalf("status=%d, want 503", resp.StatusCode)
}
if errBody.Error.Code != "staffing_unavailable" {
t.Fatalf("error=%+v", errBody)
}
}
func TestStaffingValidation(t *testing.T) {
_, ts := newTestServer(t, &stubML{}, nil)
for _, tc := range []struct {
+26 -10
View File
@@ -14,9 +14,11 @@ func (s *Server) handleHealthz(w http.ResponseWriter, _ *http.Request) {
}
func (s *Server) handleReadyz(w http.ResponseWriter, _ *http.Request) {
ready := s.store != nil && s.store.Ready()
dataOK := s.store != nil && s.store.Len() > 0 && !s.store.Context().UpdatedAt.IsZero()
modelOK := s.store != nil && s.store.Meta().ModelVersion != ""
dataOK, modelOK := false, false
if s.cache != nil {
dataOK, modelOK = s.cache.Readiness()
}
ready := dataOK && modelOK
status := http.StatusOK
if !ready {
status = http.StatusServiceUnavailable
@@ -26,12 +28,16 @@ func (s *Server) handleReadyz(w http.ResponseWriter, _ *http.Request) {
func (s *Server) handleMeta(w http.ResponseWriter, _ *http.Request) {
s.setVersions(w)
writeJSON(w, http.StatusOK, s.store.Meta())
writeJSON(w, http.StatusOK, s.meta())
}
func (s *Server) handleSensors(w http.ResponseWriter, _ *http.Request) {
s.setVersions(w)
writeJSON(w, http.StatusOK, s.store.Sensors())
if s.cache == nil {
writeJSON(w, http.StatusOK, []model.Sensor{})
return
}
writeJSON(w, http.StatusOK, s.cache.Sensors())
}
func (s *Server) handleSeries(w http.ResponseWriter, r *http.Request) {
@@ -41,7 +47,11 @@ func (s *Server) handleSeries(w http.ResponseWriter, r *http.Request) {
writeError(w, http.StatusBadRequest, "bad_request", "sensor is required")
return
}
if _, ok := s.store.Sensor(sensor); !ok {
if s.cache == nil {
writeError(w, http.StatusNotFound, "not_found", "unknown sensor")
return
}
if _, ok := s.cache.Sensor(sensor); !ok {
writeError(w, http.StatusNotFound, "not_found", "unknown sensor")
return
}
@@ -54,7 +64,7 @@ func (s *Server) handleSeries(w http.ResponseWriter, r *http.Request) {
return
}
raw, _ := s.store.Series(sensor)
raw, _ := s.cache.Series(sensor)
if len(raw) == 0 {
s.setVersions(w)
writeJSON(w, http.StatusOK, map[string]any{"sensor": sensor, "step": step, "points": []model.Point{}})
@@ -69,7 +79,7 @@ func (s *Server) handleSeries(w http.ResponseWriter, r *http.Request) {
return
}
to = parsed.UTC()
} else if len(raw) > 0 {
} else {
to = raw[len(raw)-1].T.UTC()
}
@@ -143,12 +153,18 @@ func hourlyMean(points []model.Point) []model.Point {
}
func (s *Server) handleContext(w http.ResponseWriter, _ *http.Request) {
ctx := s.store.Context()
var ctx model.ContextData
cached := true
if s.cache != nil {
ctx = s.cache.Context()
// A failed refresh keeps the old context in place, so mark it cached.
cached = !s.cache.Fresh()
}
s.setVersions(w)
writeJSON(w, http.StatusOK, model.ContextResponse{
Weather: ctx.Weather,
Holidays: ctx.Holidays,
UpdatedAt: ctx.UpdatedAt,
Cached: false,
Cached: cached,
})
}
+77 -30
View File
@@ -20,7 +20,11 @@ func (s *Server) handleForecast(w http.ResponseWriter, r *http.Request) {
writeError(w, http.StatusBadRequest, "bad_request", "sensor is required")
return
}
if _, ok := s.store.Sensor(sensor); !ok {
if s.cache == nil {
writeError(w, http.StatusNotFound, "not_found", "unknown sensor")
return
}
if _, ok := s.cache.Sensor(sensor); !ok {
writeError(w, http.StatusNotFound, "not_found", "unknown sensor")
return
}
@@ -53,11 +57,14 @@ func (s *Server) handleForecast(w http.ResponseWriter, r *http.Request) {
if !ok {
slog.Warn("forecast unavailable", "sensor", sensor, "horizon", horizon, "scenario", scenarioName, "error", err)
writeError(w, http.StatusServiceUnavailable, "forecast_unavailable",
"no live forecast and no baked forecast for this sensor, scenario and horizon")
"no live forecast and no last good forecast for this sensor, scenario and horizon")
return
}
} else {
s.rememberForecast(sensor, horizon, scenarioName, points)
}
meta := s.meta()
s.setVersions(w)
writeJSON(w, http.StatusOK, model.ForecastResponse{
Sensor: sensor,
@@ -65,9 +72,9 @@ func (s *Server) handleForecast(w http.ResponseWriter, r *http.Request) {
Scenario: scenarioName,
Points: points,
Cached: cached,
DataVersion: s.store.Meta().DataVersion,
ModelVersion: s.store.Meta().ModelVersion,
ContextVersion: s.store.Meta().ContextVersion,
DataVersion: meta.DataVersion,
ModelVersion: meta.ModelVersion,
ContextVersion: meta.ContextVersion,
})
}
@@ -78,8 +85,9 @@ func (s *Server) liveForecast(ctx context.Context, sensor, horizon, scenarioName
ctx, cancel := context.WithTimeout(ctx, s.cfg.MLTimeout)
defer cancel()
ctxData := s.store.Context()
ctxData := s.cache.Context()
precip := ctxData.Weather.Now.PrecipMM
// A zero override means "no scenario rain"; keep the cached current value.
if state.Scenario.Weather.PrecipMM > 0 {
precip = state.Scenario.Weather.PrecipMM
}
@@ -100,8 +108,9 @@ func (s *Server) liveForecast(ctx context.Context, sensor, horizon, scenarioName
return sortedForecast(points), nil
}
// fallbackForecast prefers live /simulate output for the current scenario, then
// the baked scenario grid. It never labels base data as another scenario.
// fallbackForecast prefers the live /simulate grid for the current scenario,
// then the last good /predict response for this exact key. It never labels base
// data as another scenario.
func (s *Server) fallbackForecast(sensor, scenarioName, horizon string) ([]model.ForecastPoint, bool) {
s.mu.RLock()
if s.simFor == scenarioName {
@@ -115,10 +124,31 @@ func (s *Server) fallbackForecast(sensor, scenarioName, horizon string) ([]model
}
s.mu.RUnlock()
if points, ok := s.store.Forecast(sensor, scenarioName, horizon); ok {
return sortedForecast(points), true
return s.lastGoodForecast(sensor, horizon, scenarioName)
}
func forecastKey(sensor, horizon, scenarioName string) string {
return sensor + "|" + horizon + "|" + scenarioName
}
func (s *Server) rememberForecast(sensor, horizon, scenarioName string, points []model.ForecastPoint) {
if len(points) == 0 {
return
}
return nil, false
copied := sortedForecast(points)
s.mu.Lock()
s.lastForecast[forecastKey(sensor, horizon, scenarioName)] = copied
s.mu.Unlock()
}
func (s *Server) lastGoodForecast(sensor, horizon, scenarioName string) ([]model.ForecastPoint, bool) {
s.mu.RLock()
points, ok := s.lastForecast[forecastKey(sensor, horizon, scenarioName)]
s.mu.RUnlock()
if !ok || len(points) == 0 {
return nil, false
}
return sortedForecast(points), true
}
func sortedForecast(points []model.ForecastPoint) []model.ForecastPoint {
@@ -129,8 +159,11 @@ func sortedForecast(points []model.ForecastPoint) []model.ForecastPoint {
}
func (s *Server) holidayOn(t time.Time) (string, bool) {
if s.cache == nil {
return "", false
}
day := t.In(s.loc).Format("2006-01-02")
for _, holiday := range s.store.Context().Holidays {
for _, holiday := range s.cache.Context().Holidays {
if holiday.Date == day {
return holiday.Name, true
}
@@ -145,7 +178,11 @@ func (s *Server) handleStaffing(w http.ResponseWriter, r *http.Request) {
writeError(w, http.StatusBadRequest, "bad_request", "sensor is required")
return
}
if _, ok := s.store.Sensor(sensor); !ok {
if s.cache == nil {
writeError(w, http.StatusNotFound, "not_found", "unknown sensor")
return
}
if _, ok := s.cache.Sensor(sensor); !ok {
writeError(w, http.StatusNotFound, "not_found", "unknown sensor")
return
}
@@ -169,45 +206,49 @@ func (s *Server) handleStaffing(w http.ResponseWriter, r *http.Request) {
}
state := s.scenario.Get()
points, cached, ok := s.staffingForecast(sensor, scenarioName)
if !ok {
writeError(w, http.StatusServiceUnavailable, "staffing_unavailable",
"no 7d forecast for this sensor and scenario")
return
}
meta := s.meta()
s.mu.RLock()
generation := s.simGen
s.mu.RUnlock()
key := fmt.Sprintf("%s|%s|%s|%d|%d", sensor, scenarioName, standard, state.Version, generation)
key := fmt.Sprintf("%s|%s|%s|%s|%d|%d", sensor, scenarioName, standard,
meta.ContextVersion, state.Version, generation)
s.mu.RLock()
plan, hit := s.staffing[key]
s.mu.RUnlock()
if !hit {
built, err := staffing.Build(sensor, scenarioName, standard, points, s.store.Context())
points, cached, ok := s.staffingForecast(r.Context(), sensor, scenarioName, state)
if !ok {
writeError(w, http.StatusServiceUnavailable, "staffing_unavailable",
"no live or last good 7d forecast for this sensor and scenario")
return
}
built, err := staffing.Build(sensor, scenarioName, standard, points, s.cache.Context())
if err != nil {
slog.Warn("staffing build failed", "sensor", sensor, "scenario", scenarioName, "standard", standard, "error", err)
writeError(w, http.StatusServiceUnavailable, "staffing_unavailable", "could not build a plan from the available forecast")
return
}
built.Cached = cached
s.mu.Lock()
s.staffing[key] = built
s.mu.Unlock()
plan = built
}
plan.Cached = cached
plan.DataVersion = s.store.Meta().DataVersion
plan.ModelVersion = s.store.Meta().ModelVersion
plan.ContextVersion = s.store.Meta().ContextVersion
plan.DataVersion = meta.DataVersion
plan.ModelVersion = meta.ModelVersion
plan.ContextVersion = meta.ContextVersion
s.setVersions(w)
writeJSON(w, http.StatusOK, plan)
}
// staffingForecast returns the best 7d points and whether they came from the
// fallback grid rather than live /simulate output.
func (s *Server) staffingForecast(sensor, scenarioName string) ([]model.ForecastPoint, bool, bool) {
// staffingForecast returns 7d points for a plan. Prefer the scenario simulation
// grid, then /predict, and finally the last good forecast. The bool reports
// whether the points came from the cache fallback.
func (s *Server) staffingForecast(ctx context.Context, sensor, scenarioName string, state model.ScenarioState) ([]model.ForecastPoint, bool, bool) {
s.mu.RLock()
if s.simFor == scenarioName {
if horizons, ok := s.sim[sensor]; ok {
@@ -220,8 +261,14 @@ func (s *Server) staffingForecast(sensor, scenarioName string) ([]model.Forecast
}
s.mu.RUnlock()
if points, ok := s.store.Forecast(sensor, scenarioName, "7d"); ok {
return sortedForecast(points), scenarioName != "base", true
points, err := s.liveForecast(ctx, sensor, "7d", scenarioName, state)
if err == nil {
s.rememberForecast(sensor, "7d", scenarioName, points)
return points, false, true
}
slog.Warn("staffing forecast fallback", "sensor", sensor, "scenario", scenarioName, "error", err)
if points, ok := s.lastGoodForecast(sensor, "7d", scenarioName); ok {
return points, true, true
}
return nil, false, false
}
+8
View File
@@ -83,6 +83,14 @@ func (s *Server) refreshSimulation(state model.ScenarioState) {
s.simFor = state.Scenario.Scenario
s.simGen++
s.staffing = make(map[string]model.StaffingResponse)
for sensorID, horizons := range forecasts {
for horizon, points := range horizons {
if len(points) == 0 {
continue
}
s.lastForecast[forecastKey(sensorID, horizon, state.Scenario.Scenario)] = sortedForecast(points)
}
}
s.mu.Unlock()
}
+2 -2
View File
@@ -13,8 +13,8 @@ import (
"pulsecast/api/internal/model"
)
// maxTrainBody bounds an uploaded CSV. The raw dataset is ~18 MB; 64 MB leaves
// room for a larger export without letting the API accept unbounded bodies.
// maxTrainBody bounds an uploaded CSV so a client cannot stream an unbounded
// body through the proxy.
const maxTrainBody = 64 << 20
// handleTrain proxies POST /api/train to FastAPI. The request is queued there,
+34 -21
View File
@@ -1,5 +1,5 @@
// Package api wires the HTTP handlers, scenario state, SSE hub and cache
// fallback into the Go service described by CONTRACT.md.
// fallback into the Go service.
package api
import (
@@ -12,12 +12,12 @@ import (
"sync"
"time"
"pulsecast/api/internal/cache"
"pulsecast/api/internal/config"
"pulsecast/api/internal/ml"
"pulsecast/api/internal/model"
"pulsecast/api/internal/scenario"
"pulsecast/api/internal/sse"
"pulsecast/api/internal/store"
)
// MLClient is the FastAPI prediction surface the data handlers need.
@@ -37,22 +37,23 @@ type TrainClient interface {
// Server holds process-wide state. One instance per pod.
type Server struct {
cfg config.Config
store *store.Store
cache *cache.Cache
ml MLClient
hub *sse.Hub
scenario *scenario.State
loc *time.Location
now func() time.Time
mu sync.RWMutex
sim map[string]map[string][]model.ForecastPoint // sensor -> horizon -> points
simFor string // scenario the sim belongs to
simGen int // invalidates staffing cache
staffing map[string]model.StaffingResponse
mu sync.RWMutex
sim map[string]map[string][]model.ForecastPoint // sensor -> horizon -> points
simFor string // scenario the sim belongs to
simGen int // invalidates staffing cache
staffing map[string]model.StaffingResponse
lastForecast map[string][]model.ForecastPoint // sensor|horizon|scenario -> last live points
}
// New builds a server. hub must be non-nil; main runs hub.Run.
func New(cfg config.Config, st *store.Store, client MLClient, hub *sse.Hub) *Server {
func New(cfg config.Config, snapshots *cache.Cache, client MLClient, hub *sse.Hub) *Server {
loc, err := time.LoadLocation(model.StoreHours.TZ)
if err != nil {
loc = time.UTC
@@ -61,14 +62,15 @@ func New(cfg config.Config, st *store.Store, client MLClient, hub *sse.Hub) *Ser
cfg.MLTimeout = ml.DefaultTimeout
}
return &Server{
cfg: cfg,
store: st,
ml: client,
hub: hub,
scenario: scenario.New(),
loc: loc,
now: time.Now,
staffing: make(map[string]model.StaffingResponse),
cfg: cfg,
cache: snapshots,
ml: client,
hub: hub,
scenario: scenario.New(),
loc: loc,
now: time.Now,
staffing: make(map[string]model.StaffingResponse),
lastForecast: make(map[string][]model.ForecastPoint),
}
}
@@ -106,6 +108,8 @@ var methodTable = map[string][]string{
"/api/stream": {"GET"},
}
// routeCheck keeps unknown routes and methods in the JSON error envelope;
// ServeMux would otherwise answer them in plain text.
func (s *Server) routeCheck(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
methods, ok := methodTable[r.URL.Path]
@@ -194,10 +198,19 @@ func writeError(w http.ResponseWriter, status int, code, message string) {
_, _ = w.Write(raw)
}
// meta snapshots the cache metadata once per response, so versions and the
// cache block cannot disagree when a refresh lands mid-request.
func (s *Server) meta() model.Meta {
if s.cache == nil {
return model.Meta{}
}
return s.cache.Meta()
}
// setVersions adds the contract's version headers.
func (s *Server) setVersions(w http.ResponseWriter) {
data, modelVersion, context := s.store.Versions()
w.Header().Set("X-Data-Version", data)
w.Header().Set("X-Model-Version", modelVersion)
w.Header().Set("X-Context-Version", context)
meta := s.meta()
w.Header().Set("X-Data-Version", meta.DataVersion)
w.Header().Set("X-Model-Version", meta.ModelVersion)
w.Header().Set("X-Context-Version", meta.ContextVersion)
}
+354
View File
@@ -0,0 +1,354 @@
// Package cache keeps the FastAPI /bootstrap snapshot in memory and refreshes
// it in the background. A failed refresh never discards the last good snapshot;
// the next attempt follows the contract backoff 1/2/4/8/16/30 s (cap).
package cache
import (
"context"
"fmt"
"log/slog"
"sort"
"sync"
"time"
"pulsecast/api/internal/model"
)
// Source is the only snapshot producer: the FastAPI /bootstrap endpoint.
const Source = "fastapi"
// DefaultRefreshInterval is how often a healthy snapshot is refreshed.
const DefaultRefreshInterval = 60 * time.Second
// DefaultTimeout bounds one /bootstrap request.
const DefaultTimeout = 20 * time.Second
// DefaultRetryDelays is the contract backoff schedule. The last value repeats.
var DefaultRetryDelays = []time.Duration{
1 * time.Second,
2 * time.Second,
4 * time.Second,
8 * time.Second,
16 * time.Second,
30 * time.Second,
}
// Fetcher is the FastAPI bootstrap surface. *ml.Client implements it.
type Fetcher interface {
Bootstrap(ctx context.Context) (model.Bootstrap, error)
}
// Options configures refresh behaviour. Zero values use the defaults.
type Options struct {
RefreshInterval time.Duration
Timeout time.Duration
}
// snapshot is one immutable /bootstrap payload with its lookup indexes.
type snapshot struct {
DataVersion string
ModelVersion string
ContextVersion string
Sensors []model.Sensor
Series map[string][]model.Point
Context model.ContextData
Coverage model.Coverage
Metrics model.Metrics
Limits []string
sensorBy map[string]model.Sensor
}
// Cache holds the current snapshot and its refresh status. All methods are
// safe for concurrent use.
type Cache struct {
fetcher Fetcher
refresh time.Duration
timeout time.Duration
retry []time.Duration
now func() time.Time
mu sync.RWMutex
snapshot *snapshot
loaded bool
status model.CacheStatus
}
// New returns an empty cache. Call Run (or Refresh) to load the first snapshot.
func New(fetcher Fetcher, opts Options) *Cache {
if opts.RefreshInterval <= 0 {
opts.RefreshInterval = DefaultRefreshInterval
}
if opts.Timeout <= 0 {
opts.Timeout = DefaultTimeout
}
return &Cache{
fetcher: fetcher,
refresh: opts.RefreshInterval,
timeout: opts.Timeout,
retry: append([]time.Duration(nil), DefaultRetryDelays...),
now: time.Now,
status: model.CacheStatus{Source: Source},
}
}
// Run loads the snapshot immediately, then refreshes it. Failures keep the last
// good snapshot and retry with the contract backoff. Run returns when ctx ends.
func (c *Cache) Run(ctx context.Context) {
recovering := false
for {
wasLoaded := c.Loaded()
err := c.Refresh(ctx)
if ctx.Err() != nil {
return
}
var delay time.Duration
if err != nil {
recovering = true
status := c.Status()
delay = c.retryDelay(status.Failures)
slog.Warn("bootstrap refresh failed; keeping last good snapshot",
"error", err, "failures", status.Failures, "retry_in", delay)
} else {
delay = c.refresh
switch {
case !wasLoaded:
slog.Info("bootstrap snapshot loaded", "sensors", c.Len())
case recovering:
slog.Info("bootstrap refresh recovered", "sensors", c.Len())
}
recovering = false
}
timer := time.NewTimer(delay)
select {
case <-ctx.Done():
timer.Stop()
return
case <-timer.C:
}
}
}
// Refresh performs one fetch. A decode/validation failure counts as a failed
// attempt and leaves the previous snapshot untouched.
func (c *Cache) Refresh(ctx context.Context) error {
ctx, cancel := context.WithTimeout(ctx, c.timeout)
defer cancel()
payload, err := c.fetcher.Bootstrap(ctx)
now := c.now().UTC().Truncate(time.Second)
if err == nil {
var snapshot *snapshot
snapshot, err = newSnapshot(payload)
if err == nil {
c.mu.Lock()
c.snapshot = snapshot
c.loaded = true
c.status.LastSuccessAt = timePtr(now)
c.status.LastError = nil
c.status.NextRetryAt = nil
c.status.Failures = 0
c.mu.Unlock()
return nil
}
}
c.mu.Lock()
c.status.Failures++
message := err.Error()
c.status.LastError = &message
next := now.Add(c.retryDelay(c.status.Failures))
c.status.NextRetryAt = &next
c.mu.Unlock()
return err
}
func (c *Cache) retryDelay(failures int) time.Duration {
if failures < 1 {
failures = 1
}
if failures > len(c.retry) {
failures = len(c.retry)
}
return c.retry[failures-1]
}
func newSnapshot(payload model.Bootstrap) (*snapshot, error) {
switch {
case len(payload.Sensors) == 0:
return nil, fmt.Errorf("bootstrap snapshot: no sensors")
case payload.DataVersion == "":
return nil, fmt.Errorf("bootstrap snapshot: missing data_version")
case payload.ModelVersion == "":
return nil, fmt.Errorf("bootstrap snapshot: missing model_version")
case payload.ContextVersion == "":
return nil, fmt.Errorf("bootstrap snapshot: missing context_version")
case payload.Context.UpdatedAt.IsZero():
return nil, fmt.Errorf("bootstrap snapshot: missing context.updated_at")
}
snapshot := &snapshot{
DataVersion: payload.DataVersion,
ModelVersion: payload.ModelVersion,
ContextVersion: payload.ContextVersion,
Sensors: append([]model.Sensor(nil), payload.Sensors...),
Series: make(map[string][]model.Point, len(payload.Series)),
Context: payload.Context,
Coverage: payload.Meta.Coverage,
Metrics: payload.Meta.Metrics,
Limits: append([]string(nil), payload.Meta.Limits...),
sensorBy: make(map[string]model.Sensor, len(payload.Sensors)),
}
sort.Slice(snapshot.Sensors, func(i, j int) bool { return snapshot.Sensors[i].ID < snapshot.Sensors[j].ID })
for _, sensor := range snapshot.Sensors {
snapshot.sensorBy[sensor.ID] = sensor
}
for sensorID, points := range payload.Series {
copied := append([]model.Point(nil), points...)
sort.Slice(copied, func(i, j int) bool { return copied[i].T.Before(copied[j].T) })
snapshot.Series[sensorID] = copied
}
snapshot.Context.Holidays = append([]model.Holiday(nil), payload.Context.Holidays...)
snapshot.Context.Weather.Forecast = append([]model.WeatherPoint(nil), payload.Context.Weather.Forecast...)
return snapshot, nil
}
// Loaded reports whether a snapshot has ever been accepted.
func (c *Cache) Loaded() bool {
c.mu.RLock()
defer c.mu.RUnlock()
return c.loaded
}
// Ready reports whether a complete snapshot is available. FastAPI health is
// deliberately not part of this check: the cache itself is the safety net.
func (c *Cache) Ready() bool {
data, modelOK := c.Readiness()
return data && modelOK
}
// Readiness splits readiness into the data and model signals of /api/readyz.
func (c *Cache) Readiness() (data, modelOK bool) {
c.mu.RLock()
defer c.mu.RUnlock()
if !c.loaded || c.snapshot == nil {
return false, false
}
data = len(c.snapshot.Sensors) > 0 && !c.snapshot.Context.UpdatedAt.IsZero()
modelOK = c.snapshot.ModelVersion != ""
return data, modelOK
}
// Fresh reports whether the most recent refresh attempt succeeded. It is false
// before the first snapshot and after a failed refresh.
func (c *Cache) Fresh() bool {
c.mu.RLock()
defer c.mu.RUnlock()
return c.loaded && c.status.Failures == 0
}
// Status returns a copy of the cache status.
func (c *Cache) Status() model.CacheStatus {
c.mu.RLock()
defer c.mu.RUnlock()
return cloneStatus(c.status)
}
// Meta combines the snapshot metadata with the live cache status.
func (c *Cache) Meta() model.Meta {
c.mu.RLock()
defer c.mu.RUnlock()
meta := model.Meta{Cache: cloneStatus(c.status)}
if c.snapshot == nil {
return meta
}
meta.DataVersion = c.snapshot.DataVersion
meta.ModelVersion = c.snapshot.ModelVersion
meta.ContextVersion = c.snapshot.ContextVersion
meta.Coverage = c.snapshot.Coverage
meta.Metrics = c.snapshot.Metrics
meta.Limits = append([]string(nil), c.snapshot.Limits...)
return meta
}
// Sensors returns the sensors sorted by id.
func (c *Cache) Sensors() []model.Sensor {
c.mu.RLock()
snapshot := c.snapshot
c.mu.RUnlock()
if snapshot == nil {
return []model.Sensor{}
}
out := make([]model.Sensor, len(snapshot.Sensors))
copy(out, snapshot.Sensors)
return out
}
func (c *Cache) Sensor(id string) (model.Sensor, bool) {
c.mu.RLock()
defer c.mu.RUnlock()
if c.snapshot == nil {
return model.Sensor{}, false
}
sensor, ok := c.snapshot.sensorBy[id]
return sensor, ok
}
// Series returns the immutable series for a sensor, sorted by time.
func (c *Cache) Series(id string) ([]model.Point, bool) {
c.mu.RLock()
defer c.mu.RUnlock()
if c.snapshot == nil {
return nil, false
}
points, ok := c.snapshot.Series[id]
return points, ok
}
func (c *Cache) Context() model.ContextData {
c.mu.RLock()
defer c.mu.RUnlock()
if c.snapshot == nil {
return model.ContextData{}
}
return c.snapshot.Context
}
func (c *Cache) Versions() (data, modelVersion, contextVersion string) {
c.mu.RLock()
defer c.mu.RUnlock()
if c.snapshot == nil {
return "", "", ""
}
return c.snapshot.DataVersion, c.snapshot.ModelVersion, c.snapshot.ContextVersion
}
func (c *Cache) Len() int {
c.mu.RLock()
defer c.mu.RUnlock()
if c.snapshot == nil {
return 0
}
return len(c.snapshot.Sensors)
}
func cloneStatus(s model.CacheStatus) model.CacheStatus {
out := s
if s.LastSuccessAt != nil {
v := *s.LastSuccessAt
out.LastSuccessAt = &v
}
if s.LastError != nil {
v := *s.LastError
out.LastError = &v
}
if s.NextRetryAt != nil {
v := *s.NextRetryAt
out.NextRetryAt = &v
}
return out
}
func timePtr(t time.Time) *time.Time { return &t }
+295
View File
@@ -0,0 +1,295 @@
package cache
import (
"context"
"errors"
"sync"
"testing"
"time"
"pulsecast/api/internal/model"
)
type stubFetcher struct {
mu sync.Mutex
calls int
payload model.Bootstrap
err error
fn func(int) (model.Bootstrap, error)
}
func (s *stubFetcher) Bootstrap(context.Context) (model.Bootstrap, error) {
s.mu.Lock()
s.calls++
call := s.calls
payload, err, fn := s.payload, s.err, s.fn
s.mu.Unlock()
if fn != nil {
return fn(call)
}
return payload, err
}
func (s *stubFetcher) set(payload model.Bootstrap, err error) {
s.mu.Lock()
s.payload, s.err = payload, err
s.mu.Unlock()
}
func (s *stubFetcher) callCount() int {
s.mu.Lock()
defer s.mu.Unlock()
return s.calls
}
func fixturePayload() model.Bootstrap {
t0 := time.Date(2026, 11, 14, 3, 0, 0, 0, time.UTC)
return model.Bootstrap{
DataVersion: "2026-11-07T23:51:00Z",
ModelVersion: "histgb-2026-11-14",
ContextVersion: "2026-11-14T03:00:00Z",
Sensors: []model.Sensor{
{ID: "hn-zebra-01", Name: "Zebra", Lat: 49.1, Lon: 9.2, Mean: 1, P95: 2},
{ID: "hn-alpha-01", Name: "Alpha", Lat: 49.1, Lon: 9.2, Mean: 3, P95: 4},
},
Series: map[string][]model.Point{
"hn-alpha-01": {
{T: t0.Add(2 * time.Hour), V: 3},
{T: t0, V: 1},
{T: t0.Add(time.Hour), V: 2},
},
},
Context: model.ContextData{
Weather: model.Weather{
Now: model.WeatherNow{T: t0, TempC: 8.9, HumidityPct: 79},
Forecast: []model.WeatherPoint{{T: t0.Add(time.Hour), TempC: 8.4}},
},
Holidays: []model.Holiday{{Date: "2026-11-01", Name: "Allerheiligen"}},
UpdatedAt: t0,
},
Meta: model.BootstrapMeta{
Coverage: model.Coverage{Sensors: 8, From: "2026-06-12T00:00:00Z", To: "2026-11-07T23:51:00Z", CadenceMin: 15},
Metrics: model.Metrics{Model: "weather", MAE: 3.36, WAPE: 27.8, R2: 0.779},
Limits: []string{"estimated devices, not people"},
},
}
}
func TestRetryDelaySchedule(t *testing.T) {
c := New(&stubFetcher{}, Options{})
want := []time.Duration{1 * time.Second, 2 * time.Second, 4 * time.Second, 8 * time.Second, 16 * time.Second, 30 * time.Second, 30 * time.Second}
for i, expected := range want {
if got := c.retryDelay(i + 1); got != expected {
t.Errorf("retryDelay(%d)=%s, want %s", i+1, got, expected)
}
}
for _, failures := range []int{0, -1} {
if got := c.retryDelay(failures); got != time.Second {
t.Errorf("retryDelay(%d)=%s, want 1s", failures, got)
}
}
}
func TestRefreshLoadsSnapshotAndStatus(t *testing.T) {
fetcher := &stubFetcher{payload: fixturePayload()}
c := New(fetcher, Options{RefreshInterval: time.Minute, Timeout: time.Second})
fixed := time.Date(2026, 11, 14, 4, 0, 0, 0, time.UTC)
c.now = func() time.Time { return fixed }
if err := c.Refresh(context.Background()); err != nil {
t.Fatalf("Refresh: %v", err)
}
if !c.Loaded() || !c.Ready() || !c.Fresh() {
t.Fatalf("loaded=%v ready=%v fresh=%v", c.Loaded(), c.Ready(), c.Fresh())
}
data, modelOK := c.Readiness()
if !data || !modelOK {
t.Fatalf("readiness data=%v model=%v", data, modelOK)
}
if c.Len() != 2 {
t.Fatalf("Len=%d, want 2", c.Len())
}
sensors := c.Sensors()
if len(sensors) != 2 || sensors[0].ID != "hn-alpha-01" || sensors[1].ID != "hn-zebra-01" {
t.Fatalf("sensors not sorted: %+v", sensors)
}
if sensor, ok := c.Sensor("hn-zebra-01"); !ok || sensor.Name != "Zebra" {
t.Fatalf("Sensor: ok=%v sensor=%+v", ok, sensor)
}
if _, ok := c.Sensor("nope"); ok {
t.Fatal("unexpected sensor nope")
}
series, ok := c.Series("hn-alpha-01")
if !ok || len(series) != 3 || !series[0].T.Before(series[1].T) || !series[1].T.Before(series[2].T) {
t.Fatalf("series not sorted: ok=%v points=%+v", ok, series)
}
if ctx := c.Context(); ctx.UpdatedAt.IsZero() || len(ctx.Holidays) != 1 || len(ctx.Weather.Forecast) != 1 {
t.Fatalf("context=%+v", ctx)
}
dataVersion, modelVersion, contextVersion := c.Versions()
if dataVersion != "2026-11-07T23:51:00Z" || modelVersion != "histgb-2026-11-14" || contextVersion != "2026-11-14T03:00:00Z" {
t.Fatalf("versions=%q %q %q", dataVersion, modelVersion, contextVersion)
}
meta := c.Meta()
if meta.Coverage.Sensors != 8 || meta.Metrics.MAE != 3.36 || len(meta.Limits) != 1 {
t.Fatalf("meta=%+v", meta)
}
if meta.Cache.Source != Source || meta.Cache.LastSuccessAt == nil ||
!meta.Cache.LastSuccessAt.Equal(fixed) || meta.Cache.LastError != nil ||
meta.Cache.NextRetryAt != nil || meta.Cache.Failures != 0 {
t.Fatalf("cache status=%+v", meta.Cache)
}
}
func TestRefreshFailureKeepsLastGoodAndCountsBackoff(t *testing.T) {
fetcher := &stubFetcher{payload: fixturePayload()}
c := New(fetcher, Options{RefreshInterval: time.Minute, Timeout: time.Second})
fixed := time.Date(2026, 11, 14, 4, 0, 0, 0, time.UTC)
c.now = func() time.Time { return fixed }
if err := c.Refresh(context.Background()); err != nil {
t.Fatalf("prime: %v", err)
}
primed := c.Sensors()
fetcher.set(model.Bootstrap{}, errors.New("connection refused"))
if err := c.Refresh(context.Background()); err == nil {
t.Fatal("expected refresh failure")
}
if !c.Ready() || c.Fresh() {
t.Fatalf("ready=%v fresh=%v, want ready and not fresh", c.Ready(), c.Fresh())
}
if got := c.Sensors(); len(got) != len(primed) || got[0].ID != primed[0].ID {
t.Fatalf("last good snapshot lost: %+v", got)
}
status := c.Status()
if status.Failures != 1 || status.LastError == nil || *status.LastError != "connection refused" {
t.Fatalf("status=%+v", status)
}
if status.LastSuccessAt == nil || !status.LastSuccessAt.Equal(fixed) {
t.Fatalf("last success=%v", status.LastSuccessAt)
}
if status.NextRetryAt == nil || !status.NextRetryAt.Equal(fixed.Add(time.Second)) {
t.Fatalf("next retry=%v, want %s", status.NextRetryAt, fixed.Add(time.Second))
}
if err := c.Refresh(context.Background()); err == nil {
t.Fatal("expected second refresh failure")
}
status = c.Status()
if status.Failures != 2 || status.NextRetryAt == nil || !status.NextRetryAt.Equal(fixed.Add(2*time.Second)) {
t.Fatalf("second failure status=%+v", status)
}
fetcher.set(fixturePayload(), nil)
if err := c.Refresh(context.Background()); err != nil {
t.Fatalf("recovery: %v", err)
}
status = c.Status()
if status.Failures != 0 || status.LastError != nil || status.NextRetryAt != nil || !c.Fresh() {
t.Fatalf("recovered status=%+v fresh=%v", status, c.Fresh())
}
if status.LastSuccessAt == nil || !status.LastSuccessAt.Equal(fixed) {
t.Fatalf("last success after recovery=%v", status.LastSuccessAt)
}
}
func TestRefreshRejectsIncompleteSnapshot(t *testing.T) {
for name, mutate := range map[string]func(*model.Bootstrap){
"no sensors": func(b *model.Bootstrap) { b.Sensors = nil },
"no data version": func(b *model.Bootstrap) { b.DataVersion = "" },
"no model version": func(b *model.Bootstrap) { b.ModelVersion = "" },
"no context version": func(b *model.Bootstrap) { b.ContextVersion = "" },
"no context time": func(b *model.Bootstrap) { b.Context.UpdatedAt = time.Time{} },
} {
t.Run(name, func(t *testing.T) {
payload := fixturePayload()
mutate(&payload)
c := New(&stubFetcher{payload: payload}, Options{Timeout: time.Second})
fixed := time.Date(2026, 11, 14, 4, 0, 0, 0, time.UTC)
c.now = func() time.Time { return fixed }
if err := c.Refresh(context.Background()); err == nil {
t.Fatal("expected validation error")
}
if c.Loaded() || c.Ready() {
t.Fatalf("loaded=%v ready=%v after invalid snapshot", c.Loaded(), c.Ready())
}
if status := c.Status(); status.Failures != 1 || status.LastError == nil ||
status.NextRetryAt == nil || !status.NextRetryAt.Equal(fixed.Add(time.Second)) {
t.Fatalf("status=%+v", status)
}
})
}
}
func TestRunRetriesWithBackoffThenLoads(t *testing.T) {
fetcher := &stubFetcher{fn: func(call int) (model.Bootstrap, error) {
if call <= 2 {
return model.Bootstrap{}, errors.New("not ready")
}
return fixturePayload(), nil
}}
c := New(fetcher, Options{RefreshInterval: 10 * time.Millisecond, Timeout: time.Second})
c.retry = []time.Duration{5 * time.Millisecond, 10 * time.Millisecond, 20 * time.Millisecond}
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
done := make(chan struct{})
go func() {
c.Run(ctx)
close(done)
}()
deadline := time.Now().Add(2 * time.Second)
for !c.Loaded() {
if time.Now().After(deadline) {
t.Fatalf("cache did not load after retries; calls=%d status=%+v", fetcher.callCount(), c.Status())
}
time.Sleep(5 * time.Millisecond)
}
if got := fetcher.callCount(); got < 3 {
t.Fatalf("calls=%d, want at least 3", got)
}
if status := c.Status(); status.Failures != 0 || status.LastError != nil {
t.Fatalf("status after recovery=%+v", status)
}
cancel()
select {
case <-done:
case <-time.After(time.Second):
t.Fatal("Run did not stop after context cancellation")
}
}
func TestRunStopsWhenContextCancelled(t *testing.T) {
fetcher := &stubFetcher{err: errors.New("down")}
c := New(fetcher, Options{RefreshInterval: time.Hour, Timeout: time.Second})
c.retry = []time.Duration{time.Hour}
ctx, cancel := context.WithCancel(context.Background())
done := make(chan struct{})
go func() {
c.Run(ctx)
close(done)
}()
deadline := time.Now().Add(time.Second)
for fetcher.callCount() == 0 {
if time.Now().After(deadline) {
t.Fatal("Run never attempted the first fetch")
}
time.Sleep(time.Millisecond)
}
cancel()
select {
case <-done:
case <-time.After(time.Second):
t.Fatal("Run did not stop after context cancellation")
}
if status := c.Status(); status.Failures != 1 || status.NextRetryAt == nil {
t.Fatalf("status=%+v", status)
}
}
+14 -9
View File
@@ -10,13 +10,17 @@ import (
// Config is the process configuration.
type Config struct {
Addr string
DataDir string
FastAPIURL string
ControlUser string
ControlPass string
ControlToken string
Heartbeat time.Duration
MLTimeout time.Duration
// RefreshInterval is how often a healthy /bootstrap snapshot is refreshed.
RefreshInterval time.Duration
// BootstrapTimeout bounds one /bootstrap request.
BootstrapTimeout time.Duration
}
// Load reads configuration from env with local-development defaults.
@@ -26,14 +30,15 @@ type Config struct {
func Load() Config {
port := getenv("PORT", "8080")
return Config{
Addr: ":" + port,
DataDir: getenv("DATA_DIR", "testdata/data"),
FastAPIURL: getenv("FASTAPI_URL", "http://localhost:8000"),
ControlUser: os.Getenv("CONTROL_USER"),
ControlPass: os.Getenv("CONTROL_PASS"),
ControlToken: os.Getenv("CONTROL_TOKEN"),
Heartbeat: durationEnv("SSE_HEARTBEAT", 15*time.Second),
MLTimeout: durationEnv("ML_TIMEOUT", 1500*time.Millisecond),
Addr: ":" + port,
FastAPIURL: getenv("FASTAPI_URL", "http://localhost:8000"),
ControlUser: os.Getenv("CONTROL_USER"),
ControlPass: os.Getenv("CONTROL_PASS"),
ControlToken: os.Getenv("CONTROL_TOKEN"),
Heartbeat: durationEnv("SSE_HEARTBEAT", 15*time.Second),
MLTimeout: durationEnv("ML_TIMEOUT", 1500*time.Millisecond),
RefreshInterval: durationEnv("BOOTSTRAP_REFRESH", 60*time.Second),
BootstrapTimeout: durationEnv("BOOTSTRAP_TIMEOUT", 20*time.Second),
}
}
+23 -6
View File
@@ -6,22 +6,25 @@ import (
)
func TestLoadDefaults(t *testing.T) {
for _, key := range []string{"PORT", "DATA_DIR", "FASTAPI_URL", "CONTROL_USER", "CONTROL_PASS", "CONTROL_TOKEN", "SSE_HEARTBEAT", "ML_TIMEOUT"} {
for _, key := range []string{
"PORT", "FASTAPI_URL", "CONTROL_USER", "CONTROL_PASS", "CONTROL_TOKEN",
"SSE_HEARTBEAT", "ML_TIMEOUT", "BOOTSTRAP_REFRESH", "BOOTSTRAP_TIMEOUT",
} {
t.Setenv(key, "")
}
cfg := Load()
if cfg.Addr != ":8080" {
t.Errorf("Addr=%q", cfg.Addr)
}
if cfg.DataDir != "testdata/data" {
t.Errorf("DataDir=%q", cfg.DataDir)
}
if cfg.FastAPIURL != "http://localhost:8000" {
t.Errorf("FastAPIURL=%q", cfg.FastAPIURL)
}
if cfg.Heartbeat != 15*time.Second || cfg.MLTimeout != 1500*time.Millisecond {
t.Errorf("timeouts: heartbeat=%s ml=%s", cfg.Heartbeat, cfg.MLTimeout)
}
if cfg.RefreshInterval != 60*time.Second || cfg.BootstrapTimeout != 20*time.Second {
t.Errorf("snapshot: refresh=%s timeout=%s", cfg.RefreshInterval, cfg.BootstrapTimeout)
}
if cfg.ControlAuthConfigured() {
t.Error("control auth should be unconfigured by default")
}
@@ -32,16 +35,17 @@ func TestLoadDefaults(t *testing.T) {
func TestLoadFromEnv(t *testing.T) {
t.Setenv("PORT", "9999")
t.Setenv("DATA_DIR", "/data")
t.Setenv("FASTAPI_URL", "http://ml:8000")
t.Setenv("CONTROL_USER", "user")
t.Setenv("CONTROL_PASS", "pass")
t.Setenv("CONTROL_TOKEN", "token-123")
t.Setenv("SSE_HEARTBEAT", "5s")
t.Setenv("ML_TIMEOUT", "2s")
t.Setenv("BOOTSTRAP_REFRESH", "30s")
t.Setenv("BOOTSTRAP_TIMEOUT", "10s")
cfg := Load()
if cfg.Addr != ":9999" || cfg.DataDir != "/data" || cfg.FastAPIURL != "http://ml:8000" {
if cfg.Addr != ":9999" || cfg.FastAPIURL != "http://ml:8000" {
t.Fatalf("cfg=%+v", cfg)
}
if cfg.ControlToken != "token-123" {
@@ -50,6 +54,9 @@ func TestLoadFromEnv(t *testing.T) {
if cfg.Heartbeat != 5*time.Second || cfg.MLTimeout != 2*time.Second {
t.Fatalf("timeouts: heartbeat=%s ml=%s", cfg.Heartbeat, cfg.MLTimeout)
}
if cfg.RefreshInterval != 30*time.Second || cfg.BootstrapTimeout != 10*time.Second {
t.Fatalf("snapshot: refresh=%s timeout=%s", cfg.RefreshInterval, cfg.BootstrapTimeout)
}
if !cfg.ControlAuthConfigured() {
t.Fatal("control auth should be configured")
}
@@ -58,17 +65,27 @@ func TestLoadFromEnv(t *testing.T) {
func TestDurationAcceptsPlainSeconds(t *testing.T) {
t.Setenv("SSE_HEARTBEAT", "7")
t.Setenv("ML_TIMEOUT", "3")
t.Setenv("BOOTSTRAP_REFRESH", "45")
t.Setenv("BOOTSTRAP_TIMEOUT", "12")
cfg := Load()
if cfg.Heartbeat != 7*time.Second || cfg.MLTimeout != 3*time.Second {
t.Fatalf("timeouts: heartbeat=%s ml=%s", cfg.Heartbeat, cfg.MLTimeout)
}
if cfg.RefreshInterval != 45*time.Second || cfg.BootstrapTimeout != 12*time.Second {
t.Fatalf("snapshot: refresh=%s timeout=%s", cfg.RefreshInterval, cfg.BootstrapTimeout)
}
}
func TestDurationFallsBackOnInvalidValue(t *testing.T) {
t.Setenv("SSE_HEARTBEAT", "soon")
t.Setenv("ML_TIMEOUT", "-1s")
t.Setenv("BOOTSTRAP_REFRESH", "later")
t.Setenv("BOOTSTRAP_TIMEOUT", "-3s")
cfg := Load()
if cfg.Heartbeat != 15*time.Second || cfg.MLTimeout != 1500*time.Millisecond {
t.Fatalf("timeouts: heartbeat=%s ml=%s", cfg.Heartbeat, cfg.MLTimeout)
}
if cfg.RefreshInterval != 60*time.Second || cfg.BootstrapTimeout != 20*time.Second {
t.Fatalf("snapshot: refresh=%s timeout=%s", cfg.RefreshInterval, cfg.BootstrapTimeout)
}
}
+46 -5
View File
@@ -1,5 +1,6 @@
// Package ml is the FastAPI client. Every call is bounded by a 1.5 s timeout;
// callers fall back to baked JSON on any error.
// Package ml is the FastAPI client. Prediction calls use the caller's deadline
// (1.5 s by default); callers fall back to the last good forecast on any error.
// The larger /bootstrap snapshot gets its own, longer transport timeout.
package ml
import (
@@ -23,11 +24,16 @@ const DefaultTimeout = 1500 * time.Millisecond
// larger than prediction payloads, so they get more than DefaultTimeout.
const controlTimeout = 30 * time.Second
// bootstrapTimeout is the upper transport bound for the /bootstrap snapshot,
// which carries the full hourly series for every sensor.
const bootstrapTimeout = 30 * time.Second
// Client talks to the FastAPI ML service.
type Client struct {
baseURL string
http *http.Client
control *http.Client
bootstrap *http.Client
controlToken string
}
@@ -46,11 +52,46 @@ func NewWithToken(baseURL string, timeout time.Duration, controlToken string) *C
baseURL: strings.TrimRight(baseURL, "/"),
http: &http.Client{Timeout: timeout},
control: &http.Client{Timeout: controlTimeout},
bootstrap: &http.Client{Timeout: bootstrapTimeout},
controlToken: controlToken,
}
}
// PredictRequest mirrors CONTRACT.md §3 POST /predict.
// Bootstrap fetches GET /bootstrap, the runtime snapshot the API caches. The
// caller supplies the shorter request deadline; this method only validates the
// transport response and that the snapshot is usable.
func (c *Client) Bootstrap(ctx context.Context) (model.Bootstrap, error) {
var out model.Bootstrap
req, err := http.NewRequestWithContext(ctx, http.MethodGet, c.baseURL+"/bootstrap", nil)
if err != nil {
return out, fmt.Errorf("ml: build request: %w", err)
}
req.Header.Set("Accept", "application/json")
resp, err := c.bootstrap.Do(req)
if err != nil {
return out, fmt.Errorf("ml: /bootstrap: %w", err)
}
defer resp.Body.Close()
if resp.StatusCode < 200 || resp.StatusCode > 299 {
snippet, _ := io.ReadAll(io.LimitReader(resp.Body, 256))
return out, fmt.Errorf("ml: /bootstrap: status %d: %s", resp.StatusCode, strings.TrimSpace(string(snippet)))
}
raw, err := io.ReadAll(io.LimitReader(resp.Body, 64<<20))
if err != nil {
return out, fmt.Errorf("ml: read /bootstrap: %w", err)
}
if err := json.Unmarshal(raw, &out); err != nil {
return out, fmt.Errorf("ml: decode /bootstrap: %w", err)
}
if len(out.Sensors) == 0 {
return out, fmt.Errorf("ml: /bootstrap returned no sensors")
}
return out, nil
}
// PredictRequest mirrors the API contract POST /predict.
type PredictRequest struct {
Sensor string `json:"sensor"`
Horizon string `json:"horizon"`
@@ -65,7 +106,7 @@ type WeatherInput struct {
PrecipMM float64 `json:"precip_mm"`
}
// SimulateRequest mirrors CONTRACT.md §3 POST /simulate.
// SimulateRequest mirrors the API contract POST /simulate.
type SimulateRequest struct {
Scenario string `json:"scenario"`
Weather model.WeatherInput `json:"weather"`
@@ -78,7 +119,7 @@ type TrainRequest struct {
Model string `json:"model"`
}
// TrainModels are the accepted training targets from CONTRACT.md §3.
// TrainModels are the accepted training targets.
var TrainModels = []string{"small", "forecast"}
// Predict calls POST /predict and returns the forecast points. The contract
+88
View File
@@ -121,6 +121,94 @@ func TestSimulate(t *testing.T) {
}
}
func TestBootstrapFetchesSnapshot(t *testing.T) {
var gotMethod, gotPath, gotAuth string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
gotMethod, gotPath, gotAuth = r.Method, r.URL.Path, r.Header.Get("Authorization")
_, _ = w.Write([]byte(`{
"data_version":"2026-11-07T23:51:00Z",
"model_version":"histgb-2026-11-14",
"context_version":"2026-11-14T03:00:00Z",
"sensors":[{"id":"hn-bahnhof-01","name":"Bahnhof","lat":49.14,"lon":9.2,"mean":24.2,"p95":88.2,"battery":71}],
"series":{"hn-bahnhof-01":[{"t":"2026-11-01T10:00:00Z","v":12.3}]},
"context":{"weather":{"now":{"t":"2026-11-14T03:00:00Z","temp_c":8.9,"precip_mm":0,"cloud_pct":80,"wind_kmh":11,"humidity_pct":79},"forecast":[]},"holidays":[{"date":"2026-11-01","name":"Allerheiligen"}],"updated_at":"2026-11-14T03:00:00Z"},
"meta":{"coverage":{"sensors":8,"from":"2026-06-12T00:00:00Z","to":"2026-11-07T23:51:00Z","cadence_min":15},"metrics":{"model":"weather","mae":3.36,"wape":27.8,"r2":0.779,"baseline_mae":5.4,"baseline_wape":44.7},"limits":["estimated devices, not people"]}
}`))
}))
defer srv.Close()
// The control token is for /train only; /bootstrap must stay unauthenticated.
payload, err := NewWithToken(srv.URL, time.Second, "secret-token").Bootstrap(context.Background())
if err != nil {
t.Fatalf("Bootstrap: %v", err)
}
if gotMethod != http.MethodGet || gotPath != "/bootstrap" {
t.Fatalf("request=%s %s, want GET /bootstrap", gotMethod, gotPath)
}
if gotAuth != "" {
t.Errorf("Authorization=%q, want none", gotAuth)
}
if len(payload.Sensors) != 1 || payload.Sensors[0].ID != "hn-bahnhof-01" || payload.Sensors[0].Battery == nil {
t.Fatalf("sensors=%+v", payload.Sensors)
}
if len(payload.Series["hn-bahnhof-01"]) != 1 || payload.Series["hn-bahnhof-01"][0].V != 12.3 {
t.Fatalf("series=%+v", payload.Series)
}
if payload.Context.UpdatedAt.IsZero() || len(payload.Context.Holidays) != 1 {
t.Fatalf("context=%+v", payload.Context)
}
if payload.DataVersion == "" || payload.ModelVersion == "" || payload.ContextVersion == "" {
t.Fatalf("versions=%+v", payload)
}
if payload.Meta.Coverage.Sensors != 8 || payload.Meta.Metrics.MAE != 3.36 || len(payload.Meta.Limits) != 1 {
t.Fatalf("meta=%+v", payload.Meta)
}
}
func TestBootstrapRejectsErrorAndEmptySensors(t *testing.T) {
boom := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
http.Error(w, "not ready", http.StatusServiceUnavailable)
}))
defer boom.Close()
if _, err := New(boom.URL, time.Second).Bootstrap(context.Background()); err == nil {
t.Fatal("expected an error for HTTP 503")
}
empty := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
_, _ = w.Write([]byte(`{"sensors":[]}`))
}))
defer empty.Close()
if _, err := New(empty.URL, time.Second).Bootstrap(context.Background()); err == nil {
t.Fatal("expected an error for an empty sensor list")
}
bad := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
_, _ = w.Write([]byte(`{`))
}))
defer bad.Close()
if _, err := New(bad.URL, time.Second).Bootstrap(context.Background()); err == nil {
t.Fatal("expected an error for invalid JSON")
}
}
func TestBootstrapTimeout(t *testing.T) {
slow := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
time.Sleep(150 * time.Millisecond)
_, _ = w.Write([]byte(`{"sensors":[{"id":"s"}]}`))
}))
defer slow.Close()
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Millisecond)
defer cancel()
start := time.Now()
if _, err := New(slow.URL, time.Second).Bootstrap(ctx); err == nil {
t.Fatal("expected a context timeout")
}
if elapsed := time.Since(start); elapsed > time.Second {
t.Fatalf("timeout took %s", elapsed)
}
}
func TestNewUsesDefaultTimeout(t *testing.T) {
client := New("http://example.invalid", 0)
if client.http.Timeout != DefaultTimeout {
+40 -8
View File
@@ -1,10 +1,11 @@
// Package model contains the wire types shared by the HTTP handlers, the
// baked-data store and the FastAPI client. JSON tags mirror CONTRACT.md.
// in-memory snapshot cache and the FastAPI client. JSON tags mirror the API
// contract.
package model
import "time"
// Horizons and enums from CONTRACT.md §1.
// Horizons and enums from the API contract.
var (
Horizons = []string{"15m", "1h", "48h", "7d"}
Scenarios = []string{"base", "sunny-saturday-market", "rainy-saturday", "weekday-event"}
@@ -66,12 +67,43 @@ type Metrics struct {
// Meta is GET /api/meta.
type Meta struct {
DataVersion string `json:"data_version"`
ModelVersion string `json:"model_version"`
ContextVersion string `json:"context_version"`
Coverage Coverage `json:"coverage"`
Metrics Metrics `json:"metrics"`
Limits []string `json:"limits"`
DataVersion string `json:"data_version"`
ModelVersion string `json:"model_version"`
ContextVersion string `json:"context_version"`
Coverage Coverage `json:"coverage"`
Metrics Metrics `json:"metrics"`
Limits []string `json:"limits"`
Cache CacheStatus `json:"cache"`
}
// CacheStatus is the runtime snapshot state exposed by GET /api/meta.
// LastError, LastSuccessAt and NextRetryAt are null when they do not apply.
type CacheStatus struct {
Source string `json:"source"`
LastSuccessAt *time.Time `json:"last_success_at"`
LastError *string `json:"last_error"`
NextRetryAt *time.Time `json:"next_retry_at"`
Failures int `json:"failures"`
}
// BootstrapMeta is the nested meta block of FastAPI GET /bootstrap.
type BootstrapMeta struct {
Coverage Coverage `json:"coverage"`
Metrics Metrics `json:"metrics"`
Limits []string `json:"limits"`
}
// Bootstrap is the snapshot FastAPI serves at GET /bootstrap and Go keeps in
// memory. Forecasts are deliberately absent: they come from /predict and are
// cached separately as they are requested.
type Bootstrap struct {
DataVersion string `json:"data_version"`
ModelVersion string `json:"model_version"`
ContextVersion string `json:"context_version"`
Sensors []Sensor `json:"sensors"`
Series map[string][]Point `json:"series"`
Context ContextData `json:"context"`
Meta BootstrapMeta `json:"meta"`
}
// WeatherNow is the current conditions block.
+4 -4
View File
@@ -1,6 +1,5 @@
// Package staffing implements the deterministic decision heuristic from
// CONTRACT.md §5. It never calls the model: it consumes forecast points and
// turns them into an explainable shift plan.
// Package staffing turns forecast points into an explainable shift plan using
// the deterministic heuristic in the API contract. It never calls the model.
package staffing
import (
@@ -12,7 +11,8 @@ import (
"pulsecast/api/internal/model"
)
// Build creates a plan for one sensor/scenario/standard.
// Build creates a plan for one sensor/scenario/standard from the forecast's
// complete store days.
func Build(sensor, scenario, standard string, points []model.ForecastPoint, ctx model.ContextData) (model.StaffingResponse, error) {
perStaff, ok := visitorsPerStaff(standard)
if !ok {
-189
View File
@@ -1,189 +0,0 @@
// Package store loads the baked JSON artifacts described in CONTRACT.md §4.
// The data is immutable after Load; all accessors are safe for concurrent use.
package store
import (
"encoding/json"
"errors"
"fmt"
"os"
"path/filepath"
"sort"
"strings"
"pulsecast/api/internal/model"
)
// Store is the in-memory baked dataset.
type Store struct {
meta model.Meta
sensors []model.Sensor
sensorBy map[string]model.Sensor
series map[string][]model.Point
forecasts map[string]map[string]map[string][]model.ForecastPoint
context model.ContextData
}
// Load reads every artifact from dir. It returns a usable (possibly partial)
// store together with a joined error so callers can start the API and let
// /api/readyz report 503 instead of crash-looping.
func Load(dir string) (*Store, error) {
s := &Store{
sensorBy: make(map[string]model.Sensor),
series: make(map[string][]model.Point),
forecasts: make(map[string]map[string]map[string][]model.ForecastPoint),
}
var errs []error
if err := readJSON(filepath.Join(dir, "sensors.json"), &s.sensors); err != nil {
errs = append(errs, err)
}
sort.Slice(s.sensors, func(i, j int) bool { return s.sensors[i].ID < s.sensors[j].ID })
for _, sensor := range s.sensors {
s.sensorBy[sensor.ID] = sensor
}
if err := readJSON(filepath.Join(dir, "metrics.json"), &s.meta); err != nil {
errs = append(errs, err)
}
if err := readJSON(filepath.Join(dir, "context.json"), &s.context); err != nil {
errs = append(errs, err)
}
if err := loadSeries(dir, s); err != nil {
errs = append(errs, err)
}
if err := readJSON(filepath.Join(dir, "forecasts.json"), &s.forecasts); err != nil {
errs = append(errs, err)
}
for sensorID, scenarios := range s.forecasts {
for scenario, horizons := range scenarios {
for horizon, points := range horizons {
sort.Slice(points, func(i, j int) bool { return points[i].T.Before(points[j].T) })
horizons[horizon] = points
}
scenarios[scenario] = horizons
}
s.forecasts[sensorID] = scenarios
}
return s, errors.Join(errs...)
}
func loadSeries(dir string, s *Store) error {
var errs []error
// Contract allows a single series.json mapping sensor id -> points.
var combined map[string][]model.Point
err := readJSON(filepath.Join(dir, "series.json"), &combined)
switch {
case err == nil:
for sensorID, points := range combined {
sort.Slice(points, func(i, j int) bool { return points[i].T.Before(points[j].T) })
s.series[sensorID] = points
}
case !errors.Is(err, os.ErrNotExist):
errs = append(errs, err)
}
// Contract also allows series/<id>.json; these override the combined file.
entries, err := os.ReadDir(filepath.Join(dir, "series"))
if err != nil {
if !errors.Is(err, os.ErrNotExist) {
errs = append(errs, fmt.Errorf("series dir: %w", err))
}
return errors.Join(errs...)
}
for _, entry := range entries {
if entry.IsDir() || !strings.HasSuffix(entry.Name(), ".json") {
continue
}
var points []model.Point
path := filepath.Join(dir, "series", entry.Name())
if err := readJSON(path, &points); err != nil {
errs = append(errs, err)
continue
}
sort.Slice(points, func(i, j int) bool { return points[i].T.Before(points[j].T) })
s.series[strings.TrimSuffix(entry.Name(), ".json")] = points
}
return errors.Join(errs...)
}
func readJSON(path string, dst any) error {
raw, err := os.ReadFile(path)
if err != nil {
return fmt.Errorf("read %s: %w", path, err)
}
if err := json.Unmarshal(raw, dst); err != nil {
return fmt.Errorf("decode %s: %w", path, err)
}
return nil
}
// Meta returns the model/data metadata.
func (s *Store) Meta() model.Meta { return s.meta }
// Sensors returns the sensors sorted by id.
func (s *Store) Sensors() []model.Sensor {
out := make([]model.Sensor, len(s.sensors))
copy(out, s.sensors)
return out
}
// Sensor looks up one sensor.
func (s *Store) Sensor(id string) (model.Sensor, bool) {
sensor, ok := s.sensorBy[id]
return sensor, ok
}
// Series returns the raw (15-minute) series for a sensor.
func (s *Store) Series(id string) ([]model.Point, bool) {
points, ok := s.series[id]
return points, ok
}
// Forecast returns baked points for sensor/scenario/horizon.
func (s *Store) Forecast(sensor, scenario, horizon string) ([]model.ForecastPoint, bool) {
scenarios, ok := s.forecasts[sensor]
if !ok {
return nil, false
}
horizons, ok := scenarios[scenario]
if !ok {
return nil, false
}
points, ok := horizons[horizon]
if !ok || len(points) == 0 {
return nil, false
}
return points, true
}
// Context returns the baked weather/holiday context.
func (s *Store) Context() model.ContextData { return s.context }
// Versions returns data, model and context version strings.
func (s *Store) Versions() (data, modelVersion, context string) {
return s.meta.DataVersion, s.meta.ModelVersion, s.meta.ContextVersion
}
// Ready reports whether the minimum baked set is present. FastAPI health is
// deliberately not part of readiness: the cache fallback is the safety net.
func (s *Store) Ready() bool {
if len(s.sensors) == 0 || s.meta.ModelVersion == "" || s.context.UpdatedAt.IsZero() {
return false
}
for _, sensor := range s.sensors {
if _, ok := s.Forecast(sensor.ID, "base", "7d"); ok {
return true
}
}
return false
}
// Len returns the number of loaded sensors (used in tests/logs).
func (s *Store) Len() int { return len(s.sensors) }
-65
View File
@@ -1,65 +0,0 @@
package store
import (
"testing"
)
func TestLoadFixtures(t *testing.T) {
st, err := Load("../../testdata/data")
if err != nil {
t.Fatalf("Load: %v", err)
}
if !st.Ready() {
t.Fatal("Ready()=false for the fixture dataset")
}
if st.Len() != 8 {
t.Fatalf("Len()=%d, want 8", st.Len())
}
sensors := st.Sensors()
for i := 1; i < len(sensors); i++ {
if sensors[i-1].ID >= sensors[i].ID {
t.Fatalf("sensors not sorted: %q then %q", sensors[i-1].ID, sensors[i].ID)
}
}
if _, ok := st.Sensor("hn-bahnhof-01"); !ok {
t.Fatal("missing hn-bahnhof-01")
}
if _, ok := st.Sensor("nope"); ok {
t.Fatal("unexpected sensor nope")
}
if points, ok := st.Series("hn-bahnhof-01"); !ok || len(points) == 0 {
t.Fatalf("series: ok=%v points=%d", ok, len(points))
}
if points, ok := st.Forecast("hn-bahnhof-01", "sunny-saturday-market", "7d"); !ok || len(points) != 168 {
t.Fatalf("forecast: ok=%v points=%d", ok, len(points))
}
if _, ok := st.Forecast("hn-bahnhof-01", "base", "2h"); ok {
t.Fatal("unexpected horizon 2h")
}
if ctx := st.Context(); ctx.UpdatedAt.IsZero() || len(ctx.Weather.Forecast) == 0 || len(ctx.Holidays) == 0 {
t.Fatalf("context incomplete: %+v", ctx)
}
data, modelVersion, context := st.Versions()
if data == "" || modelVersion == "" || context == "" {
t.Fatalf("versions: data=%q model=%q context=%q", data, modelVersion, context)
}
}
func TestLoadMissingDirectoryIsPartial(t *testing.T) {
st, err := Load(t.TempDir())
if err == nil {
t.Fatal("expected an error for an empty data dir")
}
if st == nil {
t.Fatal("Load returned a nil store; handlers need a usable store")
}
if st.Ready() {
t.Fatal("Ready()=true without data")
}
if st.Len() != 0 {
t.Fatalf("Len()=%d, want 0", st.Len())
}
}
+433
View File
@@ -0,0 +1,433 @@
#!/usr/bin/env bash
# Repeatable end-to-end integration + fallback verification for the Go API.
#
# It starts the real FastAPI ML service on :18000 and the Go API on :18080,
# trains the small (and optionally forecast) model through POST /api/train,
# checks SSE scenario propagation / heartbeat / reconnect, kills FastAPI to
# prove the runtime snapshot + last-good forecast fallback, restarts it, and
# verifies recovery. With --load it also runs the 120-client 5-minute SSE test.
#
# scripts/integration.sh # fast checks (~1-2 min)
# scripts/integration.sh --load # plus the 5-minute load test
#
# Requirements: go, curl, jq, uv, and the ML checkout (read-only).
# The script owns both ports; it refuses to run if something is already there.
set -euo pipefail
ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)
ML_DIR=${ML_DIR:-/home/noel/Documents/Hackathon_2026/ML-py.git/main}
RAW_CSV=${RAW_CSV:-/home/noel/Documents/Hackathon_2026/data/raw/schwarz-pax.csv}
GO_PORT=${GO_PORT:-18080}
ML_PORT=${ML_PORT:-18000}
API_URL="http://127.0.0.1:${GO_PORT}"
ML_URL="http://127.0.0.1:${ML_PORT}"
CONTROL_USER=${CONTROL_USER:-control}
# Local test values only; override both in any shared environment. The API
# itself has no credential defaults and fails closed when they are unset.
CONTROL_PASS=${CONTROL_PASS:-pulsecast-dev-pass}
CONTROL_TOKEN=${CONTROL_TOKEN:-pulsecast-control-token}
BOOTSTRAP_REFRESH=${BOOTSTRAP_REFRESH:-10s}
ML_TIMEOUT=${ML_TIMEOUT:-1500ms}
WORKDIR=${WORKDIR:-/tmp/pulsecast-integration}
RESULTS_DIR=${RESULTS_DIR:-$WORKDIR/results}
SMALL_CSV=${SMALL_CSV:-$WORKDIR/small.csv}
SMALL_FROM=${SMALL_FROM:-2026-09-01}
SMALL_TO=${SMALL_TO:-2026-10-16}
RUN_LOAD=0
RUN_FORECAST=${RUN_FORECAST:-1}
for arg in "$@"; do
case "$arg" in
--load) RUN_LOAD=1 ;;
--no-forecast) RUN_FORECAST=0 ;;
-h|--help) sed -n '2,14p' "$0"; exit 0 ;;
*) echo "unknown argument: $arg" >&2; exit 2 ;;
esac
done
mkdir -p "$WORKDIR" "$RESULTS_DIR"
API_LOG="$WORKDIR/api.log"
ML_LOG="$WORKDIR/ml.log"
API_PID=""
ML_PID=""
ms() { date +%s%3N; }
log() { printf '\n==> %s\n' "$*"; }
ok() { printf ' ok: %s\n' "$*"; }
fail() {
printf '\nFAIL: %s\n' "$*" >&2
printf 'api log: %s\nml log: %s\n' "$API_LOG" "$ML_LOG" >&2
exit 1
}
need() {
command -v "$1" >/dev/null 2>&1 || fail "missing command: $1"
}
port_busy() { ss -ltn 2>/dev/null | awk '{print $4}' | grep -q ":$1\$"; }
cleanup() {
local status=$?
if [[ -n "$API_PID" ]]; then
kill -TERM -- "-$API_PID" 2>/dev/null || kill -TERM "$API_PID" 2>/dev/null || true
fi
if [[ -n "$ML_PID" ]]; then
kill -TERM -- "-$ML_PID" 2>/dev/null || kill -TERM "$ML_PID" 2>/dev/null || true
fuser -k -TERM "$ML_PORT/tcp" >/dev/null 2>&1 || true
fi
exit "$status"
}
trap cleanup EXIT INT TERM
wait_http() { # url, deadline seconds
local url=$1 deadline=$2 start now
start=$(ms)
while :; do
if curl -sf -m 1 "$url" >/dev/null 2>&1; then return 0; fi
now=$(ms)
(( now - start < deadline * 1000 )) || return 1
sleep 0.2
done
}
json_get() { # file, jq filter
jq -r "$2" "$1" 2>/dev/null
}
get() { # path, outfile -> status code
curl -sS -m 10 -o "$2" -w '%{http_code}' "$API_URL$1"
}
need go
need curl
need jq
need uv
need awk
[[ -x "$ML_DIR/.venv/bin/python" || -f "$ML_DIR/pyproject.toml" ]] || fail "ML checkout not found at $ML_DIR"
[[ -f "$RAW_CSV" ]] || fail "raw CSV not found at $RAW_CSV"
port_busy "$GO_PORT" && fail "port $GO_PORT is already in use; stop that process first"
port_busy "$ML_PORT" && fail "port $ML_PORT is already in use; stop that process first"
# ------------------------------------------------------------------ small CSV
log "building a small training CSV ($SMALL_FROM .. $SMALL_TO)"
if [[ ! -s "$SMALL_CSV" || "$RAW_CSV" -nt "$SMALL_CSV" ]]; then
head -1 "$RAW_CSV" > "$SMALL_CSV.tmp"
awk -F',' -v from="\"$SMALL_FROM" -v to="\"$SMALL_TO" \
'NR>1 && $6 ~ /densityData/ && $1 >= from && $1 < to' "$RAW_CSV" >> "$SMALL_CSV.tmp"
mv "$SMALL_CSV.tmp" "$SMALL_CSV"
fi
small_rows=$(wc -l < "$SMALL_CSV")
(( small_rows > 1000 )) || fail "small CSV has only $small_rows lines"
ok "$small_rows rows, $(du -h "$SMALL_CSV" | cut -f1)"
# ------------------------------------------------------------------- FastAPI
log "starting FastAPI on :$ML_PORT"
(
cd "$ML_DIR"
env CONTROL_TOKEN="$CONTROL_TOKEN" \
PULSECAST_MODELS="$WORKDIR/models" \
PULSECAST_UPLOADS="$WORKDIR/uploads" \
setsid uv run --frozen uvicorn pulsecast.server:app \
--host 127.0.0.1 --port "$ML_PORT" > "$ML_LOG" 2>&1 &
echo $! > "$WORKDIR/ml.pid"
)
ML_PID=$(cat "$WORKDIR/ml.pid")
wait_http "$ML_URL/readyz" 90 || fail "FastAPI did not become ready; see $ML_LOG"
ok "readyz 200"
# ------------------------------------------------------------------- Go API
log "building and starting the Go API on :$GO_PORT"
( cd "$ROOT" && go build -o "$WORKDIR/api" ./cmd/api )
(
env PORT="$GO_PORT" FASTAPI_URL="$ML_URL" \
CONTROL_USER="$CONTROL_USER" CONTROL_PASS="$CONTROL_PASS" CONTROL_TOKEN="$CONTROL_TOKEN" \
BOOTSTRAP_REFRESH="$BOOTSTRAP_REFRESH" ML_TIMEOUT="$ML_TIMEOUT" \
setsid "$WORKDIR/api" > "$API_LOG" 2>&1 &
echo $! > "$WORKDIR/api.pid"
)
API_PID=$(cat "$WORKDIR/api.pid")
wait_http "$API_URL/api/readyz" 30 || fail "Go API did not become ready; see $API_LOG"
ok "readyz 200 after first /bootstrap snapshot"
initial_meta="$RESULTS_DIR/meta-initial.json"
get "/api/meta" "$initial_meta" >/dev/null
initial_success=$(json_get "$initial_meta" '.cache.last_success_at')
[[ -n "$initial_success" && "$initial_success" != "null" ]] || fail "no cache last_success_at"
ok "data_version=$(json_get "$initial_meta" '.data_version') model_version=$(json_get "$initial_meta" '.model_version')"
# ---------------------------------------------------------------- train small
poll_job() { # job_id, timeout seconds
local job=$1 timeout=$2 start now status
start=$(ms)
while :; do
curl -sS -m 10 -u "$CONTROL_USER:$CONTROL_PASS" \
"$API_URL/api/train/$job" > "$WORKDIR/job.json"
status=$(json_get "$WORKDIR/job.json" '.status')
if [[ "$status" == "succeeded" || "$status" == "failed" ]]; then return 0; fi
now=$(ms)
(( now - start < timeout * 1000 )) || return 1
sleep 0.05
done
}
run_train() { # model, timeout
local model=$1 timeout=$2 t0 submit_http status
t0=$(ms)
submit_http=$(curl -sS -m 60 -o "$WORKDIR/train-submit.json" -w '%{http_code}' \
-u "$CONTROL_USER:$CONTROL_PASS" \
-F "file=@$SMALL_CSV;type=text/csv" -F "model=$model" \
"$API_URL/api/train")
JOB_SUBMIT_MS=$(( $(ms) - t0 ))
JOB_SUBMIT_HTTP=$submit_http
[[ "$JOB_SUBMIT_HTTP" == "202" ]] || fail "POST /api/train ($model) returned $JOB_SUBMIT_HTTP: $(cat "$WORKDIR/train-submit.json")"
JOB_ID=$(json_get "$WORKDIR/train-submit.json" '.job_id')
[[ -n "$JOB_ID" && "$JOB_ID" != "null" ]] || fail "train response has no job_id"
poll_job "$JOB_ID" "$timeout" || fail "training job $JOB_ID ($model) did not finish within ${timeout}s"
JOB_TERMINAL_MS=$(( $(ms) - t0 ))
JOB_STATUS=$(json_get "$WORKDIR/job.json" '.status')
[[ "$JOB_STATUS" == "succeeded" ]] || fail "training job $JOB_ID ($model) status=$JOB_STATUS: $(cat "$WORKDIR/job.json")"
JOB_VERSION=$(json_get "$WORKDIR/job.json" '.model_version')
[[ -n "$JOB_VERSION" && "$JOB_VERSION" != "null" ]] || fail "job $JOB_ID has no model_version"
}
log "training model=small through POST /api/train (multipart CSV)"
run_train small 60
jq -e '.metrics.accuracy > 0.5 and .metrics.n_train > 0 and .metrics.n_test > 0 and .error == null' \
"$WORKDIR/job.json" >/dev/null || fail "small job metrics look wrong: $(cat "$WORKDIR/job.json")"
jq -n --argjson submit_http "$JOB_SUBMIT_HTTP" --argjson submit_ms "$JOB_SUBMIT_MS" \
--argjson terminal_ms "$JOB_TERMINAL_MS" --arg model small --slurpfile job "$WORKDIR/job.json" \
'{model:$model, submit_http:$submit_http, submit_ms:$submit_ms, submit_to_terminal_ms:$terminal_ms, job:$job[0]}' \
> "$RESULTS_DIR/train-small.json"
ok "job $JOB_ID succeeded in ${JOB_TERMINAL_MS}ms (submit ${JOB_SUBMIT_MS}ms); accuracy=$(json_get "$WORKDIR/job.json" '.metrics.accuracy') roc_auc=$(json_get "$WORKDIR/job.json" '.metrics.roc_auc') version=$JOB_VERSION"
forecast_before="$RESULTS_DIR/forecast-after-small.json"
code=$(get "/api/forecast?sensor=hn-stadtgalerie-01&horizon=7d&scenario=base" "$forecast_before")
[[ "$code" == "200" ]] || fail "forecast after small training returned $code"
[[ "$(json_get "$forecast_before" '.cached')" == "false" ]] || fail "forecast after training is unexpectedly cached"
[[ "$(json_get "$forecast_before" '.points | length')" == "168" ]] || fail "forecast after training has wrong point count"
ok "live forecast still works: 168 points, cached=false"
if [[ "$RUN_FORECAST" == "1" ]]; then
log "training model=forecast through POST /api/train (multipart CSV)"
run_train forecast 300
jq -n --argjson submit_http "$JOB_SUBMIT_HTTP" --argjson submit_ms "$JOB_SUBMIT_MS" \
--argjson terminal_ms "$JOB_TERMINAL_MS" --arg model forecast --slurpfile job "$WORKDIR/job.json" \
'{model:$model, submit_http:$submit_http, submit_ms:$submit_ms, submit_to_terminal_ms:$terminal_ms, job:$job[0]}' \
> "$RESULTS_DIR/train-forecast.json"
ok "job $JOB_ID succeeded in ${JOB_TERMINAL_MS}ms; mae=$(json_get "$WORKDIR/job.json" '.metrics.mae') wape=$(json_get "$WORKDIR/job.json" '.metrics.wape') version=$JOB_VERSION"
# The snapshot refresh carries the hot-swapped model version to /api/meta.
deadline=$(($(ms) + 30000))
while :; do
get "/api/meta" "$WORKDIR/meta.json" >/dev/null
[[ "$(json_get "$WORKDIR/meta.json" '.model_version')" == "$JOB_VERSION" ]] && break
(( $(ms) < deadline )) || fail "Go cache did not pick up the retrained model_version"
sleep 0.5
done
ok "Go /api/meta model_version refreshed to $JOB_VERSION"
fi
# ------------------------------------------------------------------ staffing
log "timing staffing plan generation (contract: <2s)"
staffing_url="$API_URL/api/staffing?sensor=hn-stadtgalerie-01&scenario=base&standard=standard"
staffing_cold=$(curl -sS -m 10 -o "$WORKDIR/staffing-live.json" -w '%{time_total}' "$staffing_url")
staffing_warm=$(curl -sS -m 10 -o "$WORKDIR/staffing-warm.json" -w '%{time_total}' "$staffing_url")
[[ "$(json_get "$WORKDIR/staffing-live.json" '.hours | length')" == "70" ]] || fail "staffing plan does not cover 7 days x 10 store hours"
[[ "$(json_get "$WORKDIR/staffing-live.json" '.cached')" == "false" ]] || fail "live staffing plan is marked cached"
staffing_cold_ms=$(awk -v s="$staffing_cold" 'BEGIN { printf "%d", s*1000 }')
staffing_warm_ms=$(awk -v s="$staffing_warm" 'BEGIN { printf "%d", s*1000 }')
(( staffing_cold_ms < 2000 )) || fail "staffing cold plan took ${staffing_cold_ms}ms (>=2s)"
ok "staffing: cold ${staffing_cold_ms}ms, cached ${staffing_warm_ms}ms, 70 hours, coverage=$(json_get "$WORKDIR/staffing-live.json" '.summary.coverage_pct')%"
jq -n --argjson cold_ms "$staffing_cold_ms" --argjson warm_ms "$staffing_warm_ms" \
--slurpfile plan "$WORKDIR/staffing-live.json" \
'{cold_ms:$cold_ms, warm_ms:$warm_ms, hours:($plan[0].hours|length), shifts:($plan[0].shifts|length), coverage_pct:$plan[0].summary.coverage_pct, cost_eur:$plan[0].summary.cost_eur, cached:$plan[0].cached}' \
> "$RESULTS_DIR/staffing.json"
# -------------------------------------------------------------- SSE scenario
log "checking SSE scenario propagation, heartbeat and reconnect"
sse_log="$RESULTS_DIR/sse-stream.log"
: > "$sse_log"
curl -sN "$API_URL/api/stream" > "$sse_log" &
SSE_PID=$!
sleep 0.5
grep -q '^retry: 3000$' "$sse_log" || fail "stream does not send retry: 3000"
grep -q '^event: scenario$' "$sse_log" || fail "stream does not send the current scenario on connect"
ok "connect: retry: 3000 + initial scenario event"
t0=$(ms)
curl -sS -u "$CONTROL_USER:$CONTROL_PASS" -H 'Content-Type: application/json' \
-d '{"scenario":"sunny-saturday-market","weather":{"temp_delta_c":4,"precip_mm":0},"event":{"type":"market","uplift":0.2}}' \
"$API_URL/api/scenario" > "$WORKDIR/scenario.json"
posted_version=$(json_get "$WORKDIR/scenario.json" '.version')
[[ "${posted_version:-0}" -gt 1 ]] || fail "scenario POST did not return a new version: $(cat "$WORKDIR/scenario.json")"
propagation_ms=""
for _ in $(seq 1 200); do
if grep -q "^id: ${posted_version}$" "$sse_log"; then propagation_ms=$(( $(ms) - t0 )); break; fi
sleep 0.01
done
[[ -n "$propagation_ms" ]] || fail "scenario event did not reach the SSE client within 2s"
(( propagation_ms < 2000 )) || fail "scenario propagation took ${propagation_ms}ms (>2s)"
ok "scenario v$posted_version reached the client in ${propagation_ms}ms"
pings_before=$(grep -c '^event: ping$' "$sse_log" || true)
ping_t0=$(ms)
for _ in $(seq 1 180); do
if [[ "$(grep -c '^event: ping$' "$sse_log" || true)" -gt "$pings_before" ]]; then break; fi
sleep 0.1
done
[[ "$(grep -c '^event: ping$' "$sse_log" || true)" -gt "$pings_before" ]] || fail "no heartbeat within 18s"
heartbeat_ms=$(( $(ms) - ping_t0 ))
grep -q '^data: {"t":"' "$sse_log" || fail "heartbeat has no RFC3339 timestamp"
ok "heartbeat received ${heartbeat_ms}ms after the scenario check; server interval is 15s"
kill "$SSE_PID" 2>/dev/null || true
wait "$SSE_PID" 2>/dev/null || true
reconnect_t0=$(ms)
: > "$WORKDIR/sse-reconnect.log"
curl -sN "$API_URL/api/stream" > "$WORKDIR/sse-reconnect.log" &
SSE_PID=$!
reconnect_ms=""
for _ in $(seq 1 200); do
if grep -q "^id: ${posted_version}$" "$WORKDIR/sse-reconnect.log"; then reconnect_ms=$(( $(ms) - reconnect_t0 )); break; fi
sleep 0.01
done
kill "$SSE_PID" 2>/dev/null || true
wait "$SSE_PID" 2>/dev/null || true
[[ -n "$reconnect_ms" ]] || fail "reconnect did not receive the current scenario immediately"
ok "reconnect: current scenario v$posted_version delivered immediately"
jq -n --argjson version "$posted_version" --argjson propagation_ms "$propagation_ms" \
--argjson heartbeat_after_ms "$heartbeat_ms" --argjson reconnect_ms "$reconnect_ms" \
'{scenario_version:$version, propagation_ms:$propagation_ms, heartbeat_after_ms:$heartbeat_after_ms, heartbeat_interval_s:15, retry_ms:3000, reconnect_ms:$reconnect_ms, reconnect_current_version:$version}' \
> "$RESULTS_DIR/sse.json"
# ------------------------------------------------------- fallback + recovery
log "warming base forecast and staffing, then stopping FastAPI"
get "/api/forecast?sensor=hn-stadtgalerie-01&horizon=7d&scenario=base" "$WORKDIR/warm-forecast.json" >/dev/null
[[ "$(json_get "$WORKDIR/warm-forecast.json" '.cached')" == "false" ]] || fail "warm forecast is not live"
get "/api/meta" "$WORKDIR/meta-warm.json" >/dev/null
last_success_before=$(json_get "$WORKDIR/meta-warm.json" '.cache.last_success_at')
kill -TERM -- "-$ML_PID" 2>/dev/null || kill -TERM "$ML_PID" 2>/dev/null || true
fuser -k -TERM "$ML_PORT/tcp" >/dev/null 2>&1 || true
for _ in $(seq 1 100); do curl -sf -m 1 "$ML_URL/healthz" >/dev/null 2>&1 || break; sleep 0.1; done
curl -sf -m 1 "$ML_URL/healthz" >/dev/null 2>&1 && fail "FastAPI is still reachable after kill"
ok "FastAPI stopped"
deadline=$(($(ms) + 45000))
while :; do
get "/api/meta" "$WORKDIR/meta-down.json" >/dev/null
failures=$(json_get "$WORKDIR/meta-down.json" '.cache.failures')
if [[ "${failures:-0}" -ge 1 ]]; then break; fi
(( $(ms) < deadline )) || fail "cache did not record a failed refresh within 45s"
sleep 0.25
done
jq -e '.cache.failures >= 1 and .cache.last_error != null and .cache.next_retry_at != null' \
"$WORKDIR/meta-down.json" >/dev/null || fail "cache status missing errors/next_retry_at"
last_success_down=$(json_get "$WORKDIR/meta-down.json" '.cache.last_success_at')
ok "cache failures=$failures last_error=$(json_get "$WORKDIR/meta-down.json" '.cache.last_error') next_retry_at=$(json_get "$WORKDIR/meta-down.json" '.cache.next_retry_at')"
code=$(curl -sS -o "$WORKDIR/readyz-down.json" -w '%{http_code}' "$API_URL/api/readyz")
[[ "$code" == "200" ]] || fail "readyz failed while FastAPI is down (status $code)"
ok "readyz still 200 (last good snapshot)"
code=$(get "/api/sensors" "$WORKDIR/sensors-down.json")
[[ "$code" == "200" ]] || fail "/api/sensors returned $code while FastAPI is down"
[[ "$(json_get "$WORKDIR/sensors-down.json" 'length')" == "8" ]] || fail "/api/sensors did not return 8 sensors"
code=$(get "/api/context" "$WORKDIR/context-down.json")
[[ "$code" == "200" ]] || fail "/api/context returned $code while FastAPI is down"
[[ "$(json_get "$WORKDIR/context-down.json" '.cached')" == "true" ]] || fail "/api/context is not marked cached"
code=$(get "/api/series?sensor=hn-stadtgalerie-01&step=1h" "$WORKDIR/series-down.json")
[[ "$code" == "200" ]] || fail "/api/series returned $code while FastAPI is down"
ok "sensors/context/series still served from the last good snapshot (cached=true)"
code=$(get "/api/forecast?sensor=hn-stadtgalerie-01&horizon=7d&scenario=base" "$WORKDIR/forecast-down.json")
[[ "$code" == "200" ]] || fail "/api/forecast returned $code while FastAPI is down"
[[ "$(json_get "$WORKDIR/forecast-down.json" '.cached')" == "true" ]] || fail "fallback forecast is not marked cached"
[[ "$(json_get "$WORKDIR/forecast-down.json" '.points | length')" == "168" ]] || fail "fallback forecast has wrong point count"
ok "forecast serves the last good 168 points with cached=true"
code=$(get "/api/staffing?sensor=hn-stadtgalerie-01&scenario=base&standard=standard" "$WORKDIR/staffing-down.json")
[[ "$code" == "200" ]] || fail "/api/staffing returned $code while FastAPI is down"
[[ "$(json_get "$WORKDIR/staffing-down.json" '.cached')" == "true" ]] || fail "fallback staffing is not marked cached"
ok "staffing builds from the last good forecast with cached=true"
code=$(get "/api/forecast?sensor=density-01&horizon=15m&scenario=rainy-saturday" "$WORKDIR/forecast-cold.json")
[[ "$code" == "503" ]] || fail "uncached forecast returned $code, want 503"
ok "uncached forecast correctly returns 503 (no invented data)"
fallback_failures=$failures
fallback_error=$(json_get "$WORKDIR/meta-down.json" '.cache.last_error')
fallback_next=$(json_get "$WORKDIR/meta-down.json" '.cache.next_retry_at')
log "restarting FastAPI and waiting for recovery"
(
cd "$ML_DIR"
env CONTROL_TOKEN="$CONTROL_TOKEN" \
PULSECAST_MODELS="$WORKDIR/models" \
PULSECAST_UPLOADS="$WORKDIR/uploads" \
setsid uv run --frozen uvicorn pulsecast.server:app \
--host 127.0.0.1 --port "$ML_PORT" > "$ML_LOG" 2>&1 &
echo $! > "$WORKDIR/ml.pid"
)
ML_PID=$(cat "$WORKDIR/ml.pid")
wait_http "$ML_URL/readyz" 90 || fail "FastAPI did not restart; see $ML_LOG"
recovery_start=$(ms)
deadline=$(($(ms) + 120000))
while :; do
get "/api/meta" "$WORKDIR/meta-recovered.json" >/dev/null
failures=$(json_get "$WORKDIR/meta-recovered.json" '.cache.failures')
if [[ "${failures:-1}" == "0" ]]; then break; fi
(( $(ms) < deadline )) || fail "cache did not recover within 120s"
sleep 0.25
done
recovery_ms=$(( $(ms) - recovery_start ))
recovered_success=$(json_get "$WORKDIR/meta-recovered.json" '.cache.last_success_at')
[[ "$recovered_success" != "$last_success_down" ]] || fail "last_success_at did not advance after recovery"
code=$(get "/api/forecast?sensor=hn-stadtgalerie-01&horizon=7d&scenario=base" "$WORKDIR/forecast-recovered.json")
[[ "$code" == "200" ]] || fail "forecast after recovery returned $code"
[[ "$(json_get "$WORKDIR/forecast-recovered.json" '.cached')" == "false" ]] || fail "forecast after recovery is still cached"
ok "recovered after ${recovery_ms}ms: failures=0, last_success_at=$recovered_success"
jq -n \
--argjson fallback_failures "$fallback_failures" \
--arg fallback_error "$fallback_error" \
--arg fallback_next_retry "$fallback_next" \
--argjson fallback_forecast_cached "$(json_get "$WORKDIR/forecast-down.json" '.cached')" \
--argjson fallback_staffing_cached "$(json_get "$WORKDIR/staffing-down.json" '.cached')" \
--argjson recovery_ms "$recovery_ms" \
--arg last_success_before "$last_success_before" \
--arg last_success_down "$last_success_down" \
--arg last_success_after "$recovered_success" \
--argjson recovered_cached "$(json_get "$WORKDIR/forecast-recovered.json" '.cached')" \
'{fastapi_down:{cache_failures:$fallback_failures, last_error:$fallback_error, next_retry_at:$fallback_next_retry, last_success_at:$last_success_down, forecast_cached:$fallback_forecast_cached, staffing_cached:$fallback_staffing_cached, readyz:200, sensors:8}, recovery:{after_ms:$recovery_ms, last_success_before:$last_success_before, last_success_after:$last_success_after, forecast_cached:$recovered_cached, failures:0}}' \
> "$RESULTS_DIR/fallback.json"
# ------------------------------------------------------------------ load test
if [[ "$RUN_LOAD" == "1" ]]; then
log "running the 120-client SSE load test (5 minutes)"
( cd "$ROOT" && CONTROL_USER="$CONTROL_USER" CONTROL_PASS="$CONTROL_PASS" \
go run ./scripts/sse-load -base "$API_URL" -clients 120 -duration 5m \
-changes 30s -json "$RESULTS_DIR/load.json" ) | tee "$RESULTS_DIR/load.txt"
jq -e '.passed == true' "$RESULTS_DIR/load.json" >/dev/null || fail "SSE load test failed; see $RESULTS_DIR/load.json"
ok "load test passed"
fi
log "all integration checks passed"
cat > "$RESULTS_DIR/summary.txt" <<EOF
PulseCast Go integration $(date -u +%Y-%m-%dT%H:%M:%SZ)
FastAPI $ML_URL, Go $API_URL
small CSV: $small_rows rows ($SMALL_FROM .. $SMALL_TO)
small job: $(jq -r '"status=\(.job.status) version=\(.job.model_version) accuracy=\(.job.metrics.accuracy) roc_auc=\(.job.metrics.roc_auc) submit_to_terminal_ms=\(.submit_to_terminal_ms)"' "$RESULTS_DIR/train-small.json")
staffing: $(jq -r '"cold_ms=\(.cold_ms) warm_ms=\(.warm_ms) hours=\(.hours) coverage_pct=\(.coverage_pct) cost_eur=\(.cost_eur)"' "$RESULTS_DIR/staffing.json")
SSE: $(jq -c '.' "$RESULTS_DIR/sse.json")
fallback: $(jq -c '.' "$RESULTS_DIR/fallback.json")
EOF
if [[ "$RUN_FORECAST" == "1" ]]; then
echo "forecast job: $(jq -r '"status=\(.job.status) version=\(.job.model_version) mae=\(.job.metrics.mae) wape=\(.job.metrics.wape) submit_to_terminal_ms=\(.submit_to_terminal_ms)"' "$RESULTS_DIR/train-forecast.json")" >> "$RESULTS_DIR/summary.txt"
fi
cat "$RESULTS_DIR/summary.txt"
+694
View File
@@ -0,0 +1,694 @@
// Command sse-load is the acceptance load test for GET /api/stream.
//
// It opens N concurrent SSE clients (120 by default) for a fixed duration,
// posts scenario changes on an interval, and measures the end-to-end latency
// from the POST to each client receiving the matching scenario event. It also
// verifies heartbeat cadence, counts connection drops/reconnects, and writes a
// machine-readable JSON summary.
//
// go run ./scripts/sse-load -base http://127.0.0.1:18080 -duration 5m
//
// Credentials come from -user/-pass or CONTROL_USER/CONTROL_PASS. The tool
// exits non-zero when a client drops, an event is not delivered, or the p95
// scenario latency exceeds 2 s: the contract's acceptance threshold.
package main
import (
"bufio"
"bytes"
"context"
"encoding/json"
"errors"
"flag"
"fmt"
"io"
"math"
"net/http"
"os"
"os/signal"
"sort"
"strings"
"sync"
"syscall"
"time"
)
const scenarioThreshold = 2 * time.Second
// scenarioBodies rotate so consecutive changes differ in payload as well as
// version.
var scenarioBodies = []string{
`{"scenario":"sunny-saturday-market","weather":{"temp_delta_c":4,"precip_mm":0},"event":{"type":"market","uplift":0.2}}`,
`{"scenario":"rainy-saturday","weather":{"temp_delta_c":-3,"precip_mm":4},"event":{"type":"rain","uplift":-0.15}}`,
`{"scenario":"weekday-event","weather":{"temp_delta_c":0,"precip_mm":0},"event":{"type":"event","uplift":0.1}}`,
`{"scenario":"base","weather":{"temp_delta_c":0,"precip_mm":0},"event":{"type":"","uplift":0}}`,
}
type scenarioEvent struct {
Version int `json:"version"`
At string `json:"at"`
}
type client struct {
id int
mu sync.Mutex
connected bool
receipts map[int]time.Time // scenario version -> first receipt
pings []time.Time
readyOnce sync.Once
}
func (c *client) setConnected(connected bool) {
c.mu.Lock()
c.connected = connected
c.mu.Unlock()
}
func (c *client) isConnected() bool {
c.mu.Lock()
defer c.mu.Unlock()
return c.connected
}
func (c *client) recordScenario(version int, at time.Time) {
c.mu.Lock()
if c.receipts == nil {
c.receipts = make(map[int]time.Time)
}
if _, exists := c.receipts[version]; !exists {
c.receipts[version] = at
}
c.mu.Unlock()
}
func (c *client) recordPing(at time.Time) {
c.mu.Lock()
c.pings = append(c.pings, at)
c.mu.Unlock()
}
func (c *client) receipt(version int) (time.Time, bool) {
c.mu.Lock()
defer c.mu.Unlock()
at, ok := c.receipts[version]
return at, ok
}
func (c *client) pingCount() int {
c.mu.Lock()
defer c.mu.Unlock()
return len(c.pings)
}
func (c *client) signalReady(ready chan<- struct{}) {
c.readyOnce.Do(func() {
select {
case ready <- struct{}{}:
default:
}
})
}
type changeResult struct {
Version int `json:"version"`
PostAt time.Time `json:"post_at"`
Connected int `json:"connected_at_post"`
Received int `json:"received"`
Missing int `json:"missing"`
LatenciesMS []float64 `json:"-"`
P50MS float64 `json:"p50_ms"`
P95MS float64 `json:"p95_ms"`
P99MS float64 `json:"p99_ms"`
MaxMS float64 `json:"max_ms"`
}
type summary struct {
StartedAt time.Time `json:"started_at"`
DurationS float64 `json:"duration_s"`
BaseURL string `json:"base_url"`
Clients int `json:"clients_configured"`
PeakConnected int `json:"clients_peak_connected"`
ConnectedAfter int `json:"clients_connected_after_warmup"`
Drops int `json:"connection_drops"`
PostErrors int `json:"scenario_post_errors"`
ScenarioChanges int `json:"scenario_changes"`
Received int `json:"scenario_events_received"`
Missing int `json:"scenario_events_missing"`
LatencyCount int `json:"scenario_latency_samples"`
LatencyP50MS float64 `json:"scenario_p50_ms"`
LatencyP95MS float64 `json:"scenario_p95_ms"`
LatencyP99MS float64 `json:"scenario_p99_ms"`
LatencyMaxMS float64 `json:"scenario_max_ms"`
Changes []changeResult `json:"changes"`
Ping pingSummary `json:"heartbeat"`
Errors []string `json:"errors"`
Passed bool `json:"passed"`
FailureReasons []string `json:"failure_reasons,omitempty"`
}
type pingSummary struct {
Unique int `json:"unique_server_timestamps"`
MinIntervalMS float64 `json:"min_interval_ms"`
MedianIntervalMS float64 `json:"median_interval_ms"`
P95IntervalMS float64 `json:"p95_interval_ms"`
MaxIntervalMS float64 `json:"max_interval_ms"`
PerClientMin int `json:"per_client_min"`
PerClientMedian int `json:"per_client_median"`
PerClientMax int `json:"per_client_max"`
}
type recorder struct {
mu sync.Mutex
clients []*client
connected int
peak int
drops int
postErrs int
errors []string
pingTimes map[time.Time]struct{}
}
func (r *recorder) connectedNow() int {
r.mu.Lock()
defer r.mu.Unlock()
return r.connected
}
func (r *recorder) addConnected(delta int) {
r.mu.Lock()
r.connected += delta
if r.connected > r.peak {
r.peak = r.connected
}
r.mu.Unlock()
}
func (r *recorder) addError(message string, drop bool) {
r.mu.Lock()
if drop {
r.drops++
} else {
r.postErrs++
}
if len(r.errors) < 40 {
r.errors = append(r.errors, message)
}
r.mu.Unlock()
}
func (r *recorder) addPing(t time.Time) {
r.mu.Lock()
if r.pingTimes == nil {
r.pingTimes = make(map[time.Time]struct{})
}
r.pingTimes[t] = struct{}{}
r.mu.Unlock()
}
func (r *recorder) snapshot() (peak, drops, postErrs int, errs []string, pings []time.Time) {
r.mu.Lock()
defer r.mu.Unlock()
peak, drops, postErrs = r.peak, r.drops, r.postErrs
errs = append([]string(nil), r.errors...)
pings = make([]time.Time, 0, len(r.pingTimes))
for t := range r.pingTimes {
pings = append(pings, t)
}
sort.Slice(pings, func(i, j int) bool { return pings[i].Before(pings[j]) })
return
}
func main() {
base := flag.String("base", "http://127.0.0.1:18080", "Go API base URL")
clientsN := flag.Int("clients", 120, "concurrent SSE clients")
duration := flag.Duration("duration", 5*time.Minute, "load-test duration")
changeEvery := flag.Duration("changes", 30*time.Second, "interval between scenario POSTs; 0 disables")
warmup := flag.Duration("warmup", 10*time.Second, "max time to wait for all clients to connect")
user := flag.String("user", os.Getenv("CONTROL_USER"), "basic-auth user for POST /api/scenario")
pass := flag.String("pass", os.Getenv("CONTROL_PASS"), "basic-auth password for POST /api/scenario")
jsonOut := flag.String("json", "", "write the JSON summary to this path")
flag.Parse()
if *clientsN < 1 {
fatalf("clients must be >= 1")
}
if *duration <= 0 {
fatalf("duration must be > 0")
}
if *changeEvery > 0 && (*user == "" || *pass == "") {
fatalf("POST /api/scenario needs -user/-pass or CONTROL_USER/CONTROL_PASS")
}
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
ctx, cancel := context.WithTimeout(ctx, *duration)
defer cancel()
rec := &recorder{clients: make([]*client, *clientsN), pingTimes: make(map[time.Time]struct{})}
ready := make(chan struct{}, *clientsN)
started := time.Now()
var wg sync.WaitGroup
for i := 0; i < *clientsN; i++ {
c := &client{id: i}
rec.clients[i] = c
wg.Add(1)
go func() {
defer wg.Done()
runClient(ctx, *base, c, rec, ready)
}()
}
// Wait until every client has received the initial scenario event.
connectedAfter := waitReady(ctx, ready, *clientsN, *warmup)
fmt.Printf("clients connected: %d/%d after warmup\n", connectedAfter, *clientsN)
if connectedAfter < *clientsN {
fmt.Printf("warning: only %d/%d clients connected before warmup deadline\n", connectedAfter, *clientsN)
}
changes := []changeResult{}
if *changeEvery > 0 {
changes = runChanges(ctx, *base, *user, *pass, *changeEvery, rec)
}
<-ctx.Done()
cancel()
done := make(chan struct{})
go func() { wg.Wait(); close(done) }()
select {
case <-done:
case <-time.After(5 * time.Second):
fmt.Println("warning: clients did not stop within 5s")
}
elapsed := time.Since(started)
sum := buildSummary(rec, changes, *base, *clientsN, connectedAfter, started, elapsed)
printSummary(sum)
if *jsonOut != "" {
if err := writeJSON(*jsonOut, sum); err != nil {
fatalf("write %s: %v", *jsonOut, err)
}
fmt.Printf("json written to %s\n", *jsonOut)
}
if !sum.Passed {
os.Exit(1)
}
}
func waitReady(ctx context.Context, ready <-chan struct{}, want int, timeout time.Duration) int {
timer := time.NewTimer(timeout)
defer timer.Stop()
count := 0
for count < want {
select {
case <-ready:
count++
case <-timer.C:
return count
case <-ctx.Done():
return count
}
}
return count
}
func runClient(ctx context.Context, base string, c *client, rec *recorder, ready chan<- struct{}) {
transport := &http.Transport{
Proxy: nil,
DisableCompression: true,
MaxIdleConnsPerHost: 1,
IdleConnTimeout: 0,
}
httpClient := &http.Client{Transport: transport}
defer transport.CloseIdleConnections()
streamURL := strings.TrimRight(base, "/") + "/api/stream"
failures := 0
for ctx.Err() == nil {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, streamURL, nil)
if err != nil {
rec.addError(fmt.Sprintf("client %d: build request: %v", c.id, err), true)
c.signalReady(ready)
return
}
req.Header.Set("Accept", "text/event-stream")
resp, err := httpClient.Do(req)
if err != nil {
if ctx.Err() != nil {
return
}
failures++
rec.addError(fmt.Sprintf("client %d: connect: %v", c.id, err), true)
c.signalReady(ready)
sleepRetry(ctx)
continue
}
if resp.StatusCode != http.StatusOK {
snippet, _ := io.ReadAll(io.LimitReader(resp.Body, 128))
_ = resp.Body.Close()
failures++
rec.addError(fmt.Sprintf("client %d: status %d: %s", c.id, resp.StatusCode, strings.TrimSpace(string(snippet))), true)
c.signalReady(ready)
sleepRetry(ctx)
continue
}
c.setConnected(true)
rec.addConnected(1)
readStream(ctx, resp.Body, c, rec, ready)
_ = resp.Body.Close()
c.setConnected(false)
rec.addConnected(-1)
if ctx.Err() != nil {
return
}
failures++
rec.addError(fmt.Sprintf("client %d: stream ended unexpectedly", c.id), true)
c.signalReady(ready)
sleepRetry(ctx)
}
}
// readStream parses one SSE response. It returns when the body ends or the
// context is cancelled.
func readStream(ctx context.Context, body io.Reader, c *client, rec *recorder, ready chan<- struct{}) {
scanner := bufio.NewScanner(body)
scanner.Buffer(make([]byte, 0, 64*1024), 1<<20)
eventName := ""
eventID := ""
var data []string
for ctx.Err() == nil && scanner.Scan() {
line := scanner.Text()
if line == "" {
now := time.Now()
switch eventName {
case "scenario":
version := 0
var event scenarioEvent
if err := json.Unmarshal([]byte(strings.Join(data, "\n")), &event); err == nil {
version = event.Version
}
if version == 0 && eventID != "" {
_, _ = fmt.Sscanf(eventID, "%d", &version)
}
if version > 0 {
c.recordScenario(version, now)
c.signalReady(ready)
}
case "ping":
c.recordPing(now)
var payload struct {
T string `json:"t"`
}
if err := json.Unmarshal([]byte(strings.Join(data, "\n")), &payload); err == nil {
if t, err := time.Parse(time.RFC3339, payload.T); err == nil {
rec.addPing(t.UTC())
}
}
}
eventName, eventID, data = "", "", nil
continue
}
switch {
case strings.HasPrefix(line, "event:"):
eventName = strings.TrimSpace(strings.TrimPrefix(line, "event:"))
case strings.HasPrefix(line, "id:"):
eventID = strings.TrimSpace(strings.TrimPrefix(line, "id:"))
case strings.HasPrefix(line, "data:"):
data = append(data, strings.TrimPrefix(line, "data:"))
}
}
}
func runChanges(ctx context.Context, base, user, pass string, every time.Duration, rec *recorder) []changeResult {
client := &http.Client{Timeout: 10 * time.Second}
scenarioURL := strings.TrimRight(base, "/") + "/api/scenario"
var out []changeResult
ticker := time.NewTicker(every)
defer ticker.Stop()
for i := 0; ; i++ {
select {
case <-ctx.Done():
return out
case <-ticker.C:
}
body := scenarioBodies[i%len(scenarioBodies)]
postAt := time.Now()
version, err := postScenario(ctx, client, scenarioURL, user, pass, body)
if err != nil {
if ctx.Err() != nil {
return out
}
rec.addError("scenario POST: "+err.Error(), false)
continue
}
result := changeResult{
Version: version,
PostAt: postAt.UTC(),
Connected: rec.connectedNow(),
}
// Wait up to 2s for every currently connected client to receive it.
deadline := time.Now().Add(2 * time.Second)
for {
result.Received = 0
for _, c := range rec.clients {
if !c.isConnected() {
continue
}
if _, ok := c.receipt(version); ok {
result.Received++
}
}
result.Missing = result.Connected - result.Received
if result.Missing <= 0 || time.Now().After(deadline) {
break
}
time.Sleep(10 * time.Millisecond)
}
for _, c := range rec.clients {
if !c.isConnected() {
continue
}
if at, ok := c.receipt(version); ok {
result.LatenciesMS = append(result.LatenciesMS,
float64(at.Sub(postAt).Microseconds())/1000.0)
}
}
result.P50MS, result.P95MS, result.P99MS, result.MaxMS = percentiles(result.LatenciesMS)
out = append(out, result)
fmt.Printf("scenario v%d: %d/%d clients p50=%.1fms p95=%.1fms max=%.1fms\n",
version, result.Received, result.Connected, result.P50MS, result.P95MS, result.MaxMS)
}
}
func postScenario(ctx context.Context, client *http.Client, url, user, pass, body string) (int, error) {
req, err := http.NewRequestWithContext(ctx, http.MethodPost, url, bytes.NewBufferString(body))
if err != nil {
return 0, err
}
req.Header.Set("Content-Type", "application/json")
req.SetBasicAuth(user, pass)
resp, err := client.Do(req)
if err != nil {
return 0, err
}
defer resp.Body.Close()
raw, _ := io.ReadAll(io.LimitReader(resp.Body, 4096))
if resp.StatusCode != http.StatusOK {
return 0, fmt.Errorf("status %d: %s", resp.StatusCode, strings.TrimSpace(string(raw)))
}
var parsed struct {
Version int `json:"version"`
}
if err := json.Unmarshal(raw, &parsed); err != nil {
return 0, fmt.Errorf("decode response: %w", err)
}
if parsed.Version <= 0 {
return 0, errors.New("response has no version")
}
return parsed.Version, nil
}
func buildSummary(rec *recorder, changes []changeResult, base string, clientsN, connectedAfter int, started time.Time, elapsed time.Duration) summary {
peak, drops, postErrs, errs, pingTimes := rec.snapshot()
sum := summary{
StartedAt: started.UTC(),
DurationS: math.Round(elapsed.Seconds()*1000) / 1000,
BaseURL: base,
Clients: clientsN,
PeakConnected: peak,
ConnectedAfter: connectedAfter,
Drops: drops,
PostErrors: postErrs,
ScenarioChanges: len(changes),
Changes: changes,
Errors: errs,
}
var all []float64
for _, change := range changes {
all = append(all, change.LatenciesMS...)
sum.Received += change.Received
sum.Missing += change.Missing
}
sum.LatencyCount = len(all)
sum.LatencyP50MS, sum.LatencyP95MS, sum.LatencyP99MS, sum.LatencyMaxMS = percentiles(all)
sum.Ping = buildPingSummary(pingTimes, rec.clients)
// Pass criteria from the API contract: 120 clients for 5 minutes, no dropped
// connections, p95 scenario latency below 2 s.
if peak < clientsN {
sum.FailureReasons = append(sum.FailureReasons, fmt.Sprintf("peak connections %d < %d", peak, clientsN))
}
if drops > 0 {
sum.FailureReasons = append(sum.FailureReasons, fmt.Sprintf("%d dropped/reconnected connections", drops))
}
if postErrs > 0 {
sum.FailureReasons = append(sum.FailureReasons, fmt.Sprintf("%d scenario POST errors", postErrs))
}
if sum.Missing > 0 {
sum.FailureReasons = append(sum.FailureReasons, fmt.Sprintf("%d scenario events missing", sum.Missing))
}
if len(all) > 0 && sum.LatencyP95MS > scenarioThreshold.Seconds()*1000 {
sum.FailureReasons = append(sum.FailureReasons, fmt.Sprintf("scenario p95 %.1fms > 2000ms", sum.LatencyP95MS))
}
if sum.Ping.P95IntervalMS > 17_000 {
sum.FailureReasons = append(sum.FailureReasons, fmt.Sprintf("heartbeat p95 interval %.0fms > 17000ms", sum.Ping.P95IntervalMS))
}
sum.Passed = len(sum.FailureReasons) == 0
return sum
}
func buildPingSummary(pingTimes []time.Time, clients []*client) pingSummary {
out := pingSummary{Unique: len(pingTimes)}
if len(pingTimes) > 1 {
intervals := make([]float64, 0, len(pingTimes)-1)
for i := 1; i < len(pingTimes); i++ {
intervals = append(intervals, float64(pingTimes[i].Sub(pingTimes[i-1]).Microseconds())/1000.0)
}
out.MinIntervalMS = minFloat(intervals)
out.MedianIntervalMS = percentile(intervals, 50)
out.P95IntervalMS = percentile(intervals, 95)
out.MaxIntervalMS = maxFloat(intervals)
}
counts := make([]int, 0, len(clients))
for _, c := range clients {
counts = append(counts, c.pingCount())
}
if len(counts) > 0 {
sort.Ints(counts)
out.PerClientMin = counts[0]
out.PerClientMedian = counts[len(counts)/2]
out.PerClientMax = counts[len(counts)-1]
}
return out
}
func percentiles(values []float64) (p50, p95, p99, max float64) {
if len(values) == 0 {
return 0, 0, 0, 0
}
return percentile(values, 50), percentile(values, 95), percentile(values, 99), maxFloat(values)
}
// percentile uses the nearest-rank method on a sorted copy.
func percentile(values []float64, p float64) float64 {
if len(values) == 0 {
return 0
}
sorted := append([]float64(nil), values...)
sort.Float64s(sorted)
rank := int(math.Ceil(p/100*float64(len(sorted)))) - 1
if rank < 0 {
rank = 0
}
if rank >= len(sorted) {
rank = len(sorted) - 1
}
return math.Round(sorted[rank]*10) / 10
}
func minFloat(values []float64) float64 {
m := values[0]
for _, v := range values[1:] {
if v < m {
m = v
}
}
return m
}
func maxFloat(values []float64) float64 {
m := values[0]
for _, v := range values[1:] {
if v > m {
m = v
}
}
return m
}
func sleepRetry(ctx context.Context) {
timer := time.NewTimer(3 * time.Second)
defer timer.Stop()
select {
case <-ctx.Done():
case <-timer.C:
}
}
func printSummary(sum summary) {
fmt.Println()
fmt.Println("=== SSE load test ===")
fmt.Printf("duration: %.1fs\n", sum.DurationS)
fmt.Printf("clients: %d configured, %d peak connected, %d after warmup\n",
sum.Clients, sum.PeakConnected, sum.ConnectedAfter)
fmt.Printf("connections: %d drops/reconnects, %d scenario POST errors\n", sum.Drops, sum.PostErrors)
fmt.Printf("scenario changes: %d (%d events received, %d missing)\n",
sum.ScenarioChanges, sum.Received, sum.Missing)
fmt.Printf("scenario latency: p50=%.1fms p95=%.1fms p99=%.1fms max=%.1fms (n=%d)\n",
sum.LatencyP50MS, sum.LatencyP95MS, sum.LatencyP99MS, sum.LatencyMaxMS, sum.LatencyCount)
fmt.Printf("heartbeat: %d server pings, interval min=%.0fms median=%.0fms p95=%.0fms max=%.0fms\n",
sum.Ping.Unique, sum.Ping.MinIntervalMS, sum.Ping.MedianIntervalMS,
sum.Ping.P95IntervalMS, sum.Ping.MaxIntervalMS)
fmt.Printf("heartbeat/client: min=%d median=%d max=%d\n",
sum.Ping.PerClientMin, sum.Ping.PerClientMedian, sum.Ping.PerClientMax)
if len(sum.Errors) > 0 {
fmt.Println("errors:")
for _, err := range sum.Errors {
fmt.Printf(" - %s\n", err)
}
}
if sum.Passed {
fmt.Println("result: PASS")
} else {
fmt.Println("result: FAIL")
for _, reason := range sum.FailureReasons {
fmt.Printf(" - %s\n", reason)
}
}
}
func writeJSON(path string, sum summary) error {
raw, err := json.MarshalIndent(sum, "", " ")
if err != nil {
return err
}
raw = append(raw, '\n')
return os.WriteFile(path, raw, 0o644)
}
func fatalf(format string, args ...any) {
fmt.Fprintf(os.Stderr, "sse-load: "+format+"\n", args...)
os.Exit(2)
}
+124
View File
@@ -0,0 +1,124 @@
package main
import (
"math"
"testing"
"time"
)
func TestPercentileNearestRank(t *testing.T) {
values := []float64{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}
cases := []struct {
p float64
want float64
}{
{50, 5},
{95, 10},
{99, 10},
{0, 1},
{100, 10},
}
for _, tc := range cases {
if got := percentile(values, tc.p); got != tc.want {
t.Fatalf("percentile(%v, %v)=%v, want %v", values, tc.p, got, tc.want)
}
}
if got := percentile(nil, 95); got != 0 {
t.Fatalf("empty percentile=%v, want 0", got)
}
p50, p95, p99, max := percentiles(values)
if p50 != 5 || p95 != 10 || p99 != 10 || max != 10 {
t.Fatalf("percentiles=(%v,%v,%v,%v)", p50, p95, p99, max)
}
}
func TestPercentileDoesNotMutateInput(t *testing.T) {
values := []float64{3, 1, 2}
_ = percentile(values, 50)
if values[0] != 3 || values[1] != 1 || values[2] != 2 {
t.Fatalf("input was reordered: %v", values)
}
}
func TestBuildPingSummary(t *testing.T) {
start := time.Date(2026, 11, 14, 10, 0, 0, 0, time.UTC)
var pings []time.Time
for i := 0; i < 5; i++ {
pings = append(pings, start.Add(time.Duration(i)*15*time.Second))
}
clients := []*client{
{pings: pings[:3]},
{pings: pings[:4]},
{pings: pings},
}
got := buildPingSummary(pings, clients)
if got.Unique != 5 {
t.Fatalf("unique=%d, want 5", got.Unique)
}
if math.Abs(got.MedianIntervalMS-15000) > 0.1 {
t.Fatalf("median interval=%v, want 15000", got.MedianIntervalMS)
}
if got.MaxIntervalMS != 15000 || got.MinIntervalMS != 15000 {
t.Fatalf("interval range=[%v,%v], want [15000,15000]", got.MinIntervalMS, got.MaxIntervalMS)
}
if got.PerClientMin != 3 || got.PerClientMedian != 4 || got.PerClientMax != 5 {
t.Fatalf("per-client=(%d,%d,%d), want (3,4,5)",
got.PerClientMin, got.PerClientMedian, got.PerClientMax)
}
}
func TestSummaryPassAndFail(t *testing.T) {
started := time.Now()
makeRecorder := func(peak, drops, postErrs int) *recorder {
return &recorder{
peak: peak,
drops: drops,
postErrs: postErrs,
pingTimes: make(map[time.Time]struct{}),
}
}
changes := []changeResult{{
Version: 2,
Connected: 120,
Received: 120,
LatenciesMS: []float64{4, 6, 8},
}}
pass := buildSummary(makeRecorder(120, 0, 0), changes, "http://example", 120, 120, started, 5*time.Minute)
if !pass.Passed {
t.Fatalf("expected pass, reasons=%v", pass.FailureReasons)
}
fail := buildSummary(makeRecorder(119, 1, 1), changes, "http://example", 120, 120, started, 5*time.Minute)
if fail.Passed {
t.Fatal("expected failure for low peak and drops")
}
if len(fail.FailureReasons) < 3 {
t.Fatalf("failure reasons=%v, want at least 3", fail.FailureReasons)
}
}
func TestSlowLatencyFailsThreshold(t *testing.T) {
rec := &recorder{peak: 120, pingTimes: make(map[time.Time]struct{})}
changes := []changeResult{{
Version: 3,
Connected: 120,
Received: 120,
LatenciesMS: []float64{2500},
}}
sum := buildSummary(rec, changes, "http://example", 120, 120, time.Now(), 5*time.Minute)
if sum.Passed {
t.Fatal("expected p95 latency over 2s to fail")
}
}
func TestScenarioReceiptKeepsFirst(t *testing.T) {
c := &client{}
first := time.Unix(100, 0)
c.recordScenario(4, first)
c.recordScenario(4, first.Add(time.Second))
got, ok := c.receipt(4)
if !ok || !got.Equal(first) {
t.Fatalf("receipt=%v ok=%v, want %v", got, ok, first)
}
}
+1
View File
@@ -0,0 +1 @@
{"data_version":"2026-11-07T23:51:00Z","model_version":"histgb-2026-11-14","context_version":"2026-11-14T03:00:00Z","sensors":[{"id":"hn-bahnhof-01","name":"Bahnhof","lat":49.1429,"lon":9.2076,"mean":24.2,"p95":88.2,"battery":71},{"id":"hn-deutschhof-01","name":"Deutschhof","lat":49.1397,"lon":9.2149,"mean":14.0,"p95":49.6,"battery":null},{"id":"hn-kaiserstrasse-01","name":"Kaiserstra\u00dfe","lat":49.1419,"lon":9.2096,"mean":21.1,"p95":74.1,"battery":64},{"id":"hn-marktplatz-01","name":"Marktplatz","lat":49.1426,"lon":9.212,"mean":17.9,"p95":63.8,"battery":83},{"id":"hn-neckar-01","name":"Neckarufer","lat":49.1448,"lon":9.2087,"mean":9.4,"p95":35.4,"battery":92},{"id":"hn-rathaus-01","name":"Rathaus","lat":49.1422,"lon":9.2131,"mean":15.6,"p95":55.9,"battery":58},{"id":"hn-stadtgalerie-01","name":"Stadtgalerie","lat":49.1427,"lon":9.2109,"mean":22.6,"p95":67.2,"battery":87},{"id":"hn-theater-01","name":"Theater","lat":49.1402,"lon":9.2124,"mean":11.7,"p95":41.7,"battery":49}],"series":{"hn-bahnhof-01":[{"t":"2026-10-25T00:00:00Z","v":3.6},{"t":"2026-10-25T00:15:00Z","v":4.1},{"t":"2026-10-25T00:30:00Z","v":4.4},{"t":"2026-10-25T00:45:00Z","v":4.0},{"t":"2026-10-25T01:00:00Z","v":3.7},{"t":"2026-10-25T01:15:00Z","v":3.6},{"t":"2026-10-25T01:30:00Z","v":4.3},{"t":"2026-10-25T01:45:00Z","v":3.8},{"t":"2026-10-25T02:00:00Z","v":4.5},{"t":"2026-10-25T02:15:00Z","v":3.7},{"t":"2026-10-25T02:30:00Z","v":4.4},{"t":"2026-10-25T02:45:00Z","v":4.3},{"t":"2026-10-25T03:00:00Z","v":4.2},{"t":"2026-10-25T03:15:00Z","v":4.1},{"t":"2026-10-25T03:30:00Z","v":3.9},{"t":"2026-10-25T03:45:00Z","v":3.8},{"t":"2026-10-25T04:00:00Z","v":3.8},{"t":"2026-10-25T04:15:00Z","v":3.7},{"t":"2026-10-25T04:30:00Z","v":4.4},{"t":"2026-10-25T04:45:00Z","v":4.1},{"t":"2026-10-25T05:00:00Z","v":12.8},{"t":"2026-10-25T05:15:00Z","v":12.7},{"t":"2026-10-25T05:30:00Z","v":12.9},{"t":"2026-10-25T05:45:00Z","v":11.4},{"t":"2026-10-25T06:00:00Z","v":16.3},{"t":"2026-10-25T06:15:00Z","v":17.5},{"t":"2026-10-25T06:30:00Z","v":17.3},{"t":"2026-10-25T06:45:00Z","v":17.0},{"t":"2026-10-25T07:00:00Z","v":23.5},{"t":"2026-10-25T07:15:00Z","v":23.5},{"t":"2026-10-25T07:30:00Z","v":20.4},{"t":"2026-10-25T07:45:00Z","v":19.5},{"t":"2026-10-25T08:00:00Z","v":26.7},{"t":"2026-10-25T08:15:00Z","v":25.3},{"t":"2026-10-25T08:30:00Z","v":28.2},{"t":"2026-10-25T08:45:00Z","v":23.6},{"t":"2026-10-25T09:00:00Z","v":30.1},{"t":"2026-10-25T09:15:00Z","v":31.2},{"t":"2026-10-25T09:30:00Z","v":30.4},{"t":"2026-10-25T09:45:00Z","v":33.7},{"t":"2026-10-25T10:00:00Z","v":37.2},{"t":"2026-10-25T10:15:00Z","v":32.1},{"t":"2026-10-25T10:30:00Z","v":36.9},{"t":"2026-10-25T10:45:00Z","v":30.6},{"t":"2026-10-25T11:00:00Z","v":35.9},{"t":"2026-10-25T11:15:00Z","v":31.7},{"t":"2026-10-25T11:30:00Z","v":30.7},{"t":"2026-10-25T11:45:00Z","v":35.4},{"t":"2026-10-25T12:00:00Z","v":36.0},{"t":"2026-10-25T12:15:00Z","v":31.3},{"t":"2026-10-25T12:30:00Z","v":30.0},{"t":"2026-10-25T12:45:00Z","v":32.9},{"t":"2026-10-25T13:00:00Z","v":27.5},{"t":"2026-10-25T13:15:00Z","v":26.5},{"t":"2026-10-25T13:30:00Z","v":28.1},{"t":"2026-10-25T13:45:00Z","v":29.8},{"t":"2026-10-25T14:00:00Z","v":26.4},{"t":"2026-10-25T14:15:00Z","v":26.5},{"t":"2026-10-25T14:30:00Z","v":29.2},{"t":"2026-10-25T14:45:00Z","v":29.3},{"t":"2026-10-25T15:00:00Z","v":30.5},{"t":"2026-10-25T15:15:00Z","v":30.7},{"t":"2026-10-25T15:30:00Z","v":27.4},{"t":"2026-10-25T15:45:00Z","v":25.2},{"t":"2026-10-25T16:00:00Z","v":31.8},{"t":"2026-10-25T16:15:00Z","v":28.9},{"t":"2026-10-25T16:30:00Z","v":33.3},{"t":"2026-10-25T16:45:00Z","v":32.8},{"t":"2026-10-25T17:00:00Z","v":33.7},{"t":"2026-10-25T17:15:00Z","v":31.5},{"t":"2026-10-25T17:30:00Z","v":33.8},{"t":"2026-10-25T17:45:00Z","v":31.9},{"t":"2026-10-25T18:00:00Z","v":29.9},{"t":"2026-10-25T18:15:00Z","v":32.6},{"t":"2026-10-25T18:30:00Z","v":31.8},{"t":"2026-10-25T18:45:00Z","v":32.6},{"t":"2026-10-25T19:00:00Z","v":16.7},{"t":"2026-10-25T19:15:00Z","v":15.7},{"t":"2026-10-25T19:30:00Z","v":16.2},{"t":"2026-10-25T19:45:00Z","v":15.8},{"t":"2026-10-25T20:00:00Z","v":15.8},{"t":"2026-10-25T20:15:00Z","v":14.4},{"t":"2026-10-25T20:30:00Z","v":16.5},{"t":"2026-10-25T20:45:00Z","v":15.7},{"t":"2026-10-25T21:00:00Z","v":16.7},{"t":"2026-10-25T21:15:00Z","v":13.8},{"t":"2026-10-25T21:30:00Z","v":16.7},{"t":"2026-10-25T21:45:00Z","v":14.2},{"t":"2026-10-25T22:00:00Z","v":7.2},{"t":"2026-10-25T22:15:00Z","v":6.5},{"t":"2026-10-25T22:30:00Z","v":7.1},{"t":"2026-10-25T22:45:00Z","v":5.9},{"t":"2026-10-25T23:00:00Z","v":3.6},{"t":"2026-10-25T23:15:00Z","v":4.1},{"t":"2026-10-25T23:30:00Z","v":3.9},{"t":"2026-10-25T23:45:00Z","v":3.7},{"t":"2026-10-26T00:00:00Z","v":3.4},{"t":"2026-10-26T00:15:00Z","v":3.5},{"t":"2026-10-26T00:30:00Z","v":4.0},{"t":"2026-10-26T00:45:00Z","v":3.7},{"t":"2026-10-26T01:00:00Z","v":3.9},{"t":"2026-10-26T01:15:00Z","v":3.7},{"t":"2026-10-26T01:30:00Z","v":4.0},{"t":"2026-10-26T01:45:00Z","v":3.9},{"t":"2026-10-26T02:00:00Z","v":3.5},{"t":"2026-10-26T02:15:00Z"Line truncated