feat: ci docs #3
No files matched your search
@@ -2,3 +2,7 @@
|
|||||||
.gitignore
|
.gitignore
|
||||||
*.md
|
*.md
|
||||||
opencode.json
|
opencode.json
|
||||||
|
docs
|
||||||
|
scripts
|
||||||
|
# Fixtures are for tests only; the runtime image fetches FastAPI /bootstrap.
|
||||||
|
testdata
|
||||||
@@ -10,6 +10,7 @@ permissions:
|
|||||||
jobs:
|
jobs:
|
||||||
check:
|
check:
|
||||||
runs-on: ubuntu-latest
|
runs-on: ubuntu-latest
|
||||||
|
timeout-minutes: 10
|
||||||
steps:
|
steps:
|
||||||
- name: Check out
|
- name: Check out
|
||||||
uses: actions/checkout@v4
|
uses: actions/checkout@v4
|
||||||
@@ -18,6 +19,9 @@ jobs:
|
|||||||
uses: actions/setup-go@v5
|
uses: actions/setup-go@v5
|
||||||
with:
|
with:
|
||||||
go-version-file: go.mod
|
go-version-file: go.mod
|
||||||
|
# The module is stdlib-only, so there is no go.sum for the action's
|
||||||
|
# default cache key. Remove this once the module has dependencies.
|
||||||
|
cache: false
|
||||||
|
|
||||||
- name: Check
|
- name: Check
|
||||||
run: make check
|
run: make check
|
||||||
+3
-4
@@ -17,11 +17,10 @@ FROM gcr.io/distroless/static-debian12:nonroot
|
|||||||
WORKDIR /app
|
WORKDIR /app
|
||||||
|
|
||||||
COPY --from=build /out/api /app/api
|
COPY --from=build /out/api /app/api
|
||||||
# Baked fallback data lives in the image; there is no runtime data fetch.
|
|
||||||
COPY --from=build /src/testdata/data /app/data
|
|
||||||
|
|
||||||
ENV DATA_DIR=/app/data \
|
# No data is baked into the image. The API fetches FastAPI /bootstrap at
|
||||||
PORT=8080
|
# startup, caches the snapshot in memory, and refreshes it in the background.
|
||||||
|
ENV PORT=8080
|
||||||
|
|
||||||
EXPOSE 8080
|
EXPOSE 8080
|
||||||
# Numeric UID/GID: the kubelet cannot verify named users for runAsNonRoot.
|
# Numeric UID/GID: the kubelet cannot verify named users for runAsNonRoot.
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
GO ?= go
|
GO ?= go
|
||||||
|
|
||||||
.PHONY: check vet test build
|
.PHONY: check vet test build integration integration-load load
|
||||||
|
|
||||||
# Local equivalent of the Gitea Actions CI job.
|
# Local equivalent of the Gitea Actions CI job.
|
||||||
check: vet test build
|
check: vet test build
|
||||||
@@ -13,3 +13,16 @@ test:
|
|||||||
|
|
||||||
build:
|
build:
|
||||||
$(GO) build ./...
|
$(GO) build ./...
|
||||||
|
|
||||||
|
# End-to-end: real FastAPI on :18000, Go on :18080, train small+forecast,
|
||||||
|
# SSE scenario/heartbeat/reconnect, FastAPI kill + cache fallback + recovery.
|
||||||
|
integration:
|
||||||
|
scripts/integration.sh
|
||||||
|
|
||||||
|
# The same run plus the 120-client, 5-minute SSE load test.
|
||||||
|
integration-load:
|
||||||
|
scripts/integration.sh --load
|
||||||
|
|
||||||
|
# Load-only against an already-running API (CONTROL_USER/CONTROL_PASS from env).
|
||||||
|
load:
|
||||||
|
$(GO) run ./scripts/sse-load -base http://127.0.0.1:18080 -clients 120 -duration 5m
|
||||||
@@ -1,6 +1,39 @@
|
|||||||
# api-go
|
# PulseCast API
|
||||||
|
|
||||||
Golang API that is the backbone for daily usage, while python works in batches and is heavy in compute while Golang handles concurrent usage cheap
|
Go service for PulseCast. It serves sensor series, live forecasts, staffing
|
||||||
|
plans and scenario state over SSE, and proxies training to FastAPI. It keeps the
|
||||||
|
FastAPI `/bootstrap` snapshot in memory, so reads survive FastAPI outages.
|
||||||
|
|
||||||
|
## Quick start
|
||||||
|
|
||||||
|
```sh
|
||||||
|
go run ./cmd/api
|
||||||
|
curl -s localhost:8080/api/healthz # {"ok":true}
|
||||||
|
curl -s localhost:8080/api/readyz # 503 until the first /bootstrap snapshot
|
||||||
|
```
|
||||||
|
|
||||||
|
The API listens on `:8080` and expects FastAPI at `http://localhost:8000`
|
||||||
|
(`FASTAPI_URL`). It starts without FastAPI; readiness stays `503` until the
|
||||||
|
first snapshot arrives. Control endpoints return `503` until `CONTROL_USER` and
|
||||||
|
`CONTROL_PASS` are set.
|
||||||
|
|
||||||
|
## Endpoints
|
||||||
|
|
||||||
|
| Endpoint | Purpose |
|
||||||
|
|---|---|
|
||||||
|
| `GET /api/healthz` | Liveness. |
|
||||||
|
| `GET /api/readyz` | Data and model readiness; `503` until the first snapshot. |
|
||||||
|
| `GET /api/meta` | Versions, coverage, metrics and cache status. |
|
||||||
|
| `GET /api/sensors` | Sensors sorted by id. |
|
||||||
|
| `GET /api/series` | Observed series; query `sensor`, `from`, `to`, `step=15m\|1h`. |
|
||||||
|
| `GET /api/context` | Weather and holidays. |
|
||||||
|
| `GET /api/forecast` | Live forecast or last-good fallback; query `sensor`, `horizon`, `scenario`. |
|
||||||
|
| `GET /api/staffing` | Shift plan; query `sensor`, `scenario`, `standard`. |
|
||||||
|
| `GET /api/scenario` | Current scenario. |
|
||||||
|
| `POST /api/scenario` | Set scenario (control auth) and broadcast it over SSE. |
|
||||||
|
| `GET /api/stream` | SSE scenario events and the 15 s heartbeat. |
|
||||||
|
| `POST /api/train` | Proxy training to FastAPI (control auth). |
|
||||||
|
| `GET /api/train/{job_id}` | Training job status (control auth). |
|
||||||
|
|
||||||
## Checks
|
## Checks
|
||||||
|
|
||||||
@@ -10,7 +43,63 @@ make check
|
|||||||
|
|
||||||
Runs `go vet ./...`, `go test -count=1 ./...`, and `go build ./...`. The same
|
Runs `go vet ./...`, `go test -count=1 ./...`, and `go build ./...`. The same
|
||||||
target runs in Gitea Actions on every push and pull request
|
target runs in Gitea Actions on every push and pull request
|
||||||
(`.gitea/workflows/ci.yml`).
|
(`.gitea/workflows/ci.yml`). CI does not run the integration or load tests;
|
||||||
|
those need the ML checkout and raw CSV.
|
||||||
|
|
||||||
|
## Runtime data
|
||||||
|
|
||||||
|
The API image contains **no baked data**. At startup it fetches
|
||||||
|
`GET {FASTAPI_URL}/bootstrap` and keeps `sensors`, `series`, `context` and
|
||||||
|
`meta` in memory. A background loop refreshes the snapshot; on failure it keeps
|
||||||
|
the last good snapshot and retries with exponential backoff
|
||||||
|
**1, 2, 4, 8, 16, 30 s (cap)**, resetting on success.
|
||||||
|
|
||||||
|
- `GET /api/readyz` returns `503` until the first valid snapshot is loaded.
|
||||||
|
After that, FastAPI being down does not fail readiness.
|
||||||
|
- `GET /api/meta` reports the cache state under `cache`:
|
||||||
|
`source`, `last_success_at`, `last_error`, `next_retry_at`, `failures`.
|
||||||
|
- `GET /api/forecast` calls FastAPI `/predict`, remembers the last good points
|
||||||
|
per `sensor|horizon|scenario`, and serves them with `"cached": true` when
|
||||||
|
FastAPI is unavailable.
|
||||||
|
- `testdata/` is used by tests only and is excluded from the image by
|
||||||
|
`.dockerignore`.
|
||||||
|
|
||||||
|
| Env | Default | Purpose |
|
||||||
|
|---|---|---|
|
||||||
|
| `PORT` | `8080` | HTTP listen port |
|
||||||
|
| `FASTAPI_URL` | `http://localhost:8000` | FastAPI base URL |
|
||||||
|
| `BOOTSTRAP_REFRESH` | `60s` | Healthy snapshot refresh interval |
|
||||||
|
| `BOOTSTRAP_TIMEOUT` | `20s` | Per-request `/bootstrap` timeout |
|
||||||
|
| `ML_TIMEOUT` | `1500ms` | Per-request `/predict` timeout |
|
||||||
|
| `SSE_HEARTBEAT` | `15s` | SSE ping interval |
|
||||||
|
| `CONTROL_USER` / `CONTROL_PASS` | unset | Basic auth for `POST /api/scenario` and `/api/train*` |
|
||||||
|
| `CONTROL_TOKEN` | unset | Optional Bearer token forwarded to FastAPI `/train*` |
|
||||||
|
|
||||||
|
## Integration and load test
|
||||||
|
|
||||||
|
`make integration` runs the real stack (it refuses to start if `:18000` or
|
||||||
|
`:18080` is busy):
|
||||||
|
|
||||||
|
1. starts FastAPI from the ML checkout on `:18000` and the Go API on `:18080`;
|
||||||
|
2. uploads a date-range CSV subset to `POST /api/train` and polls the job
|
||||||
|
(`small`, then `forecast` by default);
|
||||||
|
3. verifies the live forecast still answers, scenario propagation, the 15 s
|
||||||
|
heartbeat and reconnect behaviour;
|
||||||
|
4. stops FastAPI, verifies `/api/meta.cache`, snapshot-served sensors, the
|
||||||
|
last-good forecast (`cached: true`) and staffing fallback;
|
||||||
|
5. restarts FastAPI and waits for the cache to recover.
|
||||||
|
|
||||||
|
`make integration-load` additionally runs `scripts/sse-load`: 120 concurrent
|
||||||
|
SSE clients for 5 minutes with a scenario change every 30 s. It records peak
|
||||||
|
connections, drops, per-change and overall p95 latency, and heartbeat cadence.
|
||||||
|
The tool exits non-zero when a client drops, an event is missing, or p95
|
||||||
|
scenario latency exceeds the contract's 2 s threshold.
|
||||||
|
|
||||||
|
Results land in `/tmp/pulsecast-integration/results` by default. Override with
|
||||||
|
`WORKDIR=/path` or `RESULTS_DIR=/path`. `make load` runs only the load test
|
||||||
|
against an API that is already up. The verified run is written up in
|
||||||
|
[`docs/integration.md`](docs/integration.md) with raw evidence under
|
||||||
|
`docs/results/`.
|
||||||
|
|
||||||
## Container
|
## Container
|
||||||
|
|
||||||
|
|||||||
+14
-11
@@ -1,4 +1,4 @@
|
|||||||
// Command api serves the PulseCast Go API, SSE stream and cached fallback.
|
// Command api serves the PulseCast Go API, SSE stream and runtime snapshot cache.
|
||||||
package main
|
package main
|
||||||
|
|
||||||
import (
|
import (
|
||||||
@@ -12,38 +12,41 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
|
|
||||||
"pulsecast/api/internal/api"
|
"pulsecast/api/internal/api"
|
||||||
|
"pulsecast/api/internal/cache"
|
||||||
"pulsecast/api/internal/config"
|
"pulsecast/api/internal/config"
|
||||||
"pulsecast/api/internal/ml"
|
"pulsecast/api/internal/ml"
|
||||||
"pulsecast/api/internal/sse"
|
"pulsecast/api/internal/sse"
|
||||||
"pulsecast/api/internal/store"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
func main() {
|
func main() {
|
||||||
cfg := config.Load()
|
cfg := config.Load()
|
||||||
|
|
||||||
// A partial dataset is not fatal: the process starts and /api/readyz reports
|
|
||||||
// 503 until the baked artifacts are complete, instead of crash-looping.
|
|
||||||
st, err := store.Load(cfg.DataDir)
|
|
||||||
if err != nil {
|
|
||||||
slog.Error("baked data is incomplete", "dir", cfg.DataDir, "error", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
|
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
|
||||||
defer stop()
|
defer stop()
|
||||||
|
|
||||||
|
client := ml.NewWithToken(cfg.FastAPIURL, cfg.MLTimeout, cfg.ControlToken)
|
||||||
|
snapshots := cache.New(client, cache.Options{
|
||||||
|
RefreshInterval: cfg.RefreshInterval,
|
||||||
|
Timeout: cfg.BootstrapTimeout,
|
||||||
|
})
|
||||||
|
// Loading happens in the background so the process can answer /api/readyz
|
||||||
|
// with 503 immediately instead of crash-looping while FastAPI starts.
|
||||||
|
go snapshots.Run(ctx)
|
||||||
|
|
||||||
hub := sse.NewHub(cfg.Heartbeat)
|
hub := sse.NewHub(cfg.Heartbeat)
|
||||||
go hub.Run(ctx)
|
go hub.Run(ctx)
|
||||||
|
|
||||||
server := &http.Server{
|
server := &http.Server{
|
||||||
Addr: cfg.Addr,
|
Addr: cfg.Addr,
|
||||||
Handler: api.New(cfg, st, ml.NewWithToken(cfg.FastAPIURL, cfg.MLTimeout, cfg.ControlToken), hub).Handler(),
|
Handler: api.New(cfg, snapshots, client, hub).Handler(),
|
||||||
ReadHeaderTimeout: 5 * time.Second,
|
ReadHeaderTimeout: 5 * time.Second,
|
||||||
IdleTimeout: 120 * time.Second,
|
IdleTimeout: 120 * time.Second,
|
||||||
}
|
}
|
||||||
|
|
||||||
serveErr := make(chan error, 1)
|
serveErr := make(chan error, 1)
|
||||||
go func() {
|
go func() {
|
||||||
slog.Info("api listening", "addr", cfg.Addr, "data_dir", cfg.DataDir, "fastapi", cfg.FastAPIURL)
|
slog.Info("api listening", "addr", cfg.Addr, "fastapi", cfg.FastAPIURL,
|
||||||
|
"refresh", cfg.RefreshInterval)
|
||||||
serveErr <- server.ListenAndServe()
|
serveErr <- server.ListenAndServe()
|
||||||
}()
|
}()
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,143 @@
|
|||||||
|
# Integration and load-test report
|
||||||
|
|
||||||
|
Run: **2026-11-14**, Go API worktree `feat/integration`.
|
||||||
|
FastAPI: real ML service from `ML-py.git/main` on `127.0.0.1:18000`.
|
||||||
|
Go API: this worktree on `127.0.0.1:18080`, `FASTAPI_URL=http://127.0.0.1:18000`.
|
||||||
|
Everything below is reproducible with `make integration-load`; raw evidence is
|
||||||
|
in `docs/results/`.
|
||||||
|
|
||||||
|
## Command
|
||||||
|
|
||||||
|
```sh
|
||||||
|
WORKDIR=/tmp/pulsecast-full-run RESULTS_DIR="$PWD/docs/results" \
|
||||||
|
scripts/integration.sh --load
|
||||||
|
```
|
||||||
|
|
||||||
|
The script owns `:18000` and `:18080` (it refuses to run if either is busy).
|
||||||
|
It uses a 24,636-row CSV subset (`2026-09-01 .. 2026-10-16`, all 8 sensors,
|
||||||
|
`densityData` rows only) generated from the raw challenge CSV into `/tmp`.
|
||||||
|
The subset is test input only; it is not committed and not baked into the image.
|
||||||
|
|
||||||
|
## Train end-to-end (real FastAPI, through the Go proxy)
|
||||||
|
|
||||||
|
Both jobs were uploaded as multipart CSV to `POST /api/train` with control auth,
|
||||||
|
then polled through `GET /api/train/{job_id}`.
|
||||||
|
|
||||||
|
| Job | Result | Model version | Submit to terminal | Headline metrics |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| `small` | `succeeded`, `error: null` | `logit-20261114-053410` | **171 ms** (POST itself 25 ms) | accuracy 0.824, precision 0.589, recall 0.420, F1 0.490, ROC-AUC 0.849, n_train 5,363, n_test 1,728, 8 sensors |
|
||||||
|
| `forecast` | `succeeded`, `error: null` | `histgb-20261114-053413` | **3,571 ms** | MAE 4.83, WAPE 33.0, R² 0.703, n_eval 4,919 |
|
||||||
|
|
||||||
|
- `small` is a logistic-regression busy classifier; `forecast` retrains the
|
||||||
|
global HistGB. Both ran in FastAPI; the Go process only proxied and polled.
|
||||||
|
- The forecast job trained on the **small subset**, so its MAE 4.83 is not the
|
||||||
|
baked model's quality (3.32 on the full data). It proves the upload, queue,
|
||||||
|
poll and hot-swap path, not model quality.
|
||||||
|
- `/api/forecast?sensor=hn-stadtgalerie-01&horizon=7d&scenario=base` returned
|
||||||
|
**200, 168 points, `cached: false`** immediately after the small job.
|
||||||
|
- After the forecast hot-swap, the Go snapshot refresh picked up the new
|
||||||
|
`model_version` (`histgb-20261114-053413`) in `/api/meta`.
|
||||||
|
|
||||||
|
## SSE scenario, heartbeat and reconnect
|
||||||
|
|
||||||
|
| Check | Result |
|
||||||
|
|---|---|
|
||||||
|
| Connect payload | `retry: 3000`, current `scenario` event immediately |
|
||||||
|
| Scenario propagation | **15 ms** from `POST /api/scenario` to the client event (limit: 2 s) |
|
||||||
|
| Heartbeat | received during the check; server interval exactly **15,000 ms** throughout the load test |
|
||||||
|
| Reconnect | closed and reopened the stream; current scenario (v2) arrived in **18 ms** |
|
||||||
|
| Event shape | `event: scenario`, `id: <version>`, JSON body with `version`, `scenario`, `at` |
|
||||||
|
|
||||||
|
## Staffing plan generation (contract: < 2 s)
|
||||||
|
|
||||||
|
A follow-up run on the same commit (ports `18100`/`18180`, because another
|
||||||
|
verification process held `18000`/`18080`) timed `GET /api/staffing`:
|
||||||
|
|
||||||
|
| Check | Result |
|
||||||
|
|---|---|
|
||||||
|
| Cold call (live `/predict` + plan build) | **19 ms** |
|
||||||
|
| Warm call (same plan key) | **< 1 ms** |
|
||||||
|
| Plan size | 70 store hours (7 days × 10:00–20:00), 21 shift blocks |
|
||||||
|
| Coverage | 100%, cost €3,360 |
|
||||||
|
| `cached` on the live plan | `false` |
|
||||||
|
|
||||||
|
The repeat run also reproduced the other checks: small train 169 ms, scenario
|
||||||
|
propagation 13 ms, reconnect 16 ms, recovery 1,339 ms.
|
||||||
|
|
||||||
|
## 120-client / 5-minute load test
|
||||||
|
|
||||||
|
`scripts/sse-load`: 120 concurrent streams, 300.008 s, scenario change every
|
||||||
|
30 s (rotating `sunny-saturday-market`, `rainy-saturday`, `weekday-event`,
|
||||||
|
`base`).
|
||||||
|
|
||||||
|
| Metric | Result |
|
||||||
|
|---|---|
|
||||||
|
| Peak connections | **120 / 120** |
|
||||||
|
| Connected after warmup | 120 / 120 |
|
||||||
|
| Dropped or reconnected streams | **0** |
|
||||||
|
| Scenario changes | 9 |
|
||||||
|
| Scenario events received | **1,080 / 1,080** (120 per change) |
|
||||||
|
| Missing scenario events | **0** |
|
||||||
|
| Scenario latency samples | 1,080 |
|
||||||
|
| p50 | **1.3 ms** |
|
||||||
|
| p95 | **5.6 ms** |
|
||||||
|
| p99 | **5.8 ms** |
|
||||||
|
| max | **5.812 ms** |
|
||||||
|
| Heartbeat pings | 20 unique server timestamps, every **15,000 ms** (min = median = p95 = max) |
|
||||||
|
| Heartbeats per client | 20 / 20 / 20 (min / median / max) |
|
||||||
|
|
||||||
|
Per-change p95 ranged from 1.2 ms to 5.8 ms; the highest latencies were the
|
||||||
|
first change after warmup. The tool exits non-zero if a client drops, an event
|
||||||
|
is missing, or p95 exceeds the contract's 2 s limit; this run passed.
|
||||||
|
|
||||||
|
## Fallback and recovery (runtime cache)
|
||||||
|
|
||||||
|
Sequence: warm a live base forecast, stop FastAPI, observe the cache failure,
|
||||||
|
serve the snapshot/last-good data, restart FastAPI, wait for recovery.
|
||||||
|
|
||||||
|
While FastAPI was down:
|
||||||
|
|
||||||
|
| Check | Result |
|
||||||
|
|---|---|
|
||||||
|
| `/api/meta.cache.failures` | **1** |
|
||||||
|
| `/api/meta.cache.last_error` | `ml: /bootstrap: Get "http://127.0.0.1:18000/bootstrap": dial tcp 127.0.0.1:18000: connect: connection refused` |
|
||||||
|
| `/api/meta.cache.next_retry_at` | `2026-11-14T05:34:31Z` (1 s after the first failure; schedule is 1/2/4/8/16/30 s) |
|
||||||
|
| `/api/meta.cache.last_success_at` | `2026-11-14T05:34:20Z` (unchanged) |
|
||||||
|
| `/api/readyz` | **200** — a loaded snapshot keeps readiness even with FastAPI down |
|
||||||
|
| `/api/sensors` | **200**, 8 sensors, from the last good snapshot |
|
||||||
|
| `/api/context` | **200**, `cached: true` |
|
||||||
|
| `/api/series` | **200**, from the last good snapshot |
|
||||||
|
| `/api/forecast` (warm key) | **200**, 168 points, **`cached: true`** |
|
||||||
|
| `/api/staffing` | **200**, built from the last good forecast, **`cached: true`** |
|
||||||
|
| `/api/forecast` (never-warmed key) | **503** `forecast_unavailable` — no invented data |
|
||||||
|
|
||||||
|
After restarting FastAPI:
|
||||||
|
|
||||||
|
| Check | Result |
|
||||||
|
|---|---|
|
||||||
|
| Cache recovery | **1,622 ms** after restart; `failures: 0` |
|
||||||
|
| `last_success_at` | advanced from `05:34:20Z` to `05:34:33Z` |
|
||||||
|
| `/api/forecast` | **200**, `cached: false` again |
|
||||||
|
|
||||||
|
The integration script used `BOOTSTRAP_REFRESH=10s` to keep the observation
|
||||||
|
window short. The production default is 60 s; the failure path, the
|
||||||
|
1/2/4/8/16/30 s retry schedule and the recovery semantics are unchanged.
|
||||||
|
|
||||||
|
## Evidence files
|
||||||
|
|
||||||
|
- `docs/results/train-small.json`, `train-forecast.json` — raw job responses and timings
|
||||||
|
- `docs/results/staffing.json` — cold/warm plan timing from the follow-up run
|
||||||
|
- `docs/results/sse.json`, `sse-stream.log` — propagation, heartbeat, reconnect
|
||||||
|
- `docs/results/load.json`, `load.txt` — all load-test samples and per-change results
|
||||||
|
- `docs/results/fallback.json` — exact cache/fallback/recovery values
|
||||||
|
- `docs/results/summary.txt` — one-screen summary of the same run
|
||||||
|
- `docs/results/meta-initial.json`, `forecast-after-small.json` — baseline evidence
|
||||||
|
|
||||||
|
## Notes and blockers
|
||||||
|
|
||||||
|
- At one point `:18000` was held by another agent's ML end-to-end run. The test
|
||||||
|
was not started against that process; it was rerun once the port was free.
|
||||||
|
No blocker remains.
|
||||||
|
- FastAPI was trained on a subset, so the retrained forecast metrics are lower
|
||||||
|
than the baked full-data model. The baked model is restored on FastAPI
|
||||||
|
restart, and recovery was verified with that baked model.
|
||||||
@@ -0,0 +1,19 @@
|
|||||||
|
{
|
||||||
|
"fastapi_down": {
|
||||||
|
"cache_failures": 1,
|
||||||
|
"last_error": "ml: /bootstrap: Get \"http://127.0.0.1:18000/bootstrap\": dial tcp 127.0.0.1:18000: connect: connection refused",
|
||||||
|
"next_retry_at": "2026-11-14T05:34:31Z",
|
||||||
|
"last_success_at": "2026-11-14T05:34:20Z",
|
||||||
|
"forecast_cached": true,
|
||||||
|
"staffing_cached": true,
|
||||||
|
"readyz": 200,
|
||||||
|
"sensors": 8
|
||||||
|
},
|
||||||
|
"recovery": {
|
||||||
|
"after_ms": 1622,
|
||||||
|
"last_success_before": "2026-11-14T05:34:20Z",
|
||||||
|
"last_success_after": "2026-11-14T05:34:33Z",
|
||||||
|
"forecast_cached": false,
|
||||||
|
"failures": 0
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
{"sensor":"hn-stadtgalerie-01","horizon":"7d","scenario":"base","points":[{"t":"2026-11-14T06:00:00Z","v":4.35,"lo":0,"hi":9.23},{"t":"2026-11-14T07:00:00Z","v":8.07,"lo":3.25,"hi":12.95},{"t":"2026-11-14T08:00:00Z","v":23.45,"lo":18.63,"hi":28.33},{"t":"2026-11-14T09:00:00Z","v":35.8,"lo":30.98,"hi":40.68},{"t":"2026-11-14T10:00:00Z","v":67.45,"lo":62.62,"hi":72.32},{"t":"2026-11-14T11:00:00Z","v":86.07,"lo":81.25,"hi":90.95},{"t":"2026-11-14T12:00:00Z","v":88.9,"lo":84.08,"hi":93.78},{"t":"2026-11-14T13:00:00Z","v":87.06,"lo":82.23,"hi":91.94},{"t":"2026-11-14T14:00:00Z","v":88.1,"lo":83.27,"hi":92.97},{"t":"2026-11-14T15:00:00Z","v":84.2,"lo":79.37,"hi":89.07},{"t":"2026-11-14T16:00:00Z","v":85.68,"lo":80.86,"hi":90.56},{"t":"2026-11-14T17:00:00Z","v":76.15,"lo":71.33,"hi":81.03},{"t":"2026-11-14T18:00:00Z","v":45.34,"lo":40.51,"hi":50.21},{"t":"2026-11-14T19:00:00Z","v":20.09,"lo":15.26,"hi":24.97},{"t":"2026-11-14T20:00:00Z","v":13,"lo":8.17,"hi":17.87},{"t":"2026-11-14T21:00:00Z","v":7.36,"lo":2.54,"hi":12.24},{"t":"2026-11-14T22:00:00Z","v":5.2,"lo":0.37,"hi":10.07},{"t":"2026-11-14T23:00:00Z","v":4.64,"lo":0,"hi":9.52},{"t":"2026-11-15T00:00:00Z","v":3.25,"lo":0,"hi":8.13},{"t":"2026-11-15T01:00:00Z","v":2.35,"lo":0,"hi":7.22},{"t":"2026-11-15T02:00:00Z","v":2.46,"lo":0,"hi":7.34},{"t":"2026-11-15T03:00:00Z","v":1.78,"lo":0,"hi":6.66},{"t":"2026-11-15T04:00:00Z","v":1.78,"lo":0,"hi":6.65},{"t":"2026-11-15T05:00:00Z","v":1.77,"lo":0,"hi":6.65},{"t":"2026-11-15T06:00:00Z","v":2.16,"lo":0,"hi":7.04},{"t":"2026-11-15T07:00:00Z","v":2.71,"lo":0,"hi":7.59},{"t":"2026-11-15T08:00:00Z","v":4.92,"lo":0.09,"hi":9.79},{"t":"2026-11-15T09:00:00Z","v":6.51,"lo":1.68,"hi":11.39},{"t":"2026-11-15T10:00:00Z","v":6.41,"lo":1.58,"hi":11.28},{"t":"2026-11-15T11:00:00Z","v":12.24,"lo":7.41,"hi":17.11},{"t":"2026-11-15T12:00:00Z","v":17.2,"lo":12.38,"hi":22.08},{"t":"2026-11-15T13:00:00Z","v":17.44,"lo":12.61,"hi":22.31},{"t":"2026-11-15T14:00:00Z","v":17.05,"lo":12.23,"hi":21.93},{"t":"2026-11-15T15:00:00Z","v":15.94,"lo":11.12,"hi":20.82},{"t":"2026-11-15T16:00:00Z","v":16.44,"lo":11.61,"hi":21.32},{"t":"2026-11-15T17:00:00Z","v":14.1,"lo":9.27,"hi":18.97},{"t":"2026-11-15T18:00:00Z","v":8.92,"lo":4.1,"hi":13.8},{"t":"2026-11-15T19:00:00Z","v":4.78,"lo":0,"hi":9.65},{"t":"2026-11-15T20:00:00Z","v":2.49,"lo":0,"hi":7.36},{"t":"2026-11-15T21:00:00Z","v":1.6,"lo":0,"hi":6.47},{"t":"2026-11-15T22:00:00Z","v":1.84,"lo":0,"hi":6.72},{"t":"2026-11-15T23:00:00Z","v":1.69,"lo":0,"hi":6.57},{"t":"2026-11-16T00:00:00Z","v":1.59,"lo":0,"hi":6.47},{"t":"2026-11-16T01:00:00Z","v":1.57,"lo":0,"hi":6.45},{"t":"2026-11-16T02:00:00Z","v":1.57,"lo":0,"hi":6.45},{"t":"2026-11-16T03:00:00Z","v":1.57,"lo":0,"hi":6.45},{"t":"2026-11-16T04:00:00Z","v":1.61,"lo":0,"hi":6.48},{"t":"2026-11-16T05:00:00Z","v":2.59,"lo":0,"hi":7.46},{"t":"2026-11-16T06:00:00Z","v":7.05,"lo":2.22,"hi":11.93},{"t":"2026-11-16T07:00:00Z","v":14.22,"lo":9.39,"hi":19.09},{"t":"2026-11-16T08:00:00Z","v":18.83,"lo":14.01,"hi":23.71},{"t":"2026-11-16T09:00:00Z","v":28.14,"lo":23.31,"hi":33.01},{"t":"2026-11-16T10:00:00Z","v":39.14,"lo":34.31,"hi":44.01},{"t":"2026-11-16T11:00:00Z","v":51.34,"lo":46.52,"hi":56.22},{"t":"2026-11-16T12:00:00Z","v":50.18,"lo":45.35,"hi":55.05},{"t":"2026-11-16T13:00:00Z","v":50.22,"lo":45.4,"hi":55.1},{"t":"2026-11-16T14:00:00Z","v":50.3,"lo":45.47,"hi":55.17},{"t":"2026-11-16T15:00:00Z","v":49.68,"lo":44.85,"hi":54.56},{"t":"2026-11-16T16:00:00Z","v":51.29,"lo":46.46,"hi":56.16},{"t":"2026-11-16T17:00:00Z","v":43.09,"lo":38.26,"hi":47.96},{"t":"2026-11-16T18:00:00Z","v":21.34,"lo":16.51,"hi":26.22},{"t":"2026-11-16T19:00:00Z","v":11.95,"lo":7.12,"hi":16.82},{"t":"2026-11-16T20:00:00Z","v":5.95,"lo":1.13,"hi":10.83},{"t":"2026-11-16T21:00:00Z","v":2.98,"lo":0,"hi":7.85},{"t":"2026-11-16T22:00:00Z","v":2.19,"lo":0,"hi":7.07},{"t":"2026-11-16T23:00:00Z","v":2,"lo":0,"hi":6.87},{"t":"2026-11-17T00:00:00Z","v":1.92,"lo":0,"hi":6.79},{"t":"2026-11-17T01:00:00Z","v":1.68,"lo":0,"hi":6.55},{"t":"2026-11-17T02:00:00Z","v":1.59,"lo":0,"hi":6.46},{"t":"2026-11-17T03:00:00Z","v":1.59,"lo":0,"hi":6.46},{"t":"2026-11-17T04:00:00Z","v":2.4,"lo":0,"hi":7.28},{"t":"2026-11-17T05:00:00Z","v":3.38,"lo":0,"hi":8.25},{"t":"2026-11-17T06:00:00Z","v":8.12,"lo":3.29,"hi":12.99},{"t":"2026-11-17T07:00:00Z","v":16.28,"lo":11.45,"hi":21.16},{"t":"2026-11-17T08:00:00Z","v":17.6,"lo":12.77,"hi":22.47},{"t":"2026-11-17T09:00:00Z","v":27.47,"lo":22.65,"hi":32.35},{"t":"2026-11-17T10:00:00Z","v":35.28,"lo":30.45,"hi":40.15},{"t":"2026-11-17T11:00:00Z","v":50.47,"lo":45.64,"hi":55.34},{"t":"2026-11-17T12:00:00Z","v":48.11,"lo":43.28,"hi":52.98},{"t":"2026-11-17T13:00:00Z","v":42.24,"lo":37.41,"hi":47.11},{"t":"2026-11-17T14:00:00Z","v":49.05,"lo":44.22,"hi":53.92},{"t":"2026-11-17T15:00:00Z","v":48.19,"lo":43.37,"hi":53.07},{"t":"2026-11-17T16:00:00Z","v":47.31,"lo":42.48,"hi":52.18},{"t":"2026-11-17T17:00:00Z","v":32.93,"lo":28.1,"hi":37.8},{"t":"2026-11-17T18:00:00Z","vLine truncated
|
||||||
@@ -0,0 +1,131 @@
|
|||||||
|
{
|
||||||
|
"started_at": "2026-11-14T05:34:33.422221537Z",
|
||||||
|
"duration_s": 300.008,
|
||||||
|
"base_url": "http://127.0.0.1:18080",
|
||||||
|
"clients_configured": 120,
|
||||||
|
"clients_peak_connected": 120,
|
||||||
|
"clients_connected_after_warmup": 120,
|
||||||
|
"connection_drops": 0,
|
||||||
|
"scenario_post_errors": 0,
|
||||||
|
"scenario_changes": 9,
|
||||||
|
"scenario_events_received": 1080,
|
||||||
|
"scenario_events_missing": 0,
|
||||||
|
"scenario_latency_samples": 1080,
|
||||||
|
"scenario_p50_ms": 1.3,
|
||||||
|
"scenario_p95_ms": 5.6,
|
||||||
|
"scenario_p99_ms": 5.8,
|
||||||
|
"scenario_max_ms": 5.812,
|
||||||
|
"changes": [
|
||||||
|
{
|
||||||
|
"version": 3,
|
||||||
|
"post_at": "2026-11-14T06:35:03.474811874+01:00",
|
||||||
|
"connected_at_post": 120,
|
||||||
|
"received": 120,
|
||||||
|
"missing": 0,
|
||||||
|
"p50_ms": 5.5,
|
||||||
|
"p95_ms": 5.8,
|
||||||
|
"p99_ms": 5.8,
|
||||||
|
"max_ms": 5.812
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"version": 6,
|
||||||
|
"post_at": "2026-11-14T06:35:33.474996501+01:00",
|
||||||
|
"connected_at_post": 120,
|
||||||
|
"received": 120,
|
||||||
|
"missing": 0,
|
||||||
|
"p50_ms": 2.4,
|
||||||
|
"p95_ms": 2.7,
|
||||||
|
"p99_ms": 2.8,
|
||||||
|
"max_ms": 2.791
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"version": 7,
|
||||||
|
"post_at": "2026-11-14T06:36:03.474746595+01:00",
|
||||||
|
"connected_at_post": 120,
|
||||||
|
"received": 120,
|
||||||
|
"missing": 0,
|
||||||
|
"p50_ms": 1.1,
|
||||||
|
"p95_ms": 1.3,
|
||||||
|
"p99_ms": 1.4,
|
||||||
|
"max_ms": 1.353
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"version": 8,
|
||||||
|
"post_at": "2026-11-14T06:36:33.467364475+01:00",
|
||||||
|
"connected_at_post": 120,
|
||||||
|
"received": 120,
|
||||||
|
"missing": 0,
|
||||||
|
"p50_ms": 1.7,
|
||||||
|
"p95_ms": 2.4,
|
||||||
|
"p99_ms": 2.4,
|
||||||
|
"max_ms": 2.396
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"version": 9,
|
||||||
|
"post_at": "2026-11-14T06:37:03.472992832+01:00",
|
||||||
|
"connected_at_post": 120,
|
||||||
|
"received": 120,
|
||||||
|
"missing": 0,
|
||||||
|
"p50_ms": 1,
|
||||||
|
"p95_ms": 1.3,
|
||||||
|
"p99_ms": 1.3,
|
||||||
|
"max_ms": 1.342
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"version": 10,
|
||||||
|
"post_at": "2026-11-14T06:37:33.475595017+01:00",
|
||||||
|
"connected_at_post": 120,
|
||||||
|
"received": 120,
|
||||||
|
"missing": 0,
|
||||||
|
"p50_ms": 1.1,
|
||||||
|
"p95_ms": 1.3,
|
||||||
|
"p99_ms": 1.4,
|
||||||
|
"max_ms": 1.434
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"version": 11,
|
||||||
|
"post_at": "2026-11-14T06:38:03.474743535+01:00",
|
||||||
|
"connected_at_post": 120,
|
||||||
|
"received": 120,
|
||||||
|
"missing": 0,
|
||||||
|
"p50_ms": 1.3,
|
||||||
|
"p95_ms": 1.4,
|
||||||
|
"p99_ms": 1.4,
|
||||||
|
"max_ms": 1.375
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"version": 12,
|
||||||
|
"post_at": "2026-11-14T06:38:33.469461499+01:00",
|
||||||
|
"connected_at_post": 120,
|
||||||
|
"received": 120,
|
||||||
|
"missing": 0,
|
||||||
|
"p50_ms": 1.5,
|
||||||
|
"p95_ms": 1.7,
|
||||||
|
"p99_ms": 1.7,
|
||||||
|
"max_ms": 1.718
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"version": 13,
|
||||||
|
"post_at": "2026-11-14T06:39:03.475695318+01:00",
|
||||||
|
"connected_at_post": 120,
|
||||||
|
"received": 120,
|
||||||
|
"missing": 0,
|
||||||
|
"p50_ms": 1,
|
||||||
|
"p95_ms": 1.2,
|
||||||
|
"p99_ms": 1.2,
|
||||||
|
"max_ms": 1.249
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"heartbeat": {
|
||||||
|
"unique_server_timestamps": 20,
|
||||||
|
"min_interval_ms": 15000,
|
||||||
|
"median_interval_ms": 15000,
|
||||||
|
"p95_interval_ms": 15000,
|
||||||
|
"max_interval_ms": 15000,
|
||||||
|
"per_client_min": 20,
|
||||||
|
"per_client_median": 20,
|
||||||
|
"per_client_max": 20
|
||||||
|
},
|
||||||
|
"errors": null,
|
||||||
|
"passed": true
|
||||||
|
}
|
||||||
@@ -0,0 +1,21 @@
|
|||||||
|
clients connected: 120/120 after warmup
|
||||||
|
scenario v3: 120/120 clients p50=5.5ms p95=5.8ms max=5.8ms
|
||||||
|
scenario v6: 120/120 clients p50=2.4ms p95=2.7ms max=2.8ms
|
||||||
|
scenario v7: 120/120 clients p50=1.1ms p95=1.3ms max=1.4ms
|
||||||
|
scenario v8: 120/120 clients p50=1.7ms p95=2.4ms max=2.4ms
|
||||||
|
scenario v9: 120/120 clients p50=1.0ms p95=1.3ms max=1.3ms
|
||||||
|
scenario v10: 120/120 clients p50=1.1ms p95=1.3ms max=1.4ms
|
||||||
|
scenario v11: 120/120 clients p50=1.3ms p95=1.4ms max=1.4ms
|
||||||
|
scenario v12: 120/120 clients p50=1.5ms p95=1.7ms max=1.7ms
|
||||||
|
scenario v13: 120/120 clients p50=1.0ms p95=1.2ms max=1.2ms
|
||||||
|
|
||||||
|
=== SSE load test ===
|
||||||
|
duration: 300.0s
|
||||||
|
clients: 120 configured, 120 peak connected, 120 after warmup
|
||||||
|
connections: 0 drops/reconnects, 0 scenario POST errors
|
||||||
|
scenario changes: 9 (1080 events received, 0 missing)
|
||||||
|
scenario latency: p50=1.3ms p95=5.6ms p99=5.8ms max=5.8ms (n=1080)
|
||||||
|
heartbeat: 20 server pings, interval min=15000ms median=15000ms p95=15000ms max=15000ms
|
||||||
|
heartbeat/client: min=20 median=20 max=20
|
||||||
|
result: PASS
|
||||||
|
json written to /home/noel/Documents/Hackathon_2026/api-go.git/feat-integration/docs/results/load.json
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
{"data_version":"2026-11-07T23:51:00Z","model_version":"histgb-2026-11-14","context_version":"2026-11-14T03:47:09Z","coverage":{"sensors":8,"from":"2026-06-12T00:00:00Z","to":"2026-11-07T23:51:00Z","cadence_min":15},"metrics":{"model":"weather-7d","mae":3.32,"wape":27.5,"r2":0.777,"baseline_mae":5.4,"baseline_wape":44.7},"limits":["estimated devices, not people","8 sensors in a ~580 m inner-city cluster","5 months of multi-sensor data; no annual seasonality claim","observed weather in training, forecast weather in scenarios","holidays are context, not a large accuracy claim"],"cache":{"source":"fastapi","last_success_at":"2026-11-14T05:34:09Z","last_error":null,"next_retry_at":null,"failures":0}}
|
||||||
@@ -0,0 +1,13 @@
|
|||||||
|
retry: 3000
|
||||||
|
|
||||||
|
event: scenario
|
||||||
|
id: 1
|
||||||
|
data: {"version":1,"scenario":{"scenario":"base","weather":{"temp_delta_c":0,"precip_mm":0},"event":{"type":"","uplift":0}},"at":"2026-11-14T05:34:09Z"}
|
||||||
|
|
||||||
|
event: scenario
|
||||||
|
id: 2
|
||||||
|
data: {"version":2,"scenario":{"scenario":"sunny-saturday-market","weather":{"temp_delta_c":4,"precip_mm":0},"event":{"type":"market","uplift":0.2}},"at":"2026-11-14T05:34:20Z"}
|
||||||
|
|
||||||
|
event: ping
|
||||||
|
data: {"t":"2026-11-14T05:34:24Z"}
|
||||||
|
|
||||||
@@ -0,0 +1,9 @@
|
|||||||
|
{
|
||||||
|
"scenario_version": 2,
|
||||||
|
"propagation_ms": 15,
|
||||||
|
"heartbeat_after_ms": 4260,
|
||||||
|
"heartbeat_interval_s": 15,
|
||||||
|
"retry_ms": 3000,
|
||||||
|
"reconnect_ms": 18,
|
||||||
|
"reconnect_current_version": 2
|
||||||
|
}
|
||||||
@@ -0,0 +1,10 @@
|
|||||||
|
{
|
||||||
|
"cold_ms": 19,
|
||||||
|
"warm_ms": 0,
|
||||||
|
"hours": 70,
|
||||||
|
"shifts": 21,
|
||||||
|
"coverage_pct": 100,
|
||||||
|
"cost_eur": 3360,
|
||||||
|
"cached": false,
|
||||||
|
"note": "follow-up run on ports 18100/18180; same commit"
|
||||||
|
}
|
||||||
@@ -0,0 +1,7 @@
|
|||||||
|
PulseCast Go integration 2026-11-14T05:39:33Z
|
||||||
|
FastAPI http://127.0.0.1:18000, Go http://127.0.0.1:18080
|
||||||
|
small CSV: 24636 rows (2026-09-01 .. 2026-10-16)
|
||||||
|
small job: status=succeeded version=logit-20261114-053410 accuracy=0.824 roc_auc=0.849 submit_to_terminal_ms=171
|
||||||
|
SSE: {"scenario_version":2,"propagation_ms":15,"heartbeat_after_ms":4260,"heartbeat_interval_s":15,"retry_ms":3000,"reconnect_ms":18,"reconnect_current_version":2}
|
||||||
|
fallback: {"fastapi_down":{"cache_failures":1,"last_error":"ml: /bootstrap: Get \"http://127.0.0.1:18000/bootstrap\": dial tcp 127.0.0.1:18000: connect: connection refused","next_retry_at":"2026-11-14T05:34:31Z","last_success_at":"2026-11-14T05:34:20Z","forecast_cached":true,"staffing_cached":true,"readyz":200,"sensors":8},"recovery":{"after_ms":1622,"last_success_before":"2026-11-14T05:34:20Z","last_success_after":"2026-11-14T05:34:33Z","forecast_cached":false,"failures":0}}
|
||||||
|
forecast job: status=succeeded version=histgb-20261114-053413 mae=4.83 wape=33.0 submit_to_terminal_ms=3571
|
||||||
@@ -0,0 +1,77 @@
|
|||||||
|
{
|
||||||
|
"model": "forecast",
|
||||||
|
"submit_http": 202,
|
||||||
|
"submit_ms": 15,
|
||||||
|
"submit_to_terminal_ms": 3571,
|
||||||
|
"job": {
|
||||||
|
"job_id": "2f4e7c5a5bc14abd81d4eb5cdbeb0e1a",
|
||||||
|
"status": "succeeded",
|
||||||
|
"model": "forecast",
|
||||||
|
"submitted_at": "2026-11-14T05:34:10Z",
|
||||||
|
"started_at": "2026-11-14T05:34:10Z",
|
||||||
|
"finished_at": "2026-11-14T05:34:13Z",
|
||||||
|
"metrics": {
|
||||||
|
"model": "weather-7d",
|
||||||
|
"mae": 4.83,
|
||||||
|
"wape": 33.0,
|
||||||
|
"r2": 0.703,
|
||||||
|
"baseline": "seasonal_naive",
|
||||||
|
"baseline_mae": 6.8,
|
||||||
|
"baseline_wape": 46.5,
|
||||||
|
"baseline_r2": 0.318,
|
||||||
|
"n_eval": 4919,
|
||||||
|
"ablation": [
|
||||||
|
{
|
||||||
|
"name": "seasonal_naive",
|
||||||
|
"description": "exact 168-row lag (same observed hour last week)",
|
||||||
|
"mae": 6.8,
|
||||||
|
"wape": 46.5,
|
||||||
|
"r2": 0.318,
|
||||||
|
"n": 4919
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "day_no_weather",
|
||||||
|
"description": "calendar + lags 24/48/168 + 24h/168h rolling means",
|
||||||
|
"mae": 5.5,
|
||||||
|
"wape": 37.6,
|
||||||
|
"r2": 0.623,
|
||||||
|
"n": 4919
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "day_holiday",
|
||||||
|
"description": "day-ahead + Baden-Württemberg holiday flag",
|
||||||
|
"mae": 5.5,
|
||||||
|
"wape": 37.6,
|
||||||
|
"r2": 0.623,
|
||||||
|
"n": 4919
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "day_weather",
|
||||||
|
"description": "day-ahead + temperature + rain (observed weather)",
|
||||||
|
"mae": 5.03,
|
||||||
|
"wape": 34.4,
|
||||||
|
"r2": 0.682,
|
||||||
|
"n": 4919
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "week_no_weather",
|
||||||
|
"description": "7-day-safe calendar + previous same weekday/hour values",
|
||||||
|
"mae": 5.4,
|
||||||
|
"wape": 36.9,
|
||||||
|
"r2": 0.631,
|
||||||
|
"n": 4919
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "week_weather",
|
||||||
|
"description": "7-day-safe + temperature + rain (production model)",
|
||||||
|
"mae": 4.83,
|
||||||
|
"wape": 33.0,
|
||||||
|
"r2": 0.703,
|
||||||
|
"n": 4919
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"model_version": "histgb-20261114-053413",
|
||||||
|
"error": null
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,27 @@
|
|||||||
|
{
|
||||||
|
"model": "small",
|
||||||
|
"submit_http": 202,
|
||||||
|
"submit_ms": 25,
|
||||||
|
"submit_to_terminal_ms": 171,
|
||||||
|
"job": {
|
||||||
|
"job_id": "9b6112d7baa34ee8ad236db0d77b836b",
|
||||||
|
"status": "succeeded",
|
||||||
|
"model": "small",
|
||||||
|
"submitted_at": "2026-11-14T05:34:10Z",
|
||||||
|
"started_at": "2026-11-14T05:34:10Z",
|
||||||
|
"finished_at": "2026-11-14T05:34:10Z",
|
||||||
|
"metrics": {
|
||||||
|
"accuracy": 0.824,
|
||||||
|
"precision": 0.589,
|
||||||
|
"recall": 0.42,
|
||||||
|
"f1": 0.49,
|
||||||
|
"roc_auc": 0.849,
|
||||||
|
"n_train": 5363,
|
||||||
|
"n_test": 1728,
|
||||||
|
"positive_rate": 0.201,
|
||||||
|
"sensors": 8
|
||||||
|
},
|
||||||
|
"model_version": "logit-20261114-053410",
|
||||||
|
"error": null
|
||||||
|
}
|
||||||
|
}
|
||||||
+355
-30
@@ -10,18 +10,77 @@ import (
|
|||||||
"io"
|
"io"
|
||||||
"net/http"
|
"net/http"
|
||||||
"net/http/httptest"
|
"net/http/httptest"
|
||||||
|
"os"
|
||||||
"strings"
|
"strings"
|
||||||
|
"sync"
|
||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
|
"pulsecast/api/internal/cache"
|
||||||
"pulsecast/api/internal/config"
|
"pulsecast/api/internal/config"
|
||||||
"pulsecast/api/internal/ml"
|
"pulsecast/api/internal/ml"
|
||||||
"pulsecast/api/internal/model"
|
"pulsecast/api/internal/model"
|
||||||
"pulsecast/api/internal/sse"
|
"pulsecast/api/internal/sse"
|
||||||
"pulsecast/api/internal/store"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
const fixtureDir = "../../testdata/data"
|
const fixtureBootstrap = "../../testdata/data/bootstrap.json"
|
||||||
|
|
||||||
|
type stubFetcher struct {
|
||||||
|
mu sync.RWMutex
|
||||||
|
payload model.Bootstrap
|
||||||
|
err error
|
||||||
|
refreshFn func(context.Context) (model.Bootstrap, error)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *stubFetcher) Bootstrap(ctx context.Context) (model.Bootstrap, error) {
|
||||||
|
s.mu.RLock()
|
||||||
|
fn := s.refreshFn
|
||||||
|
payload, err := s.payload, s.err
|
||||||
|
s.mu.RUnlock()
|
||||||
|
if fn != nil {
|
||||||
|
return fn(ctx)
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
return model.Bootstrap{}, err
|
||||||
|
}
|
||||||
|
return payload, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *stubFetcher) fail(err error) {
|
||||||
|
s.mu.Lock()
|
||||||
|
s.err = err
|
||||||
|
s.mu.Unlock()
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *stubFetcher) recover(payload model.Bootstrap) {
|
||||||
|
s.mu.Lock()
|
||||||
|
s.payload, s.err = payload, nil
|
||||||
|
s.mu.Unlock()
|
||||||
|
}
|
||||||
|
|
||||||
|
func loadFixtureBootstrap(t *testing.T) model.Bootstrap {
|
||||||
|
t.Helper()
|
||||||
|
raw, err := os.ReadFile(fixtureBootstrap)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("read fixture: %v", err)
|
||||||
|
}
|
||||||
|
var payload model.Bootstrap
|
||||||
|
if err := json.Unmarshal(raw, &payload); err != nil {
|
||||||
|
t.Fatalf("decode fixture: %v", err)
|
||||||
|
}
|
||||||
|
return payload
|
||||||
|
}
|
||||||
|
|
||||||
|
func newFixtureCache(t *testing.T) (*cache.Cache, *stubFetcher) {
|
||||||
|
t.Helper()
|
||||||
|
payload := loadFixtureBootstrap(t)
|
||||||
|
fetcher := &stubFetcher{payload: payload}
|
||||||
|
snapshots := cache.New(fetcher, cache.Options{RefreshInterval: time.Hour, Timeout: time.Second})
|
||||||
|
if err := snapshots.Refresh(context.Background()); err != nil {
|
||||||
|
t.Fatalf("prime cache: %v", err)
|
||||||
|
}
|
||||||
|
return snapshots, fetcher
|
||||||
|
}
|
||||||
|
|
||||||
type stubML struct {
|
type stubML struct {
|
||||||
predict func(context.Context, ml.PredictRequest) ([]model.ForecastPoint, error)
|
predict func(context.Context, ml.PredictRequest) ([]model.ForecastPoint, error)
|
||||||
@@ -58,20 +117,16 @@ func (s *stubML) TrainStatus(ctx context.Context, jobID string) (int, []byte, er
|
|||||||
return s.trainStatus(ctx, jobID)
|
return s.trainStatus(ctx, jobID)
|
||||||
}
|
}
|
||||||
|
|
||||||
func loadFixtureStore(t *testing.T) *store.Store {
|
func newTestServer(t *testing.T, client MLClient, mutate func(*config.Config)) (*Server, *httptest.Server) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
st, err := store.Load(fixtureDir)
|
snapshots, _ := newFixtureCache(t)
|
||||||
if err != nil {
|
return newTestServerWithCache(t, snapshots, client, mutate)
|
||||||
t.Fatalf("load fixtures: %v", err)
|
|
||||||
}
|
|
||||||
return st
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func newTestServer(t *testing.T, client MLClient, mutate func(*config.Config)) (*Server, *httptest.Server) {
|
func newTestServerWithCache(t *testing.T, snapshots *cache.Cache, client MLClient, mutate func(*config.Config)) (*Server, *httptest.Server) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
cfg := config.Config{
|
cfg := config.Config{
|
||||||
Addr: ":0",
|
Addr: ":0",
|
||||||
DataDir: fixtureDir,
|
|
||||||
FastAPIURL: "http://127.0.0.1:1",
|
FastAPIURL: "http://127.0.0.1:1",
|
||||||
Heartbeat: time.Hour,
|
Heartbeat: time.Hour,
|
||||||
MLTimeout: 100 * time.Millisecond,
|
MLTimeout: 100 * time.Millisecond,
|
||||||
@@ -85,12 +140,47 @@ func newTestServer(t *testing.T, client MLClient, mutate func(*config.Config)) (
|
|||||||
t.Cleanup(cancel)
|
t.Cleanup(cancel)
|
||||||
go hub.Run(ctx)
|
go hub.Run(ctx)
|
||||||
|
|
||||||
srv := New(cfg, loadFixtureStore(t), client, hub)
|
srv := New(cfg, snapshots, client, hub)
|
||||||
ts := httptest.NewServer(srv.Handler())
|
ts := httptest.NewServer(srv.Handler())
|
||||||
t.Cleanup(ts.Close)
|
t.Cleanup(ts.Close)
|
||||||
return srv, ts
|
return srv, ts
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func forecastPoints(n int) []model.ForecastPoint {
|
||||||
|
start := time.Date(2026, 11, 16, 9, 0, 0, 0, time.UTC)
|
||||||
|
points := make([]model.ForecastPoint, n)
|
||||||
|
for i := range points {
|
||||||
|
points[i] = model.ForecastPoint{
|
||||||
|
T: start.Add(time.Duration(i) * time.Hour),
|
||||||
|
V: float64(10 + i),
|
||||||
|
Lo: float64(5 + i),
|
||||||
|
Hi: float64(15 + i),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return points
|
||||||
|
}
|
||||||
|
|
||||||
|
// weekForecast builds seven complete store-hour days (10:00-20:00 Berlin),
|
||||||
|
// which is what the staffing planner consumes.
|
||||||
|
func weekForecast(t *testing.T) []model.ForecastPoint {
|
||||||
|
t.Helper()
|
||||||
|
loc, err := time.LoadLocation(model.StoreHours.TZ)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
start := time.Date(2026, 11, 16, model.StoreHours.Open, 0, 0, 0, loc)
|
||||||
|
var points []model.ForecastPoint
|
||||||
|
for day := 0; day < 7; day++ {
|
||||||
|
for hour := 0; hour < model.StoreHours.Close-model.StoreHours.Open; hour++ {
|
||||||
|
points = append(points, model.ForecastPoint{
|
||||||
|
T: start.AddDate(0, 0, day).Add(time.Duration(hour) * time.Hour).UTC(),
|
||||||
|
V: 25,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return points
|
||||||
|
}
|
||||||
|
|
||||||
func getJSON(t *testing.T, url string, dst any) *http.Response {
|
func getJSON(t *testing.T, url string, dst any) *http.Response {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
resp, err := http.Get(url)
|
resp, err := http.Get(url)
|
||||||
@@ -125,13 +215,11 @@ func TestHealthAndReady(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestReadyzWithoutData(t *testing.T) {
|
func TestReadyzWithoutSnapshot(t *testing.T) {
|
||||||
cfg := config.Config{Addr: ":0", DataDir: t.TempDir(), FastAPIURL: "http://127.0.0.1:1", Heartbeat: time.Hour, MLTimeout: 50 * time.Millisecond}
|
cfg := config.Config{Addr: ":0", FastAPIURL: "http://127.0.0.1:1", Heartbeat: time.Hour, MLTimeout: 50 * time.Millisecond}
|
||||||
st, err := store.Load(cfg.DataDir)
|
fetcher := &stubFetcher{err: errors.New("fastapi unavailable")}
|
||||||
if err != nil {
|
snapshots := cache.New(fetcher, cache.Options{RefreshInterval: time.Hour, Timeout: time.Second})
|
||||||
t.Logf("expected load error: %v", err)
|
srv := New(cfg, snapshots, &stubML{}, sse.NewHub(time.Hour))
|
||||||
}
|
|
||||||
srv := New(cfg, st, &stubML{}, sse.NewHub(time.Hour))
|
|
||||||
ts := httptest.NewServer(srv.Handler())
|
ts := httptest.NewServer(srv.Handler())
|
||||||
defer ts.Close()
|
defer ts.Close()
|
||||||
|
|
||||||
@@ -143,6 +231,70 @@ func TestReadyzWithoutData(t *testing.T) {
|
|||||||
if ready["ok"] || ready["data"] || ready["model"] {
|
if ready["ok"] || ready["data"] || ready["model"] {
|
||||||
t.Fatalf("readyz body=%v, want all false", ready)
|
t.Fatalf("readyz body=%v, want all false", ready)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// One failed bootstrap attempt must not make the cache ready.
|
||||||
|
if err := snapshots.Refresh(context.Background()); err == nil {
|
||||||
|
t.Fatal("expected the bootstrap refresh to fail")
|
||||||
|
}
|
||||||
|
resp = getJSON(t, ts.URL+"/api/readyz", &ready)
|
||||||
|
if resp.StatusCode != http.StatusServiceUnavailable || ready["ok"] {
|
||||||
|
t.Fatalf("readyz after failed refresh: status=%d body=%v", resp.StatusCode, ready)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestReadyzSurvivesFastAPIFailureAfterFirstSnapshot(t *testing.T) {
|
||||||
|
snapshots, fetcher := newFixtureCache(t)
|
||||||
|
cfg := config.Config{Addr: ":0", FastAPIURL: "http://127.0.0.1:1", Heartbeat: time.Hour, MLTimeout: 50 * time.Millisecond}
|
||||||
|
srv := New(cfg, snapshots, &stubML{}, sse.NewHub(time.Hour))
|
||||||
|
ts := httptest.NewServer(srv.Handler())
|
||||||
|
defer ts.Close()
|
||||||
|
|
||||||
|
var ready map[string]bool
|
||||||
|
resp := getJSON(t, ts.URL+"/api/readyz", &ready)
|
||||||
|
if resp.StatusCode != http.StatusOK || !ready["ok"] || !ready["data"] || !ready["model"] {
|
||||||
|
t.Fatalf("readyz status=%d body=%v", resp.StatusCode, ready)
|
||||||
|
}
|
||||||
|
|
||||||
|
fetcher.fail(errors.New("connection refused"))
|
||||||
|
if err := snapshots.Refresh(context.Background()); err == nil {
|
||||||
|
t.Fatal("expected the refresh to fail")
|
||||||
|
}
|
||||||
|
|
||||||
|
resp = getJSON(t, ts.URL+"/api/readyz", &ready)
|
||||||
|
if resp.StatusCode != http.StatusOK || !ready["ok"] || !ready["data"] || !ready["model"] {
|
||||||
|
t.Fatalf("readyz after FastAPI failure: status=%d body=%v, want 200 with the last snapshot", resp.StatusCode, ready)
|
||||||
|
}
|
||||||
|
|
||||||
|
var meta model.Meta
|
||||||
|
getJSON(t, ts.URL+"/api/meta", &meta)
|
||||||
|
if meta.Cache.Failures != 1 || meta.Cache.LastError == nil ||
|
||||||
|
!strings.Contains(*meta.Cache.LastError, "connection refused") {
|
||||||
|
t.Fatalf("cache status=%+v", meta.Cache)
|
||||||
|
}
|
||||||
|
if meta.Cache.LastSuccessAt == nil || meta.Cache.NextRetryAt == nil {
|
||||||
|
t.Fatalf("cache timestamps missing: %+v", meta.Cache)
|
||||||
|
}
|
||||||
|
|
||||||
|
// The context endpoint comes from the snapshot; it is now stale.
|
||||||
|
var ctx model.ContextResponse
|
||||||
|
getJSON(t, ts.URL+"/api/context", &ctx)
|
||||||
|
if !ctx.Cached {
|
||||||
|
t.Error("context cached=false after a failed refresh, want true")
|
||||||
|
}
|
||||||
|
|
||||||
|
// A later success clears the failure state.
|
||||||
|
fetcher.recover(loadFixtureBootstrap(t))
|
||||||
|
if err := snapshots.Refresh(context.Background()); err != nil {
|
||||||
|
t.Fatalf("recovery refresh: %v", err)
|
||||||
|
}
|
||||||
|
getJSON(t, ts.URL+"/api/meta", &meta)
|
||||||
|
if meta.Cache.Failures != 0 || meta.Cache.LastError != nil || meta.Cache.NextRetryAt != nil {
|
||||||
|
t.Fatalf("cache status after recovery=%+v", meta.Cache)
|
||||||
|
}
|
||||||
|
getJSON(t, ts.URL+"/api/context", &ctx)
|
||||||
|
if ctx.Cached {
|
||||||
|
t.Error("context cached=true after a successful refresh, want false")
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestMetaSensorsAndVersionHeaders(t *testing.T) {
|
func TestMetaSensorsAndVersionHeaders(t *testing.T) {
|
||||||
@@ -156,6 +308,10 @@ func TestMetaSensorsAndVersionHeaders(t *testing.T) {
|
|||||||
if meta.Coverage.Sensors != 8 || meta.ModelVersion == "" || meta.ContextVersion == "" {
|
if meta.Coverage.Sensors != 8 || meta.ModelVersion == "" || meta.ContextVersion == "" {
|
||||||
t.Fatalf("meta=%+v", meta)
|
t.Fatalf("meta=%+v", meta)
|
||||||
}
|
}
|
||||||
|
if meta.Cache.Source != "fastapi" || meta.Cache.LastSuccessAt == nil ||
|
||||||
|
meta.Cache.LastError != nil || meta.Cache.NextRetryAt != nil || meta.Cache.Failures != 0 {
|
||||||
|
t.Fatalf("meta cache=%+v", meta.Cache)
|
||||||
|
}
|
||||||
for _, header := range []string{"X-Data-Version", "X-Model-Version", "X-Context-Version"} {
|
for _, header := range []string{"X-Data-Version", "X-Model-Version", "X-Context-Version"} {
|
||||||
if resp.Header.Get(header) == "" {
|
if resp.Header.Get(header) == "" {
|
||||||
t.Errorf("missing %s header", header)
|
t.Errorf("missing %s header", header)
|
||||||
@@ -180,6 +336,58 @@ func TestMetaSensorsAndVersionHeaders(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestDataEndpointsFollowSnapshotRefresh(t *testing.T) {
|
||||||
|
snapshots, fetcher := newFixtureCache(t)
|
||||||
|
_, ts := newTestServerWithCache(t, snapshots, &stubML{}, nil)
|
||||||
|
|
||||||
|
var sensors []model.Sensor
|
||||||
|
getJSON(t, ts.URL+"/api/sensors", &sensors)
|
||||||
|
if len(sensors) != 8 {
|
||||||
|
t.Fatalf("sensors=%d, want 8", len(sensors))
|
||||||
|
}
|
||||||
|
|
||||||
|
updated := loadFixtureBootstrap(t)
|
||||||
|
updated.DataVersion = "2026-11-14T05:00:00Z"
|
||||||
|
updated.ContextVersion = "2026-11-14T05:00:00Z"
|
||||||
|
updated.Sensors = updated.Sensors[:1]
|
||||||
|
updated.Series = map[string][]model.Point{
|
||||||
|
updated.Sensors[0].ID: {{T: time.Date(2026, 11, 1, 10, 0, 0, 0, time.UTC), V: 42}},
|
||||||
|
}
|
||||||
|
fetcher.recover(updated)
|
||||||
|
if err := snapshots.Refresh(context.Background()); err != nil {
|
||||||
|
t.Fatalf("refresh: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
sensors = nil
|
||||||
|
resp := getJSON(t, ts.URL+"/api/sensors", &sensors)
|
||||||
|
if len(sensors) != 1 || sensors[0].ID != updated.Sensors[0].ID {
|
||||||
|
t.Fatalf("sensors after refresh=%+v", sensors)
|
||||||
|
}
|
||||||
|
if got := resp.Header.Get("X-Data-Version"); got != updated.DataVersion {
|
||||||
|
t.Errorf("X-Data-Version=%q, want %q", got, updated.DataVersion)
|
||||||
|
}
|
||||||
|
|
||||||
|
var series struct {
|
||||||
|
Points []model.Point `json:"points"`
|
||||||
|
}
|
||||||
|
getJSON(t, ts.URL+"/api/series?sensor="+updated.Sensors[0].ID+"&step=15m", &series)
|
||||||
|
if len(series.Points) != 1 || series.Points[0].V != 42 {
|
||||||
|
t.Fatalf("series after refresh=%+v", series.Points)
|
||||||
|
}
|
||||||
|
|
||||||
|
var ctx model.ContextResponse
|
||||||
|
getJSON(t, ts.URL+"/api/context", &ctx)
|
||||||
|
if ctx.UpdatedAt.IsZero() || ctx.Cached {
|
||||||
|
t.Fatalf("context after refresh=%+v", ctx)
|
||||||
|
}
|
||||||
|
|
||||||
|
var meta model.Meta
|
||||||
|
getJSON(t, ts.URL+"/api/meta", &meta)
|
||||||
|
if meta.DataVersion != updated.DataVersion || meta.Cache.Failures != 0 {
|
||||||
|
t.Fatalf("meta after refresh=%+v", meta)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestSeriesRangeStepAndErrors(t *testing.T) {
|
func TestSeriesRangeStepAndErrors(t *testing.T) {
|
||||||
_, ts := newTestServer(t, &stubML{}, nil)
|
_, ts := newTestServer(t, &stubML{}, nil)
|
||||||
|
|
||||||
@@ -241,37 +449,96 @@ func TestContextEndpoint(t *testing.T) {
|
|||||||
if ctx.UpdatedAt.IsZero() {
|
if ctx.UpdatedAt.IsZero() {
|
||||||
t.Error("expected updated_at")
|
t.Error("expected updated_at")
|
||||||
}
|
}
|
||||||
|
if ctx.Cached {
|
||||||
|
t.Error("cached=true while the last snapshot refresh succeeded, want false")
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestForecastFallsBackWhenMLIsDown(t *testing.T) {
|
func TestForecastServesLastGoodWhenMLIsDown(t *testing.T) {
|
||||||
_, ts := newTestServer(t, &stubML{}, nil)
|
live := forecastPoints(168)
|
||||||
|
var down bool
|
||||||
|
var mu sync.Mutex
|
||||||
|
client := &stubML{predict: func(context.Context, ml.PredictRequest) ([]model.ForecastPoint, error) {
|
||||||
|
mu.Lock()
|
||||||
|
defer mu.Unlock()
|
||||||
|
if down {
|
||||||
|
return nil, errors.New("connection refused")
|
||||||
|
}
|
||||||
|
return live, nil
|
||||||
|
}}
|
||||||
|
_, ts := newTestServer(t, client, nil)
|
||||||
|
|
||||||
var forecast model.ForecastResponse
|
var forecast model.ForecastResponse
|
||||||
resp := getJSON(t, ts.URL+"/api/forecast?sensor=hn-bahnhof-01&horizon=7d", &forecast)
|
resp := getJSON(t, ts.URL+"/api/forecast?sensor=hn-bahnhof-01&horizon=7d", &forecast)
|
||||||
if resp.StatusCode != http.StatusOK {
|
if resp.StatusCode != http.StatusOK {
|
||||||
t.Fatalf("forecast status=%d", resp.StatusCode)
|
t.Fatalf("live forecast status=%d", resp.StatusCode)
|
||||||
}
|
}
|
||||||
if !forecast.Cached {
|
if forecast.Cached || len(forecast.Points) != 168 {
|
||||||
|
t.Fatalf("live forecast cached=%v points=%d, want false/168", forecast.Cached, len(forecast.Points))
|
||||||
|
}
|
||||||
|
|
||||||
|
mu.Lock()
|
||||||
|
down = true
|
||||||
|
mu.Unlock()
|
||||||
|
|
||||||
|
var fallback model.ForecastResponse
|
||||||
|
resp = getJSON(t, ts.URL+"/api/forecast?sensor=hn-bahnhof-01&horizon=7d", &fallback)
|
||||||
|
if resp.StatusCode != http.StatusOK {
|
||||||
|
t.Fatalf("fallback status=%d", resp.StatusCode)
|
||||||
|
}
|
||||||
|
if !fallback.Cached {
|
||||||
t.Error("cached=false, want true when FastAPI is down")
|
t.Error("cached=false, want true when FastAPI is down")
|
||||||
}
|
}
|
||||||
if len(forecast.Points) != 168 {
|
if len(fallback.Points) != 168 || fallback.Points[0].T != forecast.Points[0].T ||
|
||||||
t.Fatalf("points=%d, want 168", len(forecast.Points))
|
fallback.Points[0].V != forecast.Points[0].V {
|
||||||
|
t.Fatalf("fallback points do not match the last good forecast: %+v", fallback.Points[0])
|
||||||
}
|
}
|
||||||
if forecast.Scenario != "base" || forecast.DataVersion == "" || forecast.ModelVersion == "" {
|
if fallback.Scenario != "base" || fallback.DataVersion == "" || fallback.ModelVersion == "" {
|
||||||
t.Fatalf("forecast=%+v", forecast)
|
t.Fatalf("forecast=%+v", fallback)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestForecastTimeoutFallsBack(t *testing.T) {
|
func TestForecastWithoutLastGoodFails503(t *testing.T) {
|
||||||
|
_, ts := newTestServer(t, &stubML{}, nil)
|
||||||
|
var errBody model.ErrorBody
|
||||||
|
resp := getJSON(t, ts.URL+"/api/forecast?sensor=hn-bahnhof-01&horizon=7d", &errBody)
|
||||||
|
if resp.StatusCode != http.StatusServiceUnavailable {
|
||||||
|
t.Fatalf("status=%d, want 503", resp.StatusCode)
|
||||||
|
}
|
||||||
|
if errBody.Error.Code != "forecast_unavailable" {
|
||||||
|
t.Fatalf("error=%+v", errBody)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestForecastTimeoutServesLastGood(t *testing.T) {
|
||||||
|
live := forecastPoints(48)
|
||||||
|
var mu sync.Mutex
|
||||||
|
slow := false
|
||||||
client := &stubML{predict: func(ctx context.Context, _ ml.PredictRequest) ([]model.ForecastPoint, error) {
|
client := &stubML{predict: func(ctx context.Context, _ ml.PredictRequest) ([]model.ForecastPoint, error) {
|
||||||
|
mu.Lock()
|
||||||
|
block := slow
|
||||||
|
mu.Unlock()
|
||||||
|
if block {
|
||||||
<-ctx.Done()
|
<-ctx.Done()
|
||||||
return nil, ctx.Err()
|
return nil, ctx.Err()
|
||||||
|
}
|
||||||
|
return live, nil
|
||||||
}}
|
}}
|
||||||
_, ts := newTestServer(t, client, func(cfg *config.Config) { cfg.MLTimeout = 20 * time.Millisecond })
|
_, ts := newTestServer(t, client, func(cfg *config.Config) { cfg.MLTimeout = 20 * time.Millisecond })
|
||||||
|
|
||||||
|
var primed model.ForecastResponse
|
||||||
|
resp := getJSON(t, ts.URL+"/api/forecast?sensor=hn-stadtgalerie-01&horizon=48h&scenario=sunny-saturday-market", &primed)
|
||||||
|
if resp.StatusCode != http.StatusOK || primed.Cached {
|
||||||
|
t.Fatalf("prime forecast status=%d cached=%v", resp.StatusCode, primed.Cached)
|
||||||
|
}
|
||||||
|
|
||||||
|
mu.Lock()
|
||||||
|
slow = true
|
||||||
|
mu.Unlock()
|
||||||
|
|
||||||
start := time.Now()
|
start := time.Now()
|
||||||
var forecast model.ForecastResponse
|
var forecast model.ForecastResponse
|
||||||
resp := getJSON(t, ts.URL+"/api/forecast?sensor=hn-stadtgalerie-01&horizon=48h&scenario=sunny-saturday-market", &forecast)
|
resp = getJSON(t, ts.URL+"/api/forecast?sensor=hn-stadtgalerie-01&horizon=48h&scenario=sunny-saturday-market", &forecast)
|
||||||
if resp.StatusCode != http.StatusOK {
|
if resp.StatusCode != http.StatusOK {
|
||||||
t.Fatalf("forecast status=%d", resp.StatusCode)
|
t.Fatalf("forecast status=%d", resp.StatusCode)
|
||||||
}
|
}
|
||||||
@@ -327,7 +594,11 @@ func TestForecastValidation(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestStaffingPlan(t *testing.T) {
|
func TestStaffingPlan(t *testing.T) {
|
||||||
_, ts := newTestServer(t, &stubML{}, nil)
|
points := weekForecast(t)
|
||||||
|
client := &stubML{predict: func(context.Context, ml.PredictRequest) ([]model.ForecastPoint, error) {
|
||||||
|
return points, nil
|
||||||
|
}}
|
||||||
|
_, ts := newTestServer(t, client, nil)
|
||||||
|
|
||||||
start := time.Now()
|
start := time.Now()
|
||||||
var plan model.StaffingResponse
|
var plan model.StaffingResponse
|
||||||
@@ -350,9 +621,12 @@ func TestStaffingPlan(t *testing.T) {
|
|||||||
if plan.Summary.CoveragePct != 100 {
|
if plan.Summary.CoveragePct != 100 {
|
||||||
t.Errorf("coverage=%v, want 100", plan.Summary.CoveragePct)
|
t.Errorf("coverage=%v, want 100", plan.Summary.CoveragePct)
|
||||||
}
|
}
|
||||||
if plan.Summary.CostEUR <= 0 || plan.Summary.FlatScheduleCostEUR <= 0 {
|
if plan.Summary.CostEUR != 2240 || plan.Summary.FlatScheduleCostEUR != 2240 {
|
||||||
t.Errorf("summary=%+v", plan.Summary)
|
t.Errorf("summary=%+v", plan.Summary)
|
||||||
}
|
}
|
||||||
|
if plan.Cached {
|
||||||
|
t.Error("cached=true, want false for a live plan")
|
||||||
|
}
|
||||||
if len(plan.Why) == 0 {
|
if len(plan.Why) == 0 {
|
||||||
t.Error("expected a non-empty why list")
|
t.Error("expected a non-empty why list")
|
||||||
}
|
}
|
||||||
@@ -363,6 +637,57 @@ func TestStaffingPlan(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestStaffingFallsBackToLastGoodForecast(t *testing.T) {
|
||||||
|
points := weekForecast(t)
|
||||||
|
var mu sync.Mutex
|
||||||
|
down := false
|
||||||
|
client := &stubML{predict: func(context.Context, ml.PredictRequest) ([]model.ForecastPoint, error) {
|
||||||
|
mu.Lock()
|
||||||
|
defer mu.Unlock()
|
||||||
|
if down {
|
||||||
|
return nil, errors.New("connection refused")
|
||||||
|
}
|
||||||
|
return points, nil
|
||||||
|
}}
|
||||||
|
_, ts := newTestServer(t, client, nil)
|
||||||
|
|
||||||
|
var live model.StaffingResponse
|
||||||
|
resp := getJSON(t, ts.URL+"/api/staffing?sensor=hn-bahnhof-01&scenario=base&standard=standard", &live)
|
||||||
|
if resp.StatusCode != http.StatusOK || live.Cached {
|
||||||
|
t.Fatalf("live staffing status=%d cached=%v", resp.StatusCode, live.Cached)
|
||||||
|
}
|
||||||
|
|
||||||
|
mu.Lock()
|
||||||
|
down = true
|
||||||
|
mu.Unlock()
|
||||||
|
|
||||||
|
// Different standard -> different plan key, so the handler must rebuild
|
||||||
|
// from the last good 7d forecast.
|
||||||
|
var fallback model.StaffingResponse
|
||||||
|
resp = getJSON(t, ts.URL+"/api/staffing?sensor=hn-bahnhof-01&scenario=base&standard=premium", &fallback)
|
||||||
|
if resp.StatusCode != http.StatusOK {
|
||||||
|
t.Fatalf("fallback staffing status=%d", resp.StatusCode)
|
||||||
|
}
|
||||||
|
if !fallback.Cached {
|
||||||
|
t.Error("cached=false, want true when built from the last good forecast")
|
||||||
|
}
|
||||||
|
if len(fallback.Hours) != 70 || len(fallback.Shifts) != 21 {
|
||||||
|
t.Fatalf("fallback plan hours=%d shifts=%d", len(fallback.Hours), len(fallback.Shifts))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestStaffingUnavailableWithoutForecast(t *testing.T) {
|
||||||
|
_, ts := newTestServer(t, &stubML{}, nil)
|
||||||
|
var errBody model.ErrorBody
|
||||||
|
resp := getJSON(t, ts.URL+"/api/staffing?sensor=hn-bahnhof-01&scenario=base&standard=standard", &errBody)
|
||||||
|
if resp.StatusCode != http.StatusServiceUnavailable {
|
||||||
|
t.Fatalf("status=%d, want 503", resp.StatusCode)
|
||||||
|
}
|
||||||
|
if errBody.Error.Code != "staffing_unavailable" {
|
||||||
|
t.Fatalf("error=%+v", errBody)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestStaffingValidation(t *testing.T) {
|
func TestStaffingValidation(t *testing.T) {
|
||||||
_, ts := newTestServer(t, &stubML{}, nil)
|
_, ts := newTestServer(t, &stubML{}, nil)
|
||||||
for _, tc := range []struct {
|
for _, tc := range []struct {
|
||||||
|
|||||||
@@ -14,9 +14,11 @@ func (s *Server) handleHealthz(w http.ResponseWriter, _ *http.Request) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (s *Server) handleReadyz(w http.ResponseWriter, _ *http.Request) {
|
func (s *Server) handleReadyz(w http.ResponseWriter, _ *http.Request) {
|
||||||
ready := s.store != nil && s.store.Ready()
|
dataOK, modelOK := false, false
|
||||||
dataOK := s.store != nil && s.store.Len() > 0 && !s.store.Context().UpdatedAt.IsZero()
|
if s.cache != nil {
|
||||||
modelOK := s.store != nil && s.store.Meta().ModelVersion != ""
|
dataOK, modelOK = s.cache.Readiness()
|
||||||
|
}
|
||||||
|
ready := dataOK && modelOK
|
||||||
status := http.StatusOK
|
status := http.StatusOK
|
||||||
if !ready {
|
if !ready {
|
||||||
status = http.StatusServiceUnavailable
|
status = http.StatusServiceUnavailable
|
||||||
@@ -26,12 +28,16 @@ func (s *Server) handleReadyz(w http.ResponseWriter, _ *http.Request) {
|
|||||||
|
|
||||||
func (s *Server) handleMeta(w http.ResponseWriter, _ *http.Request) {
|
func (s *Server) handleMeta(w http.ResponseWriter, _ *http.Request) {
|
||||||
s.setVersions(w)
|
s.setVersions(w)
|
||||||
writeJSON(w, http.StatusOK, s.store.Meta())
|
writeJSON(w, http.StatusOK, s.meta())
|
||||||
}
|
}
|
||||||
|
|
||||||
func (s *Server) handleSensors(w http.ResponseWriter, _ *http.Request) {
|
func (s *Server) handleSensors(w http.ResponseWriter, _ *http.Request) {
|
||||||
s.setVersions(w)
|
s.setVersions(w)
|
||||||
writeJSON(w, http.StatusOK, s.store.Sensors())
|
if s.cache == nil {
|
||||||
|
writeJSON(w, http.StatusOK, []model.Sensor{})
|
||||||
|
return
|
||||||
|
}
|
||||||
|
writeJSON(w, http.StatusOK, s.cache.Sensors())
|
||||||
}
|
}
|
||||||
|
|
||||||
func (s *Server) handleSeries(w http.ResponseWriter, r *http.Request) {
|
func (s *Server) handleSeries(w http.ResponseWriter, r *http.Request) {
|
||||||
@@ -41,7 +47,11 @@ func (s *Server) handleSeries(w http.ResponseWriter, r *http.Request) {
|
|||||||
writeError(w, http.StatusBadRequest, "bad_request", "sensor is required")
|
writeError(w, http.StatusBadRequest, "bad_request", "sensor is required")
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
if _, ok := s.store.Sensor(sensor); !ok {
|
if s.cache == nil {
|
||||||
|
writeError(w, http.StatusNotFound, "not_found", "unknown sensor")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if _, ok := s.cache.Sensor(sensor); !ok {
|
||||||
writeError(w, http.StatusNotFound, "not_found", "unknown sensor")
|
writeError(w, http.StatusNotFound, "not_found", "unknown sensor")
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
@@ -54,7 +64,7 @@ func (s *Server) handleSeries(w http.ResponseWriter, r *http.Request) {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
raw, _ := s.store.Series(sensor)
|
raw, _ := s.cache.Series(sensor)
|
||||||
if len(raw) == 0 {
|
if len(raw) == 0 {
|
||||||
s.setVersions(w)
|
s.setVersions(w)
|
||||||
writeJSON(w, http.StatusOK, map[string]any{"sensor": sensor, "step": step, "points": []model.Point{}})
|
writeJSON(w, http.StatusOK, map[string]any{"sensor": sensor, "step": step, "points": []model.Point{}})
|
||||||
@@ -69,7 +79,7 @@ func (s *Server) handleSeries(w http.ResponseWriter, r *http.Request) {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
to = parsed.UTC()
|
to = parsed.UTC()
|
||||||
} else if len(raw) > 0 {
|
} else {
|
||||||
to = raw[len(raw)-1].T.UTC()
|
to = raw[len(raw)-1].T.UTC()
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -143,12 +153,18 @@ func hourlyMean(points []model.Point) []model.Point {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (s *Server) handleContext(w http.ResponseWriter, _ *http.Request) {
|
func (s *Server) handleContext(w http.ResponseWriter, _ *http.Request) {
|
||||||
ctx := s.store.Context()
|
var ctx model.ContextData
|
||||||
|
cached := true
|
||||||
|
if s.cache != nil {
|
||||||
|
ctx = s.cache.Context()
|
||||||
|
// A failed refresh keeps the old context in place, so mark it cached.
|
||||||
|
cached = !s.cache.Fresh()
|
||||||
|
}
|
||||||
s.setVersions(w)
|
s.setVersions(w)
|
||||||
writeJSON(w, http.StatusOK, model.ContextResponse{
|
writeJSON(w, http.StatusOK, model.ContextResponse{
|
||||||
Weather: ctx.Weather,
|
Weather: ctx.Weather,
|
||||||
Holidays: ctx.Holidays,
|
Holidays: ctx.Holidays,
|
||||||
UpdatedAt: ctx.UpdatedAt,
|
UpdatedAt: ctx.UpdatedAt,
|
||||||
Cached: false,
|
Cached: cached,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
@@ -20,7 +20,11 @@ func (s *Server) handleForecast(w http.ResponseWriter, r *http.Request) {
|
|||||||
writeError(w, http.StatusBadRequest, "bad_request", "sensor is required")
|
writeError(w, http.StatusBadRequest, "bad_request", "sensor is required")
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
if _, ok := s.store.Sensor(sensor); !ok {
|
if s.cache == nil {
|
||||||
|
writeError(w, http.StatusNotFound, "not_found", "unknown sensor")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if _, ok := s.cache.Sensor(sensor); !ok {
|
||||||
writeError(w, http.StatusNotFound, "not_found", "unknown sensor")
|
writeError(w, http.StatusNotFound, "not_found", "unknown sensor")
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
@@ -53,11 +57,14 @@ func (s *Server) handleForecast(w http.ResponseWriter, r *http.Request) {
|
|||||||
if !ok {
|
if !ok {
|
||||||
slog.Warn("forecast unavailable", "sensor", sensor, "horizon", horizon, "scenario", scenarioName, "error", err)
|
slog.Warn("forecast unavailable", "sensor", sensor, "horizon", horizon, "scenario", scenarioName, "error", err)
|
||||||
writeError(w, http.StatusServiceUnavailable, "forecast_unavailable",
|
writeError(w, http.StatusServiceUnavailable, "forecast_unavailable",
|
||||||
"no live forecast and no baked forecast for this sensor, scenario and horizon")
|
"no live forecast and no last good forecast for this sensor, scenario and horizon")
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
} else {
|
||||||
|
s.rememberForecast(sensor, horizon, scenarioName, points)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
meta := s.meta()
|
||||||
s.setVersions(w)
|
s.setVersions(w)
|
||||||
writeJSON(w, http.StatusOK, model.ForecastResponse{
|
writeJSON(w, http.StatusOK, model.ForecastResponse{
|
||||||
Sensor: sensor,
|
Sensor: sensor,
|
||||||
@@ -65,9 +72,9 @@ func (s *Server) handleForecast(w http.ResponseWriter, r *http.Request) {
|
|||||||
Scenario: scenarioName,
|
Scenario: scenarioName,
|
||||||
Points: points,
|
Points: points,
|
||||||
Cached: cached,
|
Cached: cached,
|
||||||
DataVersion: s.store.Meta().DataVersion,
|
DataVersion: meta.DataVersion,
|
||||||
ModelVersion: s.store.Meta().ModelVersion,
|
ModelVersion: meta.ModelVersion,
|
||||||
ContextVersion: s.store.Meta().ContextVersion,
|
ContextVersion: meta.ContextVersion,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -78,8 +85,9 @@ func (s *Server) liveForecast(ctx context.Context, sensor, horizon, scenarioName
|
|||||||
ctx, cancel := context.WithTimeout(ctx, s.cfg.MLTimeout)
|
ctx, cancel := context.WithTimeout(ctx, s.cfg.MLTimeout)
|
||||||
defer cancel()
|
defer cancel()
|
||||||
|
|
||||||
ctxData := s.store.Context()
|
ctxData := s.cache.Context()
|
||||||
precip := ctxData.Weather.Now.PrecipMM
|
precip := ctxData.Weather.Now.PrecipMM
|
||||||
|
// A zero override means "no scenario rain"; keep the cached current value.
|
||||||
if state.Scenario.Weather.PrecipMM > 0 {
|
if state.Scenario.Weather.PrecipMM > 0 {
|
||||||
precip = state.Scenario.Weather.PrecipMM
|
precip = state.Scenario.Weather.PrecipMM
|
||||||
}
|
}
|
||||||
@@ -100,8 +108,9 @@ func (s *Server) liveForecast(ctx context.Context, sensor, horizon, scenarioName
|
|||||||
return sortedForecast(points), nil
|
return sortedForecast(points), nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// fallbackForecast prefers live /simulate output for the current scenario, then
|
// fallbackForecast prefers the live /simulate grid for the current scenario,
|
||||||
// the baked scenario grid. It never labels base data as another scenario.
|
// then the last good /predict response for this exact key. It never labels base
|
||||||
|
// data as another scenario.
|
||||||
func (s *Server) fallbackForecast(sensor, scenarioName, horizon string) ([]model.ForecastPoint, bool) {
|
func (s *Server) fallbackForecast(sensor, scenarioName, horizon string) ([]model.ForecastPoint, bool) {
|
||||||
s.mu.RLock()
|
s.mu.RLock()
|
||||||
if s.simFor == scenarioName {
|
if s.simFor == scenarioName {
|
||||||
@@ -115,10 +124,31 @@ func (s *Server) fallbackForecast(sensor, scenarioName, horizon string) ([]model
|
|||||||
}
|
}
|
||||||
s.mu.RUnlock()
|
s.mu.RUnlock()
|
||||||
|
|
||||||
if points, ok := s.store.Forecast(sensor, scenarioName, horizon); ok {
|
return s.lastGoodForecast(sensor, horizon, scenarioName)
|
||||||
return sortedForecast(points), true
|
}
|
||||||
|
|
||||||
|
func forecastKey(sensor, horizon, scenarioName string) string {
|
||||||
|
return sensor + "|" + horizon + "|" + scenarioName
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Server) rememberForecast(sensor, horizon, scenarioName string, points []model.ForecastPoint) {
|
||||||
|
if len(points) == 0 {
|
||||||
|
return
|
||||||
}
|
}
|
||||||
|
copied := sortedForecast(points)
|
||||||
|
s.mu.Lock()
|
||||||
|
s.lastForecast[forecastKey(sensor, horizon, scenarioName)] = copied
|
||||||
|
s.mu.Unlock()
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Server) lastGoodForecast(sensor, horizon, scenarioName string) ([]model.ForecastPoint, bool) {
|
||||||
|
s.mu.RLock()
|
||||||
|
points, ok := s.lastForecast[forecastKey(sensor, horizon, scenarioName)]
|
||||||
|
s.mu.RUnlock()
|
||||||
|
if !ok || len(points) == 0 {
|
||||||
return nil, false
|
return nil, false
|
||||||
|
}
|
||||||
|
return sortedForecast(points), true
|
||||||
}
|
}
|
||||||
|
|
||||||
func sortedForecast(points []model.ForecastPoint) []model.ForecastPoint {
|
func sortedForecast(points []model.ForecastPoint) []model.ForecastPoint {
|
||||||
@@ -129,8 +159,11 @@ func sortedForecast(points []model.ForecastPoint) []model.ForecastPoint {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (s *Server) holidayOn(t time.Time) (string, bool) {
|
func (s *Server) holidayOn(t time.Time) (string, bool) {
|
||||||
|
if s.cache == nil {
|
||||||
|
return "", false
|
||||||
|
}
|
||||||
day := t.In(s.loc).Format("2006-01-02")
|
day := t.In(s.loc).Format("2006-01-02")
|
||||||
for _, holiday := range s.store.Context().Holidays {
|
for _, holiday := range s.cache.Context().Holidays {
|
||||||
if holiday.Date == day {
|
if holiday.Date == day {
|
||||||
return holiday.Name, true
|
return holiday.Name, true
|
||||||
}
|
}
|
||||||
@@ -145,7 +178,11 @@ func (s *Server) handleStaffing(w http.ResponseWriter, r *http.Request) {
|
|||||||
writeError(w, http.StatusBadRequest, "bad_request", "sensor is required")
|
writeError(w, http.StatusBadRequest, "bad_request", "sensor is required")
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
if _, ok := s.store.Sensor(sensor); !ok {
|
if s.cache == nil {
|
||||||
|
writeError(w, http.StatusNotFound, "not_found", "unknown sensor")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if _, ok := s.cache.Sensor(sensor); !ok {
|
||||||
writeError(w, http.StatusNotFound, "not_found", "unknown sensor")
|
writeError(w, http.StatusNotFound, "not_found", "unknown sensor")
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
@@ -169,45 +206,49 @@ func (s *Server) handleStaffing(w http.ResponseWriter, r *http.Request) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
state := s.scenario.Get()
|
state := s.scenario.Get()
|
||||||
points, cached, ok := s.staffingForecast(sensor, scenarioName)
|
meta := s.meta()
|
||||||
if !ok {
|
|
||||||
writeError(w, http.StatusServiceUnavailable, "staffing_unavailable",
|
|
||||||
"no 7d forecast for this sensor and scenario")
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
s.mu.RLock()
|
s.mu.RLock()
|
||||||
generation := s.simGen
|
generation := s.simGen
|
||||||
s.mu.RUnlock()
|
s.mu.RUnlock()
|
||||||
key := fmt.Sprintf("%s|%s|%s|%d|%d", sensor, scenarioName, standard, state.Version, generation)
|
key := fmt.Sprintf("%s|%s|%s|%s|%d|%d", sensor, scenarioName, standard,
|
||||||
|
meta.ContextVersion, state.Version, generation)
|
||||||
|
|
||||||
s.mu.RLock()
|
s.mu.RLock()
|
||||||
plan, hit := s.staffing[key]
|
plan, hit := s.staffing[key]
|
||||||
s.mu.RUnlock()
|
s.mu.RUnlock()
|
||||||
|
|
||||||
if !hit {
|
if !hit {
|
||||||
built, err := staffing.Build(sensor, scenarioName, standard, points, s.store.Context())
|
points, cached, ok := s.staffingForecast(r.Context(), sensor, scenarioName, state)
|
||||||
|
if !ok {
|
||||||
|
writeError(w, http.StatusServiceUnavailable, "staffing_unavailable",
|
||||||
|
"no live or last good 7d forecast for this sensor and scenario")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
built, err := staffing.Build(sensor, scenarioName, standard, points, s.cache.Context())
|
||||||
if err != nil {
|
if err != nil {
|
||||||
slog.Warn("staffing build failed", "sensor", sensor, "scenario", scenarioName, "standard", standard, "error", err)
|
slog.Warn("staffing build failed", "sensor", sensor, "scenario", scenarioName, "standard", standard, "error", err)
|
||||||
writeError(w, http.StatusServiceUnavailable, "staffing_unavailable", "could not build a plan from the available forecast")
|
writeError(w, http.StatusServiceUnavailable, "staffing_unavailable", "could not build a plan from the available forecast")
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
built.Cached = cached
|
||||||
s.mu.Lock()
|
s.mu.Lock()
|
||||||
s.staffing[key] = built
|
s.staffing[key] = built
|
||||||
s.mu.Unlock()
|
s.mu.Unlock()
|
||||||
plan = built
|
plan = built
|
||||||
}
|
}
|
||||||
|
|
||||||
plan.Cached = cached
|
plan.DataVersion = meta.DataVersion
|
||||||
plan.DataVersion = s.store.Meta().DataVersion
|
plan.ModelVersion = meta.ModelVersion
|
||||||
plan.ModelVersion = s.store.Meta().ModelVersion
|
plan.ContextVersion = meta.ContextVersion
|
||||||
plan.ContextVersion = s.store.Meta().ContextVersion
|
|
||||||
s.setVersions(w)
|
s.setVersions(w)
|
||||||
writeJSON(w, http.StatusOK, plan)
|
writeJSON(w, http.StatusOK, plan)
|
||||||
}
|
}
|
||||||
|
|
||||||
// staffingForecast returns the best 7d points and whether they came from the
|
// staffingForecast returns 7d points for a plan. Prefer the scenario simulation
|
||||||
// fallback grid rather than live /simulate output.
|
// grid, then /predict, and finally the last good forecast. The bool reports
|
||||||
func (s *Server) staffingForecast(sensor, scenarioName string) ([]model.ForecastPoint, bool, bool) {
|
// whether the points came from the cache fallback.
|
||||||
|
func (s *Server) staffingForecast(ctx context.Context, sensor, scenarioName string, state model.ScenarioState) ([]model.ForecastPoint, bool, bool) {
|
||||||
s.mu.RLock()
|
s.mu.RLock()
|
||||||
if s.simFor == scenarioName {
|
if s.simFor == scenarioName {
|
||||||
if horizons, ok := s.sim[sensor]; ok {
|
if horizons, ok := s.sim[sensor]; ok {
|
||||||
@@ -220,8 +261,14 @@ func (s *Server) staffingForecast(sensor, scenarioName string) ([]model.Forecast
|
|||||||
}
|
}
|
||||||
s.mu.RUnlock()
|
s.mu.RUnlock()
|
||||||
|
|
||||||
if points, ok := s.store.Forecast(sensor, scenarioName, "7d"); ok {
|
points, err := s.liveForecast(ctx, sensor, "7d", scenarioName, state)
|
||||||
return sortedForecast(points), scenarioName != "base", true
|
if err == nil {
|
||||||
|
s.rememberForecast(sensor, "7d", scenarioName, points)
|
||||||
|
return points, false, true
|
||||||
|
}
|
||||||
|
slog.Warn("staffing forecast fallback", "sensor", sensor, "scenario", scenarioName, "error", err)
|
||||||
|
if points, ok := s.lastGoodForecast(sensor, "7d", scenarioName); ok {
|
||||||
|
return points, true, true
|
||||||
}
|
}
|
||||||
return nil, false, false
|
return nil, false, false
|
||||||
}
|
}
|
||||||
@@ -83,6 +83,14 @@ func (s *Server) refreshSimulation(state model.ScenarioState) {
|
|||||||
s.simFor = state.Scenario.Scenario
|
s.simFor = state.Scenario.Scenario
|
||||||
s.simGen++
|
s.simGen++
|
||||||
s.staffing = make(map[string]model.StaffingResponse)
|
s.staffing = make(map[string]model.StaffingResponse)
|
||||||
|
for sensorID, horizons := range forecasts {
|
||||||
|
for horizon, points := range horizons {
|
||||||
|
if len(points) == 0 {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
s.lastForecast[forecastKey(sensorID, horizon, state.Scenario.Scenario)] = sortedForecast(points)
|
||||||
|
}
|
||||||
|
}
|
||||||
s.mu.Unlock()
|
s.mu.Unlock()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -13,8 +13,8 @@ import (
|
|||||||
"pulsecast/api/internal/model"
|
"pulsecast/api/internal/model"
|
||||||
)
|
)
|
||||||
|
|
||||||
// maxTrainBody bounds an uploaded CSV. The raw dataset is ~18 MB; 64 MB leaves
|
// maxTrainBody bounds an uploaded CSV so a client cannot stream an unbounded
|
||||||
// room for a larger export without letting the API accept unbounded bodies.
|
// body through the proxy.
|
||||||
const maxTrainBody = 64 << 20
|
const maxTrainBody = 64 << 20
|
||||||
|
|
||||||
// handleTrain proxies POST /api/train to FastAPI. The request is queued there,
|
// handleTrain proxies POST /api/train to FastAPI. The request is queued there,
|
||||||
|
|||||||
+22
-9
@@ -1,5 +1,5 @@
|
|||||||
// Package api wires the HTTP handlers, scenario state, SSE hub and cache
|
// Package api wires the HTTP handlers, scenario state, SSE hub and cache
|
||||||
// fallback into the Go service described by CONTRACT.md.
|
// fallback into the Go service.
|
||||||
package api
|
package api
|
||||||
|
|
||||||
import (
|
import (
|
||||||
@@ -12,12 +12,12 @@ import (
|
|||||||
"sync"
|
"sync"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
|
"pulsecast/api/internal/cache"
|
||||||
"pulsecast/api/internal/config"
|
"pulsecast/api/internal/config"
|
||||||
"pulsecast/api/internal/ml"
|
"pulsecast/api/internal/ml"
|
||||||
"pulsecast/api/internal/model"
|
"pulsecast/api/internal/model"
|
||||||
"pulsecast/api/internal/scenario"
|
"pulsecast/api/internal/scenario"
|
||||||
"pulsecast/api/internal/sse"
|
"pulsecast/api/internal/sse"
|
||||||
"pulsecast/api/internal/store"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// MLClient is the FastAPI prediction surface the data handlers need.
|
// MLClient is the FastAPI prediction surface the data handlers need.
|
||||||
@@ -37,7 +37,7 @@ type TrainClient interface {
|
|||||||
// Server holds process-wide state. One instance per pod.
|
// Server holds process-wide state. One instance per pod.
|
||||||
type Server struct {
|
type Server struct {
|
||||||
cfg config.Config
|
cfg config.Config
|
||||||
store *store.Store
|
cache *cache.Cache
|
||||||
ml MLClient
|
ml MLClient
|
||||||
hub *sse.Hub
|
hub *sse.Hub
|
||||||
scenario *scenario.State
|
scenario *scenario.State
|
||||||
@@ -49,10 +49,11 @@ type Server struct {
|
|||||||
simFor string // scenario the sim belongs to
|
simFor string // scenario the sim belongs to
|
||||||
simGen int // invalidates staffing cache
|
simGen int // invalidates staffing cache
|
||||||
staffing map[string]model.StaffingResponse
|
staffing map[string]model.StaffingResponse
|
||||||
|
lastForecast map[string][]model.ForecastPoint // sensor|horizon|scenario -> last live points
|
||||||
}
|
}
|
||||||
|
|
||||||
// New builds a server. hub must be non-nil; main runs hub.Run.
|
// New builds a server. hub must be non-nil; main runs hub.Run.
|
||||||
func New(cfg config.Config, st *store.Store, client MLClient, hub *sse.Hub) *Server {
|
func New(cfg config.Config, snapshots *cache.Cache, client MLClient, hub *sse.Hub) *Server {
|
||||||
loc, err := time.LoadLocation(model.StoreHours.TZ)
|
loc, err := time.LoadLocation(model.StoreHours.TZ)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
loc = time.UTC
|
loc = time.UTC
|
||||||
@@ -62,13 +63,14 @@ func New(cfg config.Config, st *store.Store, client MLClient, hub *sse.Hub) *Ser
|
|||||||
}
|
}
|
||||||
return &Server{
|
return &Server{
|
||||||
cfg: cfg,
|
cfg: cfg,
|
||||||
store: st,
|
cache: snapshots,
|
||||||
ml: client,
|
ml: client,
|
||||||
hub: hub,
|
hub: hub,
|
||||||
scenario: scenario.New(),
|
scenario: scenario.New(),
|
||||||
loc: loc,
|
loc: loc,
|
||||||
now: time.Now,
|
now: time.Now,
|
||||||
staffing: make(map[string]model.StaffingResponse),
|
staffing: make(map[string]model.StaffingResponse),
|
||||||
|
lastForecast: make(map[string][]model.ForecastPoint),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -106,6 +108,8 @@ var methodTable = map[string][]string{
|
|||||||
"/api/stream": {"GET"},
|
"/api/stream": {"GET"},
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// routeCheck keeps unknown routes and methods in the JSON error envelope;
|
||||||
|
// ServeMux would otherwise answer them in plain text.
|
||||||
func (s *Server) routeCheck(next http.Handler) http.Handler {
|
func (s *Server) routeCheck(next http.Handler) http.Handler {
|
||||||
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
methods, ok := methodTable[r.URL.Path]
|
methods, ok := methodTable[r.URL.Path]
|
||||||
@@ -194,10 +198,19 @@ func writeError(w http.ResponseWriter, status int, code, message string) {
|
|||||||
_, _ = w.Write(raw)
|
_, _ = w.Write(raw)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// meta snapshots the cache metadata once per response, so versions and the
|
||||||
|
// cache block cannot disagree when a refresh lands mid-request.
|
||||||
|
func (s *Server) meta() model.Meta {
|
||||||
|
if s.cache == nil {
|
||||||
|
return model.Meta{}
|
||||||
|
}
|
||||||
|
return s.cache.Meta()
|
||||||
|
}
|
||||||
|
|
||||||
// setVersions adds the contract's version headers.
|
// setVersions adds the contract's version headers.
|
||||||
func (s *Server) setVersions(w http.ResponseWriter) {
|
func (s *Server) setVersions(w http.ResponseWriter) {
|
||||||
data, modelVersion, context := s.store.Versions()
|
meta := s.meta()
|
||||||
w.Header().Set("X-Data-Version", data)
|
w.Header().Set("X-Data-Version", meta.DataVersion)
|
||||||
w.Header().Set("X-Model-Version", modelVersion)
|
w.Header().Set("X-Model-Version", meta.ModelVersion)
|
||||||
w.Header().Set("X-Context-Version", context)
|
w.Header().Set("X-Context-Version", meta.ContextVersion)
|
||||||
}
|
}
|
||||||
Vendored
+354
@@ -0,0 +1,354 @@
|
|||||||
|
// Package cache keeps the FastAPI /bootstrap snapshot in memory and refreshes
|
||||||
|
// it in the background. A failed refresh never discards the last good snapshot;
|
||||||
|
// the next attempt follows the contract backoff 1/2/4/8/16/30 s (cap).
|
||||||
|
package cache
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"log/slog"
|
||||||
|
"sort"
|
||||||
|
"sync"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"pulsecast/api/internal/model"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Source is the only snapshot producer: the FastAPI /bootstrap endpoint.
|
||||||
|
const Source = "fastapi"
|
||||||
|
|
||||||
|
// DefaultRefreshInterval is how often a healthy snapshot is refreshed.
|
||||||
|
const DefaultRefreshInterval = 60 * time.Second
|
||||||
|
|
||||||
|
// DefaultTimeout bounds one /bootstrap request.
|
||||||
|
const DefaultTimeout = 20 * time.Second
|
||||||
|
|
||||||
|
// DefaultRetryDelays is the contract backoff schedule. The last value repeats.
|
||||||
|
var DefaultRetryDelays = []time.Duration{
|
||||||
|
1 * time.Second,
|
||||||
|
2 * time.Second,
|
||||||
|
4 * time.Second,
|
||||||
|
8 * time.Second,
|
||||||
|
16 * time.Second,
|
||||||
|
30 * time.Second,
|
||||||
|
}
|
||||||
|
|
||||||
|
// Fetcher is the FastAPI bootstrap surface. *ml.Client implements it.
|
||||||
|
type Fetcher interface {
|
||||||
|
Bootstrap(ctx context.Context) (model.Bootstrap, error)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Options configures refresh behaviour. Zero values use the defaults.
|
||||||
|
type Options struct {
|
||||||
|
RefreshInterval time.Duration
|
||||||
|
Timeout time.Duration
|
||||||
|
}
|
||||||
|
|
||||||
|
// snapshot is one immutable /bootstrap payload with its lookup indexes.
|
||||||
|
type snapshot struct {
|
||||||
|
DataVersion string
|
||||||
|
ModelVersion string
|
||||||
|
ContextVersion string
|
||||||
|
Sensors []model.Sensor
|
||||||
|
Series map[string][]model.Point
|
||||||
|
Context model.ContextData
|
||||||
|
Coverage model.Coverage
|
||||||
|
Metrics model.Metrics
|
||||||
|
Limits []string
|
||||||
|
|
||||||
|
sensorBy map[string]model.Sensor
|
||||||
|
}
|
||||||
|
|
||||||
|
// Cache holds the current snapshot and its refresh status. All methods are
|
||||||
|
// safe for concurrent use.
|
||||||
|
type Cache struct {
|
||||||
|
fetcher Fetcher
|
||||||
|
refresh time.Duration
|
||||||
|
timeout time.Duration
|
||||||
|
retry []time.Duration
|
||||||
|
now func() time.Time
|
||||||
|
mu sync.RWMutex
|
||||||
|
snapshot *snapshot
|
||||||
|
loaded bool
|
||||||
|
status model.CacheStatus
|
||||||
|
}
|
||||||
|
|
||||||
|
// New returns an empty cache. Call Run (or Refresh) to load the first snapshot.
|
||||||
|
func New(fetcher Fetcher, opts Options) *Cache {
|
||||||
|
if opts.RefreshInterval <= 0 {
|
||||||
|
opts.RefreshInterval = DefaultRefreshInterval
|
||||||
|
}
|
||||||
|
if opts.Timeout <= 0 {
|
||||||
|
opts.Timeout = DefaultTimeout
|
||||||
|
}
|
||||||
|
return &Cache{
|
||||||
|
fetcher: fetcher,
|
||||||
|
refresh: opts.RefreshInterval,
|
||||||
|
timeout: opts.Timeout,
|
||||||
|
retry: append([]time.Duration(nil), DefaultRetryDelays...),
|
||||||
|
now: time.Now,
|
||||||
|
status: model.CacheStatus{Source: Source},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Run loads the snapshot immediately, then refreshes it. Failures keep the last
|
||||||
|
// good snapshot and retry with the contract backoff. Run returns when ctx ends.
|
||||||
|
func (c *Cache) Run(ctx context.Context) {
|
||||||
|
recovering := false
|
||||||
|
for {
|
||||||
|
wasLoaded := c.Loaded()
|
||||||
|
err := c.Refresh(ctx)
|
||||||
|
if ctx.Err() != nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
var delay time.Duration
|
||||||
|
if err != nil {
|
||||||
|
recovering = true
|
||||||
|
status := c.Status()
|
||||||
|
delay = c.retryDelay(status.Failures)
|
||||||
|
slog.Warn("bootstrap refresh failed; keeping last good snapshot",
|
||||||
|
"error", err, "failures", status.Failures, "retry_in", delay)
|
||||||
|
} else {
|
||||||
|
delay = c.refresh
|
||||||
|
switch {
|
||||||
|
case !wasLoaded:
|
||||||
|
slog.Info("bootstrap snapshot loaded", "sensors", c.Len())
|
||||||
|
case recovering:
|
||||||
|
slog.Info("bootstrap refresh recovered", "sensors", c.Len())
|
||||||
|
}
|
||||||
|
recovering = false
|
||||||
|
}
|
||||||
|
|
||||||
|
timer := time.NewTimer(delay)
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
timer.Stop()
|
||||||
|
return
|
||||||
|
case <-timer.C:
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Refresh performs one fetch. A decode/validation failure counts as a failed
|
||||||
|
// attempt and leaves the previous snapshot untouched.
|
||||||
|
func (c *Cache) Refresh(ctx context.Context) error {
|
||||||
|
ctx, cancel := context.WithTimeout(ctx, c.timeout)
|
||||||
|
defer cancel()
|
||||||
|
|
||||||
|
payload, err := c.fetcher.Bootstrap(ctx)
|
||||||
|
now := c.now().UTC().Truncate(time.Second)
|
||||||
|
if err == nil {
|
||||||
|
var snapshot *snapshot
|
||||||
|
snapshot, err = newSnapshot(payload)
|
||||||
|
if err == nil {
|
||||||
|
c.mu.Lock()
|
||||||
|
c.snapshot = snapshot
|
||||||
|
c.loaded = true
|
||||||
|
c.status.LastSuccessAt = timePtr(now)
|
||||||
|
c.status.LastError = nil
|
||||||
|
c.status.NextRetryAt = nil
|
||||||
|
c.status.Failures = 0
|
||||||
|
c.mu.Unlock()
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
c.mu.Lock()
|
||||||
|
c.status.Failures++
|
||||||
|
message := err.Error()
|
||||||
|
c.status.LastError = &message
|
||||||
|
next := now.Add(c.retryDelay(c.status.Failures))
|
||||||
|
c.status.NextRetryAt = &next
|
||||||
|
c.mu.Unlock()
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *Cache) retryDelay(failures int) time.Duration {
|
||||||
|
if failures < 1 {
|
||||||
|
failures = 1
|
||||||
|
}
|
||||||
|
if failures > len(c.retry) {
|
||||||
|
failures = len(c.retry)
|
||||||
|
}
|
||||||
|
return c.retry[failures-1]
|
||||||
|
}
|
||||||
|
|
||||||
|
func newSnapshot(payload model.Bootstrap) (*snapshot, error) {
|
||||||
|
switch {
|
||||||
|
case len(payload.Sensors) == 0:
|
||||||
|
return nil, fmt.Errorf("bootstrap snapshot: no sensors")
|
||||||
|
case payload.DataVersion == "":
|
||||||
|
return nil, fmt.Errorf("bootstrap snapshot: missing data_version")
|
||||||
|
case payload.ModelVersion == "":
|
||||||
|
return nil, fmt.Errorf("bootstrap snapshot: missing model_version")
|
||||||
|
case payload.ContextVersion == "":
|
||||||
|
return nil, fmt.Errorf("bootstrap snapshot: missing context_version")
|
||||||
|
case payload.Context.UpdatedAt.IsZero():
|
||||||
|
return nil, fmt.Errorf("bootstrap snapshot: missing context.updated_at")
|
||||||
|
}
|
||||||
|
|
||||||
|
snapshot := &snapshot{
|
||||||
|
DataVersion: payload.DataVersion,
|
||||||
|
ModelVersion: payload.ModelVersion,
|
||||||
|
ContextVersion: payload.ContextVersion,
|
||||||
|
Sensors: append([]model.Sensor(nil), payload.Sensors...),
|
||||||
|
Series: make(map[string][]model.Point, len(payload.Series)),
|
||||||
|
Context: payload.Context,
|
||||||
|
Coverage: payload.Meta.Coverage,
|
||||||
|
Metrics: payload.Meta.Metrics,
|
||||||
|
Limits: append([]string(nil), payload.Meta.Limits...),
|
||||||
|
sensorBy: make(map[string]model.Sensor, len(payload.Sensors)),
|
||||||
|
}
|
||||||
|
sort.Slice(snapshot.Sensors, func(i, j int) bool { return snapshot.Sensors[i].ID < snapshot.Sensors[j].ID })
|
||||||
|
for _, sensor := range snapshot.Sensors {
|
||||||
|
snapshot.sensorBy[sensor.ID] = sensor
|
||||||
|
}
|
||||||
|
for sensorID, points := range payload.Series {
|
||||||
|
copied := append([]model.Point(nil), points...)
|
||||||
|
sort.Slice(copied, func(i, j int) bool { return copied[i].T.Before(copied[j].T) })
|
||||||
|
snapshot.Series[sensorID] = copied
|
||||||
|
}
|
||||||
|
snapshot.Context.Holidays = append([]model.Holiday(nil), payload.Context.Holidays...)
|
||||||
|
snapshot.Context.Weather.Forecast = append([]model.WeatherPoint(nil), payload.Context.Weather.Forecast...)
|
||||||
|
return snapshot, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// Loaded reports whether a snapshot has ever been accepted.
|
||||||
|
func (c *Cache) Loaded() bool {
|
||||||
|
c.mu.RLock()
|
||||||
|
defer c.mu.RUnlock()
|
||||||
|
return c.loaded
|
||||||
|
}
|
||||||
|
|
||||||
|
// Ready reports whether a complete snapshot is available. FastAPI health is
|
||||||
|
// deliberately not part of this check: the cache itself is the safety net.
|
||||||
|
func (c *Cache) Ready() bool {
|
||||||
|
data, modelOK := c.Readiness()
|
||||||
|
return data && modelOK
|
||||||
|
}
|
||||||
|
|
||||||
|
// Readiness splits readiness into the data and model signals of /api/readyz.
|
||||||
|
func (c *Cache) Readiness() (data, modelOK bool) {
|
||||||
|
c.mu.RLock()
|
||||||
|
defer c.mu.RUnlock()
|
||||||
|
if !c.loaded || c.snapshot == nil {
|
||||||
|
return false, false
|
||||||
|
}
|
||||||
|
data = len(c.snapshot.Sensors) > 0 && !c.snapshot.Context.UpdatedAt.IsZero()
|
||||||
|
modelOK = c.snapshot.ModelVersion != ""
|
||||||
|
return data, modelOK
|
||||||
|
}
|
||||||
|
|
||||||
|
// Fresh reports whether the most recent refresh attempt succeeded. It is false
|
||||||
|
// before the first snapshot and after a failed refresh.
|
||||||
|
func (c *Cache) Fresh() bool {
|
||||||
|
c.mu.RLock()
|
||||||
|
defer c.mu.RUnlock()
|
||||||
|
return c.loaded && c.status.Failures == 0
|
||||||
|
}
|
||||||
|
|
||||||
|
// Status returns a copy of the cache status.
|
||||||
|
func (c *Cache) Status() model.CacheStatus {
|
||||||
|
c.mu.RLock()
|
||||||
|
defer c.mu.RUnlock()
|
||||||
|
return cloneStatus(c.status)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Meta combines the snapshot metadata with the live cache status.
|
||||||
|
func (c *Cache) Meta() model.Meta {
|
||||||
|
c.mu.RLock()
|
||||||
|
defer c.mu.RUnlock()
|
||||||
|
|
||||||
|
meta := model.Meta{Cache: cloneStatus(c.status)}
|
||||||
|
if c.snapshot == nil {
|
||||||
|
return meta
|
||||||
|
}
|
||||||
|
meta.DataVersion = c.snapshot.DataVersion
|
||||||
|
meta.ModelVersion = c.snapshot.ModelVersion
|
||||||
|
meta.ContextVersion = c.snapshot.ContextVersion
|
||||||
|
meta.Coverage = c.snapshot.Coverage
|
||||||
|
meta.Metrics = c.snapshot.Metrics
|
||||||
|
meta.Limits = append([]string(nil), c.snapshot.Limits...)
|
||||||
|
return meta
|
||||||
|
}
|
||||||
|
|
||||||
|
// Sensors returns the sensors sorted by id.
|
||||||
|
func (c *Cache) Sensors() []model.Sensor {
|
||||||
|
c.mu.RLock()
|
||||||
|
snapshot := c.snapshot
|
||||||
|
c.mu.RUnlock()
|
||||||
|
if snapshot == nil {
|
||||||
|
return []model.Sensor{}
|
||||||
|
}
|
||||||
|
out := make([]model.Sensor, len(snapshot.Sensors))
|
||||||
|
copy(out, snapshot.Sensors)
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *Cache) Sensor(id string) (model.Sensor, bool) {
|
||||||
|
c.mu.RLock()
|
||||||
|
defer c.mu.RUnlock()
|
||||||
|
if c.snapshot == nil {
|
||||||
|
return model.Sensor{}, false
|
||||||
|
}
|
||||||
|
sensor, ok := c.snapshot.sensorBy[id]
|
||||||
|
return sensor, ok
|
||||||
|
}
|
||||||
|
|
||||||
|
// Series returns the immutable series for a sensor, sorted by time.
|
||||||
|
func (c *Cache) Series(id string) ([]model.Point, bool) {
|
||||||
|
c.mu.RLock()
|
||||||
|
defer c.mu.RUnlock()
|
||||||
|
if c.snapshot == nil {
|
||||||
|
return nil, false
|
||||||
|
}
|
||||||
|
points, ok := c.snapshot.Series[id]
|
||||||
|
return points, ok
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *Cache) Context() model.ContextData {
|
||||||
|
c.mu.RLock()
|
||||||
|
defer c.mu.RUnlock()
|
||||||
|
if c.snapshot == nil {
|
||||||
|
return model.ContextData{}
|
||||||
|
}
|
||||||
|
return c.snapshot.Context
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *Cache) Versions() (data, modelVersion, contextVersion string) {
|
||||||
|
c.mu.RLock()
|
||||||
|
defer c.mu.RUnlock()
|
||||||
|
if c.snapshot == nil {
|
||||||
|
return "", "", ""
|
||||||
|
}
|
||||||
|
return c.snapshot.DataVersion, c.snapshot.ModelVersion, c.snapshot.ContextVersion
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *Cache) Len() int {
|
||||||
|
c.mu.RLock()
|
||||||
|
defer c.mu.RUnlock()
|
||||||
|
if c.snapshot == nil {
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
return len(c.snapshot.Sensors)
|
||||||
|
}
|
||||||
|
|
||||||
|
func cloneStatus(s model.CacheStatus) model.CacheStatus {
|
||||||
|
out := s
|
||||||
|
if s.LastSuccessAt != nil {
|
||||||
|
v := *s.LastSuccessAt
|
||||||
|
out.LastSuccessAt = &v
|
||||||
|
}
|
||||||
|
if s.LastError != nil {
|
||||||
|
v := *s.LastError
|
||||||
|
out.LastError = &v
|
||||||
|
}
|
||||||
|
if s.NextRetryAt != nil {
|
||||||
|
v := *s.NextRetryAt
|
||||||
|
out.NextRetryAt = &v
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func timePtr(t time.Time) *time.Time { return &t }
|
||||||
Vendored
+295
@@ -0,0 +1,295 @@
|
|||||||
|
package cache
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"sync"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"pulsecast/api/internal/model"
|
||||||
|
)
|
||||||
|
|
||||||
|
type stubFetcher struct {
|
||||||
|
mu sync.Mutex
|
||||||
|
calls int
|
||||||
|
payload model.Bootstrap
|
||||||
|
err error
|
||||||
|
fn func(int) (model.Bootstrap, error)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *stubFetcher) Bootstrap(context.Context) (model.Bootstrap, error) {
|
||||||
|
s.mu.Lock()
|
||||||
|
s.calls++
|
||||||
|
call := s.calls
|
||||||
|
payload, err, fn := s.payload, s.err, s.fn
|
||||||
|
s.mu.Unlock()
|
||||||
|
if fn != nil {
|
||||||
|
return fn(call)
|
||||||
|
}
|
||||||
|
return payload, err
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *stubFetcher) set(payload model.Bootstrap, err error) {
|
||||||
|
s.mu.Lock()
|
||||||
|
s.payload, s.err = payload, err
|
||||||
|
s.mu.Unlock()
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *stubFetcher) callCount() int {
|
||||||
|
s.mu.Lock()
|
||||||
|
defer s.mu.Unlock()
|
||||||
|
return s.calls
|
||||||
|
}
|
||||||
|
|
||||||
|
func fixturePayload() model.Bootstrap {
|
||||||
|
t0 := time.Date(2026, 11, 14, 3, 0, 0, 0, time.UTC)
|
||||||
|
return model.Bootstrap{
|
||||||
|
DataVersion: "2026-11-07T23:51:00Z",
|
||||||
|
ModelVersion: "histgb-2026-11-14",
|
||||||
|
ContextVersion: "2026-11-14T03:00:00Z",
|
||||||
|
Sensors: []model.Sensor{
|
||||||
|
{ID: "hn-zebra-01", Name: "Zebra", Lat: 49.1, Lon: 9.2, Mean: 1, P95: 2},
|
||||||
|
{ID: "hn-alpha-01", Name: "Alpha", Lat: 49.1, Lon: 9.2, Mean: 3, P95: 4},
|
||||||
|
},
|
||||||
|
Series: map[string][]model.Point{
|
||||||
|
"hn-alpha-01": {
|
||||||
|
{T: t0.Add(2 * time.Hour), V: 3},
|
||||||
|
{T: t0, V: 1},
|
||||||
|
{T: t0.Add(time.Hour), V: 2},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
Context: model.ContextData{
|
||||||
|
Weather: model.Weather{
|
||||||
|
Now: model.WeatherNow{T: t0, TempC: 8.9, HumidityPct: 79},
|
||||||
|
Forecast: []model.WeatherPoint{{T: t0.Add(time.Hour), TempC: 8.4}},
|
||||||
|
},
|
||||||
|
Holidays: []model.Holiday{{Date: "2026-11-01", Name: "Allerheiligen"}},
|
||||||
|
UpdatedAt: t0,
|
||||||
|
},
|
||||||
|
Meta: model.BootstrapMeta{
|
||||||
|
Coverage: model.Coverage{Sensors: 8, From: "2026-06-12T00:00:00Z", To: "2026-11-07T23:51:00Z", CadenceMin: 15},
|
||||||
|
Metrics: model.Metrics{Model: "weather", MAE: 3.36, WAPE: 27.8, R2: 0.779},
|
||||||
|
Limits: []string{"estimated devices, not people"},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRetryDelaySchedule(t *testing.T) {
|
||||||
|
c := New(&stubFetcher{}, Options{})
|
||||||
|
want := []time.Duration{1 * time.Second, 2 * time.Second, 4 * time.Second, 8 * time.Second, 16 * time.Second, 30 * time.Second, 30 * time.Second}
|
||||||
|
for i, expected := range want {
|
||||||
|
if got := c.retryDelay(i + 1); got != expected {
|
||||||
|
t.Errorf("retryDelay(%d)=%s, want %s", i+1, got, expected)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for _, failures := range []int{0, -1} {
|
||||||
|
if got := c.retryDelay(failures); got != time.Second {
|
||||||
|
t.Errorf("retryDelay(%d)=%s, want 1s", failures, got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRefreshLoadsSnapshotAndStatus(t *testing.T) {
|
||||||
|
fetcher := &stubFetcher{payload: fixturePayload()}
|
||||||
|
c := New(fetcher, Options{RefreshInterval: time.Minute, Timeout: time.Second})
|
||||||
|
fixed := time.Date(2026, 11, 14, 4, 0, 0, 0, time.UTC)
|
||||||
|
c.now = func() time.Time { return fixed }
|
||||||
|
|
||||||
|
if err := c.Refresh(context.Background()); err != nil {
|
||||||
|
t.Fatalf("Refresh: %v", err)
|
||||||
|
}
|
||||||
|
if !c.Loaded() || !c.Ready() || !c.Fresh() {
|
||||||
|
t.Fatalf("loaded=%v ready=%v fresh=%v", c.Loaded(), c.Ready(), c.Fresh())
|
||||||
|
}
|
||||||
|
data, modelOK := c.Readiness()
|
||||||
|
if !data || !modelOK {
|
||||||
|
t.Fatalf("readiness data=%v model=%v", data, modelOK)
|
||||||
|
}
|
||||||
|
if c.Len() != 2 {
|
||||||
|
t.Fatalf("Len=%d, want 2", c.Len())
|
||||||
|
}
|
||||||
|
|
||||||
|
sensors := c.Sensors()
|
||||||
|
if len(sensors) != 2 || sensors[0].ID != "hn-alpha-01" || sensors[1].ID != "hn-zebra-01" {
|
||||||
|
t.Fatalf("sensors not sorted: %+v", sensors)
|
||||||
|
}
|
||||||
|
if sensor, ok := c.Sensor("hn-zebra-01"); !ok || sensor.Name != "Zebra" {
|
||||||
|
t.Fatalf("Sensor: ok=%v sensor=%+v", ok, sensor)
|
||||||
|
}
|
||||||
|
if _, ok := c.Sensor("nope"); ok {
|
||||||
|
t.Fatal("unexpected sensor nope")
|
||||||
|
}
|
||||||
|
series, ok := c.Series("hn-alpha-01")
|
||||||
|
if !ok || len(series) != 3 || !series[0].T.Before(series[1].T) || !series[1].T.Before(series[2].T) {
|
||||||
|
t.Fatalf("series not sorted: ok=%v points=%+v", ok, series)
|
||||||
|
}
|
||||||
|
if ctx := c.Context(); ctx.UpdatedAt.IsZero() || len(ctx.Holidays) != 1 || len(ctx.Weather.Forecast) != 1 {
|
||||||
|
t.Fatalf("context=%+v", ctx)
|
||||||
|
}
|
||||||
|
dataVersion, modelVersion, contextVersion := c.Versions()
|
||||||
|
if dataVersion != "2026-11-07T23:51:00Z" || modelVersion != "histgb-2026-11-14" || contextVersion != "2026-11-14T03:00:00Z" {
|
||||||
|
t.Fatalf("versions=%q %q %q", dataVersion, modelVersion, contextVersion)
|
||||||
|
}
|
||||||
|
|
||||||
|
meta := c.Meta()
|
||||||
|
if meta.Coverage.Sensors != 8 || meta.Metrics.MAE != 3.36 || len(meta.Limits) != 1 {
|
||||||
|
t.Fatalf("meta=%+v", meta)
|
||||||
|
}
|
||||||
|
if meta.Cache.Source != Source || meta.Cache.LastSuccessAt == nil ||
|
||||||
|
!meta.Cache.LastSuccessAt.Equal(fixed) || meta.Cache.LastError != nil ||
|
||||||
|
meta.Cache.NextRetryAt != nil || meta.Cache.Failures != 0 {
|
||||||
|
t.Fatalf("cache status=%+v", meta.Cache)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRefreshFailureKeepsLastGoodAndCountsBackoff(t *testing.T) {
|
||||||
|
fetcher := &stubFetcher{payload: fixturePayload()}
|
||||||
|
c := New(fetcher, Options{RefreshInterval: time.Minute, Timeout: time.Second})
|
||||||
|
fixed := time.Date(2026, 11, 14, 4, 0, 0, 0, time.UTC)
|
||||||
|
c.now = func() time.Time { return fixed }
|
||||||
|
|
||||||
|
if err := c.Refresh(context.Background()); err != nil {
|
||||||
|
t.Fatalf("prime: %v", err)
|
||||||
|
}
|
||||||
|
primed := c.Sensors()
|
||||||
|
|
||||||
|
fetcher.set(model.Bootstrap{}, errors.New("connection refused"))
|
||||||
|
if err := c.Refresh(context.Background()); err == nil {
|
||||||
|
t.Fatal("expected refresh failure")
|
||||||
|
}
|
||||||
|
if !c.Ready() || c.Fresh() {
|
||||||
|
t.Fatalf("ready=%v fresh=%v, want ready and not fresh", c.Ready(), c.Fresh())
|
||||||
|
}
|
||||||
|
if got := c.Sensors(); len(got) != len(primed) || got[0].ID != primed[0].ID {
|
||||||
|
t.Fatalf("last good snapshot lost: %+v", got)
|
||||||
|
}
|
||||||
|
status := c.Status()
|
||||||
|
if status.Failures != 1 || status.LastError == nil || *status.LastError != "connection refused" {
|
||||||
|
t.Fatalf("status=%+v", status)
|
||||||
|
}
|
||||||
|
if status.LastSuccessAt == nil || !status.LastSuccessAt.Equal(fixed) {
|
||||||
|
t.Fatalf("last success=%v", status.LastSuccessAt)
|
||||||
|
}
|
||||||
|
if status.NextRetryAt == nil || !status.NextRetryAt.Equal(fixed.Add(time.Second)) {
|
||||||
|
t.Fatalf("next retry=%v, want %s", status.NextRetryAt, fixed.Add(time.Second))
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := c.Refresh(context.Background()); err == nil {
|
||||||
|
t.Fatal("expected second refresh failure")
|
||||||
|
}
|
||||||
|
status = c.Status()
|
||||||
|
if status.Failures != 2 || status.NextRetryAt == nil || !status.NextRetryAt.Equal(fixed.Add(2*time.Second)) {
|
||||||
|
t.Fatalf("second failure status=%+v", status)
|
||||||
|
}
|
||||||
|
|
||||||
|
fetcher.set(fixturePayload(), nil)
|
||||||
|
if err := c.Refresh(context.Background()); err != nil {
|
||||||
|
t.Fatalf("recovery: %v", err)
|
||||||
|
}
|
||||||
|
status = c.Status()
|
||||||
|
if status.Failures != 0 || status.LastError != nil || status.NextRetryAt != nil || !c.Fresh() {
|
||||||
|
t.Fatalf("recovered status=%+v fresh=%v", status, c.Fresh())
|
||||||
|
}
|
||||||
|
if status.LastSuccessAt == nil || !status.LastSuccessAt.Equal(fixed) {
|
||||||
|
t.Fatalf("last success after recovery=%v", status.LastSuccessAt)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRefreshRejectsIncompleteSnapshot(t *testing.T) {
|
||||||
|
for name, mutate := range map[string]func(*model.Bootstrap){
|
||||||
|
"no sensors": func(b *model.Bootstrap) { b.Sensors = nil },
|
||||||
|
"no data version": func(b *model.Bootstrap) { b.DataVersion = "" },
|
||||||
|
"no model version": func(b *model.Bootstrap) { b.ModelVersion = "" },
|
||||||
|
"no context version": func(b *model.Bootstrap) { b.ContextVersion = "" },
|
||||||
|
"no context time": func(b *model.Bootstrap) { b.Context.UpdatedAt = time.Time{} },
|
||||||
|
} {
|
||||||
|
t.Run(name, func(t *testing.T) {
|
||||||
|
payload := fixturePayload()
|
||||||
|
mutate(&payload)
|
||||||
|
c := New(&stubFetcher{payload: payload}, Options{Timeout: time.Second})
|
||||||
|
fixed := time.Date(2026, 11, 14, 4, 0, 0, 0, time.UTC)
|
||||||
|
c.now = func() time.Time { return fixed }
|
||||||
|
if err := c.Refresh(context.Background()); err == nil {
|
||||||
|
t.Fatal("expected validation error")
|
||||||
|
}
|
||||||
|
if c.Loaded() || c.Ready() {
|
||||||
|
t.Fatalf("loaded=%v ready=%v after invalid snapshot", c.Loaded(), c.Ready())
|
||||||
|
}
|
||||||
|
if status := c.Status(); status.Failures != 1 || status.LastError == nil ||
|
||||||
|
status.NextRetryAt == nil || !status.NextRetryAt.Equal(fixed.Add(time.Second)) {
|
||||||
|
t.Fatalf("status=%+v", status)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRunRetriesWithBackoffThenLoads(t *testing.T) {
|
||||||
|
fetcher := &stubFetcher{fn: func(call int) (model.Bootstrap, error) {
|
||||||
|
if call <= 2 {
|
||||||
|
return model.Bootstrap{}, errors.New("not ready")
|
||||||
|
}
|
||||||
|
return fixturePayload(), nil
|
||||||
|
}}
|
||||||
|
c := New(fetcher, Options{RefreshInterval: 10 * time.Millisecond, Timeout: time.Second})
|
||||||
|
c.retry = []time.Duration{5 * time.Millisecond, 10 * time.Millisecond, 20 * time.Millisecond}
|
||||||
|
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
defer cancel()
|
||||||
|
done := make(chan struct{})
|
||||||
|
go func() {
|
||||||
|
c.Run(ctx)
|
||||||
|
close(done)
|
||||||
|
}()
|
||||||
|
|
||||||
|
deadline := time.Now().Add(2 * time.Second)
|
||||||
|
for !c.Loaded() {
|
||||||
|
if time.Now().After(deadline) {
|
||||||
|
t.Fatalf("cache did not load after retries; calls=%d status=%+v", fetcher.callCount(), c.Status())
|
||||||
|
}
|
||||||
|
time.Sleep(5 * time.Millisecond)
|
||||||
|
}
|
||||||
|
if got := fetcher.callCount(); got < 3 {
|
||||||
|
t.Fatalf("calls=%d, want at least 3", got)
|
||||||
|
}
|
||||||
|
if status := c.Status(); status.Failures != 0 || status.LastError != nil {
|
||||||
|
t.Fatalf("status after recovery=%+v", status)
|
||||||
|
}
|
||||||
|
|
||||||
|
cancel()
|
||||||
|
select {
|
||||||
|
case <-done:
|
||||||
|
case <-time.After(time.Second):
|
||||||
|
t.Fatal("Run did not stop after context cancellation")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRunStopsWhenContextCancelled(t *testing.T) {
|
||||||
|
fetcher := &stubFetcher{err: errors.New("down")}
|
||||||
|
c := New(fetcher, Options{RefreshInterval: time.Hour, Timeout: time.Second})
|
||||||
|
c.retry = []time.Duration{time.Hour}
|
||||||
|
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
done := make(chan struct{})
|
||||||
|
go func() {
|
||||||
|
c.Run(ctx)
|
||||||
|
close(done)
|
||||||
|
}()
|
||||||
|
|
||||||
|
deadline := time.Now().Add(time.Second)
|
||||||
|
for fetcher.callCount() == 0 {
|
||||||
|
if time.Now().After(deadline) {
|
||||||
|
t.Fatal("Run never attempted the first fetch")
|
||||||
|
}
|
||||||
|
time.Sleep(time.Millisecond)
|
||||||
|
}
|
||||||
|
cancel()
|
||||||
|
select {
|
||||||
|
case <-done:
|
||||||
|
case <-time.After(time.Second):
|
||||||
|
t.Fatal("Run did not stop after context cancellation")
|
||||||
|
}
|
||||||
|
if status := c.Status(); status.Failures != 1 || status.NextRetryAt == nil {
|
||||||
|
t.Fatalf("status=%+v", status)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -10,13 +10,17 @@ import (
|
|||||||
// Config is the process configuration.
|
// Config is the process configuration.
|
||||||
type Config struct {
|
type Config struct {
|
||||||
Addr string
|
Addr string
|
||||||
DataDir string
|
|
||||||
FastAPIURL string
|
FastAPIURL string
|
||||||
ControlUser string
|
ControlUser string
|
||||||
ControlPass string
|
ControlPass string
|
||||||
ControlToken string
|
ControlToken string
|
||||||
Heartbeat time.Duration
|
Heartbeat time.Duration
|
||||||
MLTimeout time.Duration
|
MLTimeout time.Duration
|
||||||
|
|
||||||
|
// RefreshInterval is how often a healthy /bootstrap snapshot is refreshed.
|
||||||
|
RefreshInterval time.Duration
|
||||||
|
// BootstrapTimeout bounds one /bootstrap request.
|
||||||
|
BootstrapTimeout time.Duration
|
||||||
}
|
}
|
||||||
|
|
||||||
// Load reads configuration from env with local-development defaults.
|
// Load reads configuration from env with local-development defaults.
|
||||||
@@ -27,13 +31,14 @@ func Load() Config {
|
|||||||
port := getenv("PORT", "8080")
|
port := getenv("PORT", "8080")
|
||||||
return Config{
|
return Config{
|
||||||
Addr: ":" + port,
|
Addr: ":" + port,
|
||||||
DataDir: getenv("DATA_DIR", "testdata/data"),
|
|
||||||
FastAPIURL: getenv("FASTAPI_URL", "http://localhost:8000"),
|
FastAPIURL: getenv("FASTAPI_URL", "http://localhost:8000"),
|
||||||
ControlUser: os.Getenv("CONTROL_USER"),
|
ControlUser: os.Getenv("CONTROL_USER"),
|
||||||
ControlPass: os.Getenv("CONTROL_PASS"),
|
ControlPass: os.Getenv("CONTROL_PASS"),
|
||||||
ControlToken: os.Getenv("CONTROL_TOKEN"),
|
ControlToken: os.Getenv("CONTROL_TOKEN"),
|
||||||
Heartbeat: durationEnv("SSE_HEARTBEAT", 15*time.Second),
|
Heartbeat: durationEnv("SSE_HEARTBEAT", 15*time.Second),
|
||||||
MLTimeout: durationEnv("ML_TIMEOUT", 1500*time.Millisecond),
|
MLTimeout: durationEnv("ML_TIMEOUT", 1500*time.Millisecond),
|
||||||
|
RefreshInterval: durationEnv("BOOTSTRAP_REFRESH", 60*time.Second),
|
||||||
|
BootstrapTimeout: durationEnv("BOOTSTRAP_TIMEOUT", 20*time.Second),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -6,22 +6,25 @@ import (
|
|||||||
)
|
)
|
||||||
|
|
||||||
func TestLoadDefaults(t *testing.T) {
|
func TestLoadDefaults(t *testing.T) {
|
||||||
for _, key := range []string{"PORT", "DATA_DIR", "FASTAPI_URL", "CONTROL_USER", "CONTROL_PASS", "CONTROL_TOKEN", "SSE_HEARTBEAT", "ML_TIMEOUT"} {
|
for _, key := range []string{
|
||||||
|
"PORT", "FASTAPI_URL", "CONTROL_USER", "CONTROL_PASS", "CONTROL_TOKEN",
|
||||||
|
"SSE_HEARTBEAT", "ML_TIMEOUT", "BOOTSTRAP_REFRESH", "BOOTSTRAP_TIMEOUT",
|
||||||
|
} {
|
||||||
t.Setenv(key, "")
|
t.Setenv(key, "")
|
||||||
}
|
}
|
||||||
cfg := Load()
|
cfg := Load()
|
||||||
if cfg.Addr != ":8080" {
|
if cfg.Addr != ":8080" {
|
||||||
t.Errorf("Addr=%q", cfg.Addr)
|
t.Errorf("Addr=%q", cfg.Addr)
|
||||||
}
|
}
|
||||||
if cfg.DataDir != "testdata/data" {
|
|
||||||
t.Errorf("DataDir=%q", cfg.DataDir)
|
|
||||||
}
|
|
||||||
if cfg.FastAPIURL != "http://localhost:8000" {
|
if cfg.FastAPIURL != "http://localhost:8000" {
|
||||||
t.Errorf("FastAPIURL=%q", cfg.FastAPIURL)
|
t.Errorf("FastAPIURL=%q", cfg.FastAPIURL)
|
||||||
}
|
}
|
||||||
if cfg.Heartbeat != 15*time.Second || cfg.MLTimeout != 1500*time.Millisecond {
|
if cfg.Heartbeat != 15*time.Second || cfg.MLTimeout != 1500*time.Millisecond {
|
||||||
t.Errorf("timeouts: heartbeat=%s ml=%s", cfg.Heartbeat, cfg.MLTimeout)
|
t.Errorf("timeouts: heartbeat=%s ml=%s", cfg.Heartbeat, cfg.MLTimeout)
|
||||||
}
|
}
|
||||||
|
if cfg.RefreshInterval != 60*time.Second || cfg.BootstrapTimeout != 20*time.Second {
|
||||||
|
t.Errorf("snapshot: refresh=%s timeout=%s", cfg.RefreshInterval, cfg.BootstrapTimeout)
|
||||||
|
}
|
||||||
if cfg.ControlAuthConfigured() {
|
if cfg.ControlAuthConfigured() {
|
||||||
t.Error("control auth should be unconfigured by default")
|
t.Error("control auth should be unconfigured by default")
|
||||||
}
|
}
|
||||||
@@ -32,16 +35,17 @@ func TestLoadDefaults(t *testing.T) {
|
|||||||
|
|
||||||
func TestLoadFromEnv(t *testing.T) {
|
func TestLoadFromEnv(t *testing.T) {
|
||||||
t.Setenv("PORT", "9999")
|
t.Setenv("PORT", "9999")
|
||||||
t.Setenv("DATA_DIR", "/data")
|
|
||||||
t.Setenv("FASTAPI_URL", "http://ml:8000")
|
t.Setenv("FASTAPI_URL", "http://ml:8000")
|
||||||
t.Setenv("CONTROL_USER", "user")
|
t.Setenv("CONTROL_USER", "user")
|
||||||
t.Setenv("CONTROL_PASS", "pass")
|
t.Setenv("CONTROL_PASS", "pass")
|
||||||
t.Setenv("CONTROL_TOKEN", "token-123")
|
t.Setenv("CONTROL_TOKEN", "token-123")
|
||||||
t.Setenv("SSE_HEARTBEAT", "5s")
|
t.Setenv("SSE_HEARTBEAT", "5s")
|
||||||
t.Setenv("ML_TIMEOUT", "2s")
|
t.Setenv("ML_TIMEOUT", "2s")
|
||||||
|
t.Setenv("BOOTSTRAP_REFRESH", "30s")
|
||||||
|
t.Setenv("BOOTSTRAP_TIMEOUT", "10s")
|
||||||
|
|
||||||
cfg := Load()
|
cfg := Load()
|
||||||
if cfg.Addr != ":9999" || cfg.DataDir != "/data" || cfg.FastAPIURL != "http://ml:8000" {
|
if cfg.Addr != ":9999" || cfg.FastAPIURL != "http://ml:8000" {
|
||||||
t.Fatalf("cfg=%+v", cfg)
|
t.Fatalf("cfg=%+v", cfg)
|
||||||
}
|
}
|
||||||
if cfg.ControlToken != "token-123" {
|
if cfg.ControlToken != "token-123" {
|
||||||
@@ -50,6 +54,9 @@ func TestLoadFromEnv(t *testing.T) {
|
|||||||
if cfg.Heartbeat != 5*time.Second || cfg.MLTimeout != 2*time.Second {
|
if cfg.Heartbeat != 5*time.Second || cfg.MLTimeout != 2*time.Second {
|
||||||
t.Fatalf("timeouts: heartbeat=%s ml=%s", cfg.Heartbeat, cfg.MLTimeout)
|
t.Fatalf("timeouts: heartbeat=%s ml=%s", cfg.Heartbeat, cfg.MLTimeout)
|
||||||
}
|
}
|
||||||
|
if cfg.RefreshInterval != 30*time.Second || cfg.BootstrapTimeout != 10*time.Second {
|
||||||
|
t.Fatalf("snapshot: refresh=%s timeout=%s", cfg.RefreshInterval, cfg.BootstrapTimeout)
|
||||||
|
}
|
||||||
if !cfg.ControlAuthConfigured() {
|
if !cfg.ControlAuthConfigured() {
|
||||||
t.Fatal("control auth should be configured")
|
t.Fatal("control auth should be configured")
|
||||||
}
|
}
|
||||||
@@ -58,17 +65,27 @@ func TestLoadFromEnv(t *testing.T) {
|
|||||||
func TestDurationAcceptsPlainSeconds(t *testing.T) {
|
func TestDurationAcceptsPlainSeconds(t *testing.T) {
|
||||||
t.Setenv("SSE_HEARTBEAT", "7")
|
t.Setenv("SSE_HEARTBEAT", "7")
|
||||||
t.Setenv("ML_TIMEOUT", "3")
|
t.Setenv("ML_TIMEOUT", "3")
|
||||||
|
t.Setenv("BOOTSTRAP_REFRESH", "45")
|
||||||
|
t.Setenv("BOOTSTRAP_TIMEOUT", "12")
|
||||||
cfg := Load()
|
cfg := Load()
|
||||||
if cfg.Heartbeat != 7*time.Second || cfg.MLTimeout != 3*time.Second {
|
if cfg.Heartbeat != 7*time.Second || cfg.MLTimeout != 3*time.Second {
|
||||||
t.Fatalf("timeouts: heartbeat=%s ml=%s", cfg.Heartbeat, cfg.MLTimeout)
|
t.Fatalf("timeouts: heartbeat=%s ml=%s", cfg.Heartbeat, cfg.MLTimeout)
|
||||||
}
|
}
|
||||||
|
if cfg.RefreshInterval != 45*time.Second || cfg.BootstrapTimeout != 12*time.Second {
|
||||||
|
t.Fatalf("snapshot: refresh=%s timeout=%s", cfg.RefreshInterval, cfg.BootstrapTimeout)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestDurationFallsBackOnInvalidValue(t *testing.T) {
|
func TestDurationFallsBackOnInvalidValue(t *testing.T) {
|
||||||
t.Setenv("SSE_HEARTBEAT", "soon")
|
t.Setenv("SSE_HEARTBEAT", "soon")
|
||||||
t.Setenv("ML_TIMEOUT", "-1s")
|
t.Setenv("ML_TIMEOUT", "-1s")
|
||||||
|
t.Setenv("BOOTSTRAP_REFRESH", "later")
|
||||||
|
t.Setenv("BOOTSTRAP_TIMEOUT", "-3s")
|
||||||
cfg := Load()
|
cfg := Load()
|
||||||
if cfg.Heartbeat != 15*time.Second || cfg.MLTimeout != 1500*time.Millisecond {
|
if cfg.Heartbeat != 15*time.Second || cfg.MLTimeout != 1500*time.Millisecond {
|
||||||
t.Fatalf("timeouts: heartbeat=%s ml=%s", cfg.Heartbeat, cfg.MLTimeout)
|
t.Fatalf("timeouts: heartbeat=%s ml=%s", cfg.Heartbeat, cfg.MLTimeout)
|
||||||
}
|
}
|
||||||
|
if cfg.RefreshInterval != 60*time.Second || cfg.BootstrapTimeout != 20*time.Second {
|
||||||
|
t.Fatalf("snapshot: refresh=%s timeout=%s", cfg.RefreshInterval, cfg.BootstrapTimeout)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
+46
-5
@@ -1,5 +1,6 @@
|
|||||||
// Package ml is the FastAPI client. Every call is bounded by a 1.5 s timeout;
|
// Package ml is the FastAPI client. Prediction calls use the caller's deadline
|
||||||
// callers fall back to baked JSON on any error.
|
// (1.5 s by default); callers fall back to the last good forecast on any error.
|
||||||
|
// The larger /bootstrap snapshot gets its own, longer transport timeout.
|
||||||
package ml
|
package ml
|
||||||
|
|
||||||
import (
|
import (
|
||||||
@@ -23,11 +24,16 @@ const DefaultTimeout = 1500 * time.Millisecond
|
|||||||
// larger than prediction payloads, so they get more than DefaultTimeout.
|
// larger than prediction payloads, so they get more than DefaultTimeout.
|
||||||
const controlTimeout = 30 * time.Second
|
const controlTimeout = 30 * time.Second
|
||||||
|
|
||||||
|
// bootstrapTimeout is the upper transport bound for the /bootstrap snapshot,
|
||||||
|
// which carries the full hourly series for every sensor.
|
||||||
|
const bootstrapTimeout = 30 * time.Second
|
||||||
|
|
||||||
// Client talks to the FastAPI ML service.
|
// Client talks to the FastAPI ML service.
|
||||||
type Client struct {
|
type Client struct {
|
||||||
baseURL string
|
baseURL string
|
||||||
http *http.Client
|
http *http.Client
|
||||||
control *http.Client
|
control *http.Client
|
||||||
|
bootstrap *http.Client
|
||||||
controlToken string
|
controlToken string
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -46,11 +52,46 @@ func NewWithToken(baseURL string, timeout time.Duration, controlToken string) *C
|
|||||||
baseURL: strings.TrimRight(baseURL, "/"),
|
baseURL: strings.TrimRight(baseURL, "/"),
|
||||||
http: &http.Client{Timeout: timeout},
|
http: &http.Client{Timeout: timeout},
|
||||||
control: &http.Client{Timeout: controlTimeout},
|
control: &http.Client{Timeout: controlTimeout},
|
||||||
|
bootstrap: &http.Client{Timeout: bootstrapTimeout},
|
||||||
controlToken: controlToken,
|
controlToken: controlToken,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// PredictRequest mirrors CONTRACT.md §3 POST /predict.
|
// Bootstrap fetches GET /bootstrap, the runtime snapshot the API caches. The
|
||||||
|
// caller supplies the shorter request deadline; this method only validates the
|
||||||
|
// transport response and that the snapshot is usable.
|
||||||
|
func (c *Client) Bootstrap(ctx context.Context) (model.Bootstrap, error) {
|
||||||
|
var out model.Bootstrap
|
||||||
|
req, err := http.NewRequestWithContext(ctx, http.MethodGet, c.baseURL+"/bootstrap", nil)
|
||||||
|
if err != nil {
|
||||||
|
return out, fmt.Errorf("ml: build request: %w", err)
|
||||||
|
}
|
||||||
|
req.Header.Set("Accept", "application/json")
|
||||||
|
|
||||||
|
resp, err := c.bootstrap.Do(req)
|
||||||
|
if err != nil {
|
||||||
|
return out, fmt.Errorf("ml: /bootstrap: %w", err)
|
||||||
|
}
|
||||||
|
defer resp.Body.Close()
|
||||||
|
|
||||||
|
if resp.StatusCode < 200 || resp.StatusCode > 299 {
|
||||||
|
snippet, _ := io.ReadAll(io.LimitReader(resp.Body, 256))
|
||||||
|
return out, fmt.Errorf("ml: /bootstrap: status %d: %s", resp.StatusCode, strings.TrimSpace(string(snippet)))
|
||||||
|
}
|
||||||
|
raw, err := io.ReadAll(io.LimitReader(resp.Body, 64<<20))
|
||||||
|
if err != nil {
|
||||||
|
return out, fmt.Errorf("ml: read /bootstrap: %w", err)
|
||||||
|
}
|
||||||
|
if err := json.Unmarshal(raw, &out); err != nil {
|
||||||
|
return out, fmt.Errorf("ml: decode /bootstrap: %w", err)
|
||||||
|
}
|
||||||
|
if len(out.Sensors) == 0 {
|
||||||
|
return out, fmt.Errorf("ml: /bootstrap returned no sensors")
|
||||||
|
}
|
||||||
|
return out, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// PredictRequest mirrors the API contract POST /predict.
|
||||||
type PredictRequest struct {
|
type PredictRequest struct {
|
||||||
Sensor string `json:"sensor"`
|
Sensor string `json:"sensor"`
|
||||||
Horizon string `json:"horizon"`
|
Horizon string `json:"horizon"`
|
||||||
@@ -65,7 +106,7 @@ type WeatherInput struct {
|
|||||||
PrecipMM float64 `json:"precip_mm"`
|
PrecipMM float64 `json:"precip_mm"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// SimulateRequest mirrors CONTRACT.md §3 POST /simulate.
|
// SimulateRequest mirrors the API contract POST /simulate.
|
||||||
type SimulateRequest struct {
|
type SimulateRequest struct {
|
||||||
Scenario string `json:"scenario"`
|
Scenario string `json:"scenario"`
|
||||||
Weather model.WeatherInput `json:"weather"`
|
Weather model.WeatherInput `json:"weather"`
|
||||||
@@ -78,7 +119,7 @@ type TrainRequest struct {
|
|||||||
Model string `json:"model"`
|
Model string `json:"model"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// TrainModels are the accepted training targets from CONTRACT.md §3.
|
// TrainModels are the accepted training targets.
|
||||||
var TrainModels = []string{"small", "forecast"}
|
var TrainModels = []string{"small", "forecast"}
|
||||||
|
|
||||||
// Predict calls POST /predict and returns the forecast points. The contract
|
// Predict calls POST /predict and returns the forecast points. The contract
|
||||||
|
|||||||
@@ -121,6 +121,94 @@ func TestSimulate(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestBootstrapFetchesSnapshot(t *testing.T) {
|
||||||
|
var gotMethod, gotPath, gotAuth string
|
||||||
|
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
gotMethod, gotPath, gotAuth = r.Method, r.URL.Path, r.Header.Get("Authorization")
|
||||||
|
_, _ = w.Write([]byte(`{
|
||||||
|
"data_version":"2026-11-07T23:51:00Z",
|
||||||
|
"model_version":"histgb-2026-11-14",
|
||||||
|
"context_version":"2026-11-14T03:00:00Z",
|
||||||
|
"sensors":[{"id":"hn-bahnhof-01","name":"Bahnhof","lat":49.14,"lon":9.2,"mean":24.2,"p95":88.2,"battery":71}],
|
||||||
|
"series":{"hn-bahnhof-01":[{"t":"2026-11-01T10:00:00Z","v":12.3}]},
|
||||||
|
"context":{"weather":{"now":{"t":"2026-11-14T03:00:00Z","temp_c":8.9,"precip_mm":0,"cloud_pct":80,"wind_kmh":11,"humidity_pct":79},"forecast":[]},"holidays":[{"date":"2026-11-01","name":"Allerheiligen"}],"updated_at":"2026-11-14T03:00:00Z"},
|
||||||
|
"meta":{"coverage":{"sensors":8,"from":"2026-06-12T00:00:00Z","to":"2026-11-07T23:51:00Z","cadence_min":15},"metrics":{"model":"weather","mae":3.36,"wape":27.8,"r2":0.779,"baseline_mae":5.4,"baseline_wape":44.7},"limits":["estimated devices, not people"]}
|
||||||
|
}`))
|
||||||
|
}))
|
||||||
|
defer srv.Close()
|
||||||
|
|
||||||
|
// The control token is for /train only; /bootstrap must stay unauthenticated.
|
||||||
|
payload, err := NewWithToken(srv.URL, time.Second, "secret-token").Bootstrap(context.Background())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Bootstrap: %v", err)
|
||||||
|
}
|
||||||
|
if gotMethod != http.MethodGet || gotPath != "/bootstrap" {
|
||||||
|
t.Fatalf("request=%s %s, want GET /bootstrap", gotMethod, gotPath)
|
||||||
|
}
|
||||||
|
if gotAuth != "" {
|
||||||
|
t.Errorf("Authorization=%q, want none", gotAuth)
|
||||||
|
}
|
||||||
|
if len(payload.Sensors) != 1 || payload.Sensors[0].ID != "hn-bahnhof-01" || payload.Sensors[0].Battery == nil {
|
||||||
|
t.Fatalf("sensors=%+v", payload.Sensors)
|
||||||
|
}
|
||||||
|
if len(payload.Series["hn-bahnhof-01"]) != 1 || payload.Series["hn-bahnhof-01"][0].V != 12.3 {
|
||||||
|
t.Fatalf("series=%+v", payload.Series)
|
||||||
|
}
|
||||||
|
if payload.Context.UpdatedAt.IsZero() || len(payload.Context.Holidays) != 1 {
|
||||||
|
t.Fatalf("context=%+v", payload.Context)
|
||||||
|
}
|
||||||
|
if payload.DataVersion == "" || payload.ModelVersion == "" || payload.ContextVersion == "" {
|
||||||
|
t.Fatalf("versions=%+v", payload)
|
||||||
|
}
|
||||||
|
if payload.Meta.Coverage.Sensors != 8 || payload.Meta.Metrics.MAE != 3.36 || len(payload.Meta.Limits) != 1 {
|
||||||
|
t.Fatalf("meta=%+v", payload.Meta)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBootstrapRejectsErrorAndEmptySensors(t *testing.T) {
|
||||||
|
boom := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||||
|
http.Error(w, "not ready", http.StatusServiceUnavailable)
|
||||||
|
}))
|
||||||
|
defer boom.Close()
|
||||||
|
if _, err := New(boom.URL, time.Second).Bootstrap(context.Background()); err == nil {
|
||||||
|
t.Fatal("expected an error for HTTP 503")
|
||||||
|
}
|
||||||
|
|
||||||
|
empty := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||||
|
_, _ = w.Write([]byte(`{"sensors":[]}`))
|
||||||
|
}))
|
||||||
|
defer empty.Close()
|
||||||
|
if _, err := New(empty.URL, time.Second).Bootstrap(context.Background()); err == nil {
|
||||||
|
t.Fatal("expected an error for an empty sensor list")
|
||||||
|
}
|
||||||
|
|
||||||
|
bad := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||||
|
_, _ = w.Write([]byte(`{`))
|
||||||
|
}))
|
||||||
|
defer bad.Close()
|
||||||
|
if _, err := New(bad.URL, time.Second).Bootstrap(context.Background()); err == nil {
|
||||||
|
t.Fatal("expected an error for invalid JSON")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBootstrapTimeout(t *testing.T) {
|
||||||
|
slow := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||||
|
time.Sleep(150 * time.Millisecond)
|
||||||
|
_, _ = w.Write([]byte(`{"sensors":[{"id":"s"}]}`))
|
||||||
|
}))
|
||||||
|
defer slow.Close()
|
||||||
|
|
||||||
|
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Millisecond)
|
||||||
|
defer cancel()
|
||||||
|
start := time.Now()
|
||||||
|
if _, err := New(slow.URL, time.Second).Bootstrap(ctx); err == nil {
|
||||||
|
t.Fatal("expected a context timeout")
|
||||||
|
}
|
||||||
|
if elapsed := time.Since(start); elapsed > time.Second {
|
||||||
|
t.Fatalf("timeout took %s", elapsed)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestNewUsesDefaultTimeout(t *testing.T) {
|
func TestNewUsesDefaultTimeout(t *testing.T) {
|
||||||
client := New("http://example.invalid", 0)
|
client := New("http://example.invalid", 0)
|
||||||
if client.http.Timeout != DefaultTimeout {
|
if client.http.Timeout != DefaultTimeout {
|
||||||
|
|||||||
+34
-2
@@ -1,10 +1,11 @@
|
|||||||
// Package model contains the wire types shared by the HTTP handlers, the
|
// Package model contains the wire types shared by the HTTP handlers, the
|
||||||
// baked-data store and the FastAPI client. JSON tags mirror CONTRACT.md.
|
// in-memory snapshot cache and the FastAPI client. JSON tags mirror the API
|
||||||
|
// contract.
|
||||||
package model
|
package model
|
||||||
|
|
||||||
import "time"
|
import "time"
|
||||||
|
|
||||||
// Horizons and enums from CONTRACT.md §1.
|
// Horizons and enums from the API contract.
|
||||||
var (
|
var (
|
||||||
Horizons = []string{"15m", "1h", "48h", "7d"}
|
Horizons = []string{"15m", "1h", "48h", "7d"}
|
||||||
Scenarios = []string{"base", "sunny-saturday-market", "rainy-saturday", "weekday-event"}
|
Scenarios = []string{"base", "sunny-saturday-market", "rainy-saturday", "weekday-event"}
|
||||||
@@ -72,6 +73,37 @@ type Meta struct {
|
|||||||
Coverage Coverage `json:"coverage"`
|
Coverage Coverage `json:"coverage"`
|
||||||
Metrics Metrics `json:"metrics"`
|
Metrics Metrics `json:"metrics"`
|
||||||
Limits []string `json:"limits"`
|
Limits []string `json:"limits"`
|
||||||
|
Cache CacheStatus `json:"cache"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// CacheStatus is the runtime snapshot state exposed by GET /api/meta.
|
||||||
|
// LastError, LastSuccessAt and NextRetryAt are null when they do not apply.
|
||||||
|
type CacheStatus struct {
|
||||||
|
Source string `json:"source"`
|
||||||
|
LastSuccessAt *time.Time `json:"last_success_at"`
|
||||||
|
LastError *string `json:"last_error"`
|
||||||
|
NextRetryAt *time.Time `json:"next_retry_at"`
|
||||||
|
Failures int `json:"failures"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// BootstrapMeta is the nested meta block of FastAPI GET /bootstrap.
|
||||||
|
type BootstrapMeta struct {
|
||||||
|
Coverage Coverage `json:"coverage"`
|
||||||
|
Metrics Metrics `json:"metrics"`
|
||||||
|
Limits []string `json:"limits"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// Bootstrap is the snapshot FastAPI serves at GET /bootstrap and Go keeps in
|
||||||
|
// memory. Forecasts are deliberately absent: they come from /predict and are
|
||||||
|
// cached separately as they are requested.
|
||||||
|
type Bootstrap struct {
|
||||||
|
DataVersion string `json:"data_version"`
|
||||||
|
ModelVersion string `json:"model_version"`
|
||||||
|
ContextVersion string `json:"context_version"`
|
||||||
|
Sensors []Sensor `json:"sensors"`
|
||||||
|
Series map[string][]Point `json:"series"`
|
||||||
|
Context ContextData `json:"context"`
|
||||||
|
Meta BootstrapMeta `json:"meta"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// WeatherNow is the current conditions block.
|
// WeatherNow is the current conditions block.
|
||||||
|
|||||||
@@ -1,6 +1,5 @@
|
|||||||
// Package staffing implements the deterministic decision heuristic from
|
// Package staffing turns forecast points into an explainable shift plan using
|
||||||
// CONTRACT.md §5. It never calls the model: it consumes forecast points and
|
// the deterministic heuristic in the API contract. It never calls the model.
|
||||||
// turns them into an explainable shift plan.
|
|
||||||
package staffing
|
package staffing
|
||||||
|
|
||||||
import (
|
import (
|
||||||
@@ -12,7 +11,8 @@ import (
|
|||||||
"pulsecast/api/internal/model"
|
"pulsecast/api/internal/model"
|
||||||
)
|
)
|
||||||
|
|
||||||
// Build creates a plan for one sensor/scenario/standard.
|
// Build creates a plan for one sensor/scenario/standard from the forecast's
|
||||||
|
// complete store days.
|
||||||
func Build(sensor, scenario, standard string, points []model.ForecastPoint, ctx model.ContextData) (model.StaffingResponse, error) {
|
func Build(sensor, scenario, standard string, points []model.ForecastPoint, ctx model.ContextData) (model.StaffingResponse, error) {
|
||||||
perStaff, ok := visitorsPerStaff(standard)
|
perStaff, ok := visitorsPerStaff(standard)
|
||||||
if !ok {
|
if !ok {
|
||||||
|
|||||||
@@ -1,189 +0,0 @@
|
|||||||
// Package store loads the baked JSON artifacts described in CONTRACT.md §4.
|
|
||||||
// The data is immutable after Load; all accessors are safe for concurrent use.
|
|
||||||
package store
|
|
||||||
|
|
||||||
import (
|
|
||||||
"encoding/json"
|
|
||||||
"errors"
|
|
||||||
"fmt"
|
|
||||||
"os"
|
|
||||||
"path/filepath"
|
|
||||||
"sort"
|
|
||||||
"strings"
|
|
||||||
|
|
||||||
"pulsecast/api/internal/model"
|
|
||||||
)
|
|
||||||
|
|
||||||
// Store is the in-memory baked dataset.
|
|
||||||
type Store struct {
|
|
||||||
meta model.Meta
|
|
||||||
sensors []model.Sensor
|
|
||||||
sensorBy map[string]model.Sensor
|
|
||||||
series map[string][]model.Point
|
|
||||||
forecasts map[string]map[string]map[string][]model.ForecastPoint
|
|
||||||
context model.ContextData
|
|
||||||
}
|
|
||||||
|
|
||||||
// Load reads every artifact from dir. It returns a usable (possibly partial)
|
|
||||||
// store together with a joined error so callers can start the API and let
|
|
||||||
// /api/readyz report 503 instead of crash-looping.
|
|
||||||
func Load(dir string) (*Store, error) {
|
|
||||||
s := &Store{
|
|
||||||
sensorBy: make(map[string]model.Sensor),
|
|
||||||
series: make(map[string][]model.Point),
|
|
||||||
forecasts: make(map[string]map[string]map[string][]model.ForecastPoint),
|
|
||||||
}
|
|
||||||
|
|
||||||
var errs []error
|
|
||||||
|
|
||||||
if err := readJSON(filepath.Join(dir, "sensors.json"), &s.sensors); err != nil {
|
|
||||||
errs = append(errs, err)
|
|
||||||
}
|
|
||||||
sort.Slice(s.sensors, func(i, j int) bool { return s.sensors[i].ID < s.sensors[j].ID })
|
|
||||||
for _, sensor := range s.sensors {
|
|
||||||
s.sensorBy[sensor.ID] = sensor
|
|
||||||
}
|
|
||||||
|
|
||||||
if err := readJSON(filepath.Join(dir, "metrics.json"), &s.meta); err != nil {
|
|
||||||
errs = append(errs, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if err := readJSON(filepath.Join(dir, "context.json"), &s.context); err != nil {
|
|
||||||
errs = append(errs, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if err := loadSeries(dir, s); err != nil {
|
|
||||||
errs = append(errs, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if err := readJSON(filepath.Join(dir, "forecasts.json"), &s.forecasts); err != nil {
|
|
||||||
errs = append(errs, err)
|
|
||||||
}
|
|
||||||
for sensorID, scenarios := range s.forecasts {
|
|
||||||
for scenario, horizons := range scenarios {
|
|
||||||
for horizon, points := range horizons {
|
|
||||||
sort.Slice(points, func(i, j int) bool { return points[i].T.Before(points[j].T) })
|
|
||||||
horizons[horizon] = points
|
|
||||||
}
|
|
||||||
scenarios[scenario] = horizons
|
|
||||||
}
|
|
||||||
s.forecasts[sensorID] = scenarios
|
|
||||||
}
|
|
||||||
|
|
||||||
return s, errors.Join(errs...)
|
|
||||||
}
|
|
||||||
|
|
||||||
func loadSeries(dir string, s *Store) error {
|
|
||||||
var errs []error
|
|
||||||
|
|
||||||
// Contract allows a single series.json mapping sensor id -> points.
|
|
||||||
var combined map[string][]model.Point
|
|
||||||
err := readJSON(filepath.Join(dir, "series.json"), &combined)
|
|
||||||
switch {
|
|
||||||
case err == nil:
|
|
||||||
for sensorID, points := range combined {
|
|
||||||
sort.Slice(points, func(i, j int) bool { return points[i].T.Before(points[j].T) })
|
|
||||||
s.series[sensorID] = points
|
|
||||||
}
|
|
||||||
case !errors.Is(err, os.ErrNotExist):
|
|
||||||
errs = append(errs, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Contract also allows series/<id>.json; these override the combined file.
|
|
||||||
entries, err := os.ReadDir(filepath.Join(dir, "series"))
|
|
||||||
if err != nil {
|
|
||||||
if !errors.Is(err, os.ErrNotExist) {
|
|
||||||
errs = append(errs, fmt.Errorf("series dir: %w", err))
|
|
||||||
}
|
|
||||||
return errors.Join(errs...)
|
|
||||||
}
|
|
||||||
for _, entry := range entries {
|
|
||||||
if entry.IsDir() || !strings.HasSuffix(entry.Name(), ".json") {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
var points []model.Point
|
|
||||||
path := filepath.Join(dir, "series", entry.Name())
|
|
||||||
if err := readJSON(path, &points); err != nil {
|
|
||||||
errs = append(errs, err)
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
sort.Slice(points, func(i, j int) bool { return points[i].T.Before(points[j].T) })
|
|
||||||
s.series[strings.TrimSuffix(entry.Name(), ".json")] = points
|
|
||||||
}
|
|
||||||
return errors.Join(errs...)
|
|
||||||
}
|
|
||||||
|
|
||||||
func readJSON(path string, dst any) error {
|
|
||||||
raw, err := os.ReadFile(path)
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("read %s: %w", path, err)
|
|
||||||
}
|
|
||||||
if err := json.Unmarshal(raw, dst); err != nil {
|
|
||||||
return fmt.Errorf("decode %s: %w", path, err)
|
|
||||||
}
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// Meta returns the model/data metadata.
|
|
||||||
func (s *Store) Meta() model.Meta { return s.meta }
|
|
||||||
|
|
||||||
// Sensors returns the sensors sorted by id.
|
|
||||||
func (s *Store) Sensors() []model.Sensor {
|
|
||||||
out := make([]model.Sensor, len(s.sensors))
|
|
||||||
copy(out, s.sensors)
|
|
||||||
return out
|
|
||||||
}
|
|
||||||
|
|
||||||
// Sensor looks up one sensor.
|
|
||||||
func (s *Store) Sensor(id string) (model.Sensor, bool) {
|
|
||||||
sensor, ok := s.sensorBy[id]
|
|
||||||
return sensor, ok
|
|
||||||
}
|
|
||||||
|
|
||||||
// Series returns the raw (15-minute) series for a sensor.
|
|
||||||
func (s *Store) Series(id string) ([]model.Point, bool) {
|
|
||||||
points, ok := s.series[id]
|
|
||||||
return points, ok
|
|
||||||
}
|
|
||||||
|
|
||||||
// Forecast returns baked points for sensor/scenario/horizon.
|
|
||||||
func (s *Store) Forecast(sensor, scenario, horizon string) ([]model.ForecastPoint, bool) {
|
|
||||||
scenarios, ok := s.forecasts[sensor]
|
|
||||||
if !ok {
|
|
||||||
return nil, false
|
|
||||||
}
|
|
||||||
horizons, ok := scenarios[scenario]
|
|
||||||
if !ok {
|
|
||||||
return nil, false
|
|
||||||
}
|
|
||||||
points, ok := horizons[horizon]
|
|
||||||
if !ok || len(points) == 0 {
|
|
||||||
return nil, false
|
|
||||||
}
|
|
||||||
return points, true
|
|
||||||
}
|
|
||||||
|
|
||||||
// Context returns the baked weather/holiday context.
|
|
||||||
func (s *Store) Context() model.ContextData { return s.context }
|
|
||||||
|
|
||||||
// Versions returns data, model and context version strings.
|
|
||||||
func (s *Store) Versions() (data, modelVersion, context string) {
|
|
||||||
return s.meta.DataVersion, s.meta.ModelVersion, s.meta.ContextVersion
|
|
||||||
}
|
|
||||||
|
|
||||||
// Ready reports whether the minimum baked set is present. FastAPI health is
|
|
||||||
// deliberately not part of readiness: the cache fallback is the safety net.
|
|
||||||
func (s *Store) Ready() bool {
|
|
||||||
if len(s.sensors) == 0 || s.meta.ModelVersion == "" || s.context.UpdatedAt.IsZero() {
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
for _, sensor := range s.sensors {
|
|
||||||
if _, ok := s.Forecast(sensor.ID, "base", "7d"); ok {
|
|
||||||
return true
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
|
|
||||||
// Len returns the number of loaded sensors (used in tests/logs).
|
|
||||||
func (s *Store) Len() int { return len(s.sensors) }
|
|
||||||
@@ -1,65 +0,0 @@
|
|||||||
package store
|
|
||||||
|
|
||||||
import (
|
|
||||||
"testing"
|
|
||||||
)
|
|
||||||
|
|
||||||
func TestLoadFixtures(t *testing.T) {
|
|
||||||
st, err := Load("../../testdata/data")
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("Load: %v", err)
|
|
||||||
}
|
|
||||||
if !st.Ready() {
|
|
||||||
t.Fatal("Ready()=false for the fixture dataset")
|
|
||||||
}
|
|
||||||
if st.Len() != 8 {
|
|
||||||
t.Fatalf("Len()=%d, want 8", st.Len())
|
|
||||||
}
|
|
||||||
|
|
||||||
sensors := st.Sensors()
|
|
||||||
for i := 1; i < len(sensors); i++ {
|
|
||||||
if sensors[i-1].ID >= sensors[i].ID {
|
|
||||||
t.Fatalf("sensors not sorted: %q then %q", sensors[i-1].ID, sensors[i].ID)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if _, ok := st.Sensor("hn-bahnhof-01"); !ok {
|
|
||||||
t.Fatal("missing hn-bahnhof-01")
|
|
||||||
}
|
|
||||||
if _, ok := st.Sensor("nope"); ok {
|
|
||||||
t.Fatal("unexpected sensor nope")
|
|
||||||
}
|
|
||||||
|
|
||||||
if points, ok := st.Series("hn-bahnhof-01"); !ok || len(points) == 0 {
|
|
||||||
t.Fatalf("series: ok=%v points=%d", ok, len(points))
|
|
||||||
}
|
|
||||||
if points, ok := st.Forecast("hn-bahnhof-01", "sunny-saturday-market", "7d"); !ok || len(points) != 168 {
|
|
||||||
t.Fatalf("forecast: ok=%v points=%d", ok, len(points))
|
|
||||||
}
|
|
||||||
if _, ok := st.Forecast("hn-bahnhof-01", "base", "2h"); ok {
|
|
||||||
t.Fatal("unexpected horizon 2h")
|
|
||||||
}
|
|
||||||
if ctx := st.Context(); ctx.UpdatedAt.IsZero() || len(ctx.Weather.Forecast) == 0 || len(ctx.Holidays) == 0 {
|
|
||||||
t.Fatalf("context incomplete: %+v", ctx)
|
|
||||||
}
|
|
||||||
|
|
||||||
data, modelVersion, context := st.Versions()
|
|
||||||
if data == "" || modelVersion == "" || context == "" {
|
|
||||||
t.Fatalf("versions: data=%q model=%q context=%q", data, modelVersion, context)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestLoadMissingDirectoryIsPartial(t *testing.T) {
|
|
||||||
st, err := Load(t.TempDir())
|
|
||||||
if err == nil {
|
|
||||||
t.Fatal("expected an error for an empty data dir")
|
|
||||||
}
|
|
||||||
if st == nil {
|
|
||||||
t.Fatal("Load returned a nil store; handlers need a usable store")
|
|
||||||
}
|
|
||||||
if st.Ready() {
|
|
||||||
t.Fatal("Ready()=true without data")
|
|
||||||
}
|
|
||||||
if st.Len() != 0 {
|
|
||||||
t.Fatalf("Len()=%d, want 0", st.Len())
|
|
||||||
}
|
|
||||||
}
|
|
||||||
Executable
+433
@@ -0,0 +1,433 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Repeatable end-to-end integration + fallback verification for the Go API.
|
||||||
|
#
|
||||||
|
# It starts the real FastAPI ML service on :18000 and the Go API on :18080,
|
||||||
|
# trains the small (and optionally forecast) model through POST /api/train,
|
||||||
|
# checks SSE scenario propagation / heartbeat / reconnect, kills FastAPI to
|
||||||
|
# prove the runtime snapshot + last-good forecast fallback, restarts it, and
|
||||||
|
# verifies recovery. With --load it also runs the 120-client 5-minute SSE test.
|
||||||
|
#
|
||||||
|
# scripts/integration.sh # fast checks (~1-2 min)
|
||||||
|
# scripts/integration.sh --load # plus the 5-minute load test
|
||||||
|
#
|
||||||
|
# Requirements: go, curl, jq, uv, and the ML checkout (read-only).
|
||||||
|
# The script owns both ports; it refuses to run if something is already there.
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)
|
||||||
|
ML_DIR=${ML_DIR:-/home/noel/Documents/Hackathon_2026/ML-py.git/main}
|
||||||
|
RAW_CSV=${RAW_CSV:-/home/noel/Documents/Hackathon_2026/data/raw/schwarz-pax.csv}
|
||||||
|
|
||||||
|
GO_PORT=${GO_PORT:-18080}
|
||||||
|
ML_PORT=${ML_PORT:-18000}
|
||||||
|
API_URL="http://127.0.0.1:${GO_PORT}"
|
||||||
|
ML_URL="http://127.0.0.1:${ML_PORT}"
|
||||||
|
|
||||||
|
CONTROL_USER=${CONTROL_USER:-control}
|
||||||
|
# Local test values only; override both in any shared environment. The API
|
||||||
|
# itself has no credential defaults and fails closed when they are unset.
|
||||||
|
CONTROL_PASS=${CONTROL_PASS:-pulsecast-dev-pass}
|
||||||
|
CONTROL_TOKEN=${CONTROL_TOKEN:-pulsecast-control-token}
|
||||||
|
BOOTSTRAP_REFRESH=${BOOTSTRAP_REFRESH:-10s}
|
||||||
|
ML_TIMEOUT=${ML_TIMEOUT:-1500ms}
|
||||||
|
|
||||||
|
WORKDIR=${WORKDIR:-/tmp/pulsecast-integration}
|
||||||
|
RESULTS_DIR=${RESULTS_DIR:-$WORKDIR/results}
|
||||||
|
SMALL_CSV=${SMALL_CSV:-$WORKDIR/small.csv}
|
||||||
|
SMALL_FROM=${SMALL_FROM:-2026-09-01}
|
||||||
|
SMALL_TO=${SMALL_TO:-2026-10-16}
|
||||||
|
|
||||||
|
RUN_LOAD=0
|
||||||
|
RUN_FORECAST=${RUN_FORECAST:-1}
|
||||||
|
for arg in "$@"; do
|
||||||
|
case "$arg" in
|
||||||
|
--load) RUN_LOAD=1 ;;
|
||||||
|
--no-forecast) RUN_FORECAST=0 ;;
|
||||||
|
-h|--help) sed -n '2,14p' "$0"; exit 0 ;;
|
||||||
|
*) echo "unknown argument: $arg" >&2; exit 2 ;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
|
||||||
|
mkdir -p "$WORKDIR" "$RESULTS_DIR"
|
||||||
|
API_LOG="$WORKDIR/api.log"
|
||||||
|
ML_LOG="$WORKDIR/ml.log"
|
||||||
|
API_PID=""
|
||||||
|
ML_PID=""
|
||||||
|
|
||||||
|
ms() { date +%s%3N; }
|
||||||
|
|
||||||
|
log() { printf '\n==> %s\n' "$*"; }
|
||||||
|
ok() { printf ' ok: %s\n' "$*"; }
|
||||||
|
fail() {
|
||||||
|
printf '\nFAIL: %s\n' "$*" >&2
|
||||||
|
printf 'api log: %s\nml log: %s\n' "$API_LOG" "$ML_LOG" >&2
|
||||||
|
exit 1
|
||||||
|
}
|
||||||
|
|
||||||
|
need() {
|
||||||
|
command -v "$1" >/dev/null 2>&1 || fail "missing command: $1"
|
||||||
|
}
|
||||||
|
|
||||||
|
port_busy() { ss -ltn 2>/dev/null | awk '{print $4}' | grep -q ":$1\$"; }
|
||||||
|
|
||||||
|
cleanup() {
|
||||||
|
local status=$?
|
||||||
|
if [[ -n "$API_PID" ]]; then
|
||||||
|
kill -TERM -- "-$API_PID" 2>/dev/null || kill -TERM "$API_PID" 2>/dev/null || true
|
||||||
|
fi
|
||||||
|
if [[ -n "$ML_PID" ]]; then
|
||||||
|
kill -TERM -- "-$ML_PID" 2>/dev/null || kill -TERM "$ML_PID" 2>/dev/null || true
|
||||||
|
fuser -k -TERM "$ML_PORT/tcp" >/dev/null 2>&1 || true
|
||||||
|
fi
|
||||||
|
exit "$status"
|
||||||
|
}
|
||||||
|
trap cleanup EXIT INT TERM
|
||||||
|
|
||||||
|
wait_http() { # url, deadline seconds
|
||||||
|
local url=$1 deadline=$2 start now
|
||||||
|
start=$(ms)
|
||||||
|
while :; do
|
||||||
|
if curl -sf -m 1 "$url" >/dev/null 2>&1; then return 0; fi
|
||||||
|
now=$(ms)
|
||||||
|
(( now - start < deadline * 1000 )) || return 1
|
||||||
|
sleep 0.2
|
||||||
|
done
|
||||||
|
}
|
||||||
|
|
||||||
|
json_get() { # file, jq filter
|
||||||
|
jq -r "$2" "$1" 2>/dev/null
|
||||||
|
}
|
||||||
|
|
||||||
|
get() { # path, outfile -> status code
|
||||||
|
curl -sS -m 10 -o "$2" -w '%{http_code}' "$API_URL$1"
|
||||||
|
}
|
||||||
|
|
||||||
|
need go
|
||||||
|
need curl
|
||||||
|
need jq
|
||||||
|
need uv
|
||||||
|
need awk
|
||||||
|
[[ -x "$ML_DIR/.venv/bin/python" || -f "$ML_DIR/pyproject.toml" ]] || fail "ML checkout not found at $ML_DIR"
|
||||||
|
[[ -f "$RAW_CSV" ]] || fail "raw CSV not found at $RAW_CSV"
|
||||||
|
port_busy "$GO_PORT" && fail "port $GO_PORT is already in use; stop that process first"
|
||||||
|
port_busy "$ML_PORT" && fail "port $ML_PORT is already in use; stop that process first"
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------ small CSV
|
||||||
|
log "building a small training CSV ($SMALL_FROM .. $SMALL_TO)"
|
||||||
|
if [[ ! -s "$SMALL_CSV" || "$RAW_CSV" -nt "$SMALL_CSV" ]]; then
|
||||||
|
head -1 "$RAW_CSV" > "$SMALL_CSV.tmp"
|
||||||
|
awk -F',' -v from="\"$SMALL_FROM" -v to="\"$SMALL_TO" \
|
||||||
|
'NR>1 && $6 ~ /densityData/ && $1 >= from && $1 < to' "$RAW_CSV" >> "$SMALL_CSV.tmp"
|
||||||
|
mv "$SMALL_CSV.tmp" "$SMALL_CSV"
|
||||||
|
fi
|
||||||
|
small_rows=$(wc -l < "$SMALL_CSV")
|
||||||
|
(( small_rows > 1000 )) || fail "small CSV has only $small_rows lines"
|
||||||
|
ok "$small_rows rows, $(du -h "$SMALL_CSV" | cut -f1)"
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------- FastAPI
|
||||||
|
log "starting FastAPI on :$ML_PORT"
|
||||||
|
(
|
||||||
|
cd "$ML_DIR"
|
||||||
|
env CONTROL_TOKEN="$CONTROL_TOKEN" \
|
||||||
|
PULSECAST_MODELS="$WORKDIR/models" \
|
||||||
|
PULSECAST_UPLOADS="$WORKDIR/uploads" \
|
||||||
|
setsid uv run --frozen uvicorn pulsecast.server:app \
|
||||||
|
--host 127.0.0.1 --port "$ML_PORT" > "$ML_LOG" 2>&1 &
|
||||||
|
echo $! > "$WORKDIR/ml.pid"
|
||||||
|
)
|
||||||
|
ML_PID=$(cat "$WORKDIR/ml.pid")
|
||||||
|
wait_http "$ML_URL/readyz" 90 || fail "FastAPI did not become ready; see $ML_LOG"
|
||||||
|
ok "readyz 200"
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------- Go API
|
||||||
|
log "building and starting the Go API on :$GO_PORT"
|
||||||
|
( cd "$ROOT" && go build -o "$WORKDIR/api" ./cmd/api )
|
||||||
|
(
|
||||||
|
env PORT="$GO_PORT" FASTAPI_URL="$ML_URL" \
|
||||||
|
CONTROL_USER="$CONTROL_USER" CONTROL_PASS="$CONTROL_PASS" CONTROL_TOKEN="$CONTROL_TOKEN" \
|
||||||
|
BOOTSTRAP_REFRESH="$BOOTSTRAP_REFRESH" ML_TIMEOUT="$ML_TIMEOUT" \
|
||||||
|
setsid "$WORKDIR/api" > "$API_LOG" 2>&1 &
|
||||||
|
echo $! > "$WORKDIR/api.pid"
|
||||||
|
)
|
||||||
|
API_PID=$(cat "$WORKDIR/api.pid")
|
||||||
|
wait_http "$API_URL/api/readyz" 30 || fail "Go API did not become ready; see $API_LOG"
|
||||||
|
ok "readyz 200 after first /bootstrap snapshot"
|
||||||
|
|
||||||
|
initial_meta="$RESULTS_DIR/meta-initial.json"
|
||||||
|
get "/api/meta" "$initial_meta" >/dev/null
|
||||||
|
initial_success=$(json_get "$initial_meta" '.cache.last_success_at')
|
||||||
|
[[ -n "$initial_success" && "$initial_success" != "null" ]] || fail "no cache last_success_at"
|
||||||
|
ok "data_version=$(json_get "$initial_meta" '.data_version') model_version=$(json_get "$initial_meta" '.model_version')"
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------- train small
|
||||||
|
poll_job() { # job_id, timeout seconds
|
||||||
|
local job=$1 timeout=$2 start now status
|
||||||
|
start=$(ms)
|
||||||
|
while :; do
|
||||||
|
curl -sS -m 10 -u "$CONTROL_USER:$CONTROL_PASS" \
|
||||||
|
"$API_URL/api/train/$job" > "$WORKDIR/job.json"
|
||||||
|
status=$(json_get "$WORKDIR/job.json" '.status')
|
||||||
|
if [[ "$status" == "succeeded" || "$status" == "failed" ]]; then return 0; fi
|
||||||
|
now=$(ms)
|
||||||
|
(( now - start < timeout * 1000 )) || return 1
|
||||||
|
sleep 0.05
|
||||||
|
done
|
||||||
|
}
|
||||||
|
|
||||||
|
run_train() { # model, timeout
|
||||||
|
local model=$1 timeout=$2 t0 submit_http status
|
||||||
|
t0=$(ms)
|
||||||
|
submit_http=$(curl -sS -m 60 -o "$WORKDIR/train-submit.json" -w '%{http_code}' \
|
||||||
|
-u "$CONTROL_USER:$CONTROL_PASS" \
|
||||||
|
-F "file=@$SMALL_CSV;type=text/csv" -F "model=$model" \
|
||||||
|
"$API_URL/api/train")
|
||||||
|
JOB_SUBMIT_MS=$(( $(ms) - t0 ))
|
||||||
|
JOB_SUBMIT_HTTP=$submit_http
|
||||||
|
[[ "$JOB_SUBMIT_HTTP" == "202" ]] || fail "POST /api/train ($model) returned $JOB_SUBMIT_HTTP: $(cat "$WORKDIR/train-submit.json")"
|
||||||
|
JOB_ID=$(json_get "$WORKDIR/train-submit.json" '.job_id')
|
||||||
|
[[ -n "$JOB_ID" && "$JOB_ID" != "null" ]] || fail "train response has no job_id"
|
||||||
|
poll_job "$JOB_ID" "$timeout" || fail "training job $JOB_ID ($model) did not finish within ${timeout}s"
|
||||||
|
JOB_TERMINAL_MS=$(( $(ms) - t0 ))
|
||||||
|
JOB_STATUS=$(json_get "$WORKDIR/job.json" '.status')
|
||||||
|
[[ "$JOB_STATUS" == "succeeded" ]] || fail "training job $JOB_ID ($model) status=$JOB_STATUS: $(cat "$WORKDIR/job.json")"
|
||||||
|
JOB_VERSION=$(json_get "$WORKDIR/job.json" '.model_version')
|
||||||
|
[[ -n "$JOB_VERSION" && "$JOB_VERSION" != "null" ]] || fail "job $JOB_ID has no model_version"
|
||||||
|
}
|
||||||
|
|
||||||
|
log "training model=small through POST /api/train (multipart CSV)"
|
||||||
|
run_train small 60
|
||||||
|
jq -e '.metrics.accuracy > 0.5 and .metrics.n_train > 0 and .metrics.n_test > 0 and .error == null' \
|
||||||
|
"$WORKDIR/job.json" >/dev/null || fail "small job metrics look wrong: $(cat "$WORKDIR/job.json")"
|
||||||
|
jq -n --argjson submit_http "$JOB_SUBMIT_HTTP" --argjson submit_ms "$JOB_SUBMIT_MS" \
|
||||||
|
--argjson terminal_ms "$JOB_TERMINAL_MS" --arg model small --slurpfile job "$WORKDIR/job.json" \
|
||||||
|
'{model:$model, submit_http:$submit_http, submit_ms:$submit_ms, submit_to_terminal_ms:$terminal_ms, job:$job[0]}' \
|
||||||
|
> "$RESULTS_DIR/train-small.json"
|
||||||
|
ok "job $JOB_ID succeeded in ${JOB_TERMINAL_MS}ms (submit ${JOB_SUBMIT_MS}ms); accuracy=$(json_get "$WORKDIR/job.json" '.metrics.accuracy') roc_auc=$(json_get "$WORKDIR/job.json" '.metrics.roc_auc') version=$JOB_VERSION"
|
||||||
|
|
||||||
|
forecast_before="$RESULTS_DIR/forecast-after-small.json"
|
||||||
|
code=$(get "/api/forecast?sensor=hn-stadtgalerie-01&horizon=7d&scenario=base" "$forecast_before")
|
||||||
|
[[ "$code" == "200" ]] || fail "forecast after small training returned $code"
|
||||||
|
[[ "$(json_get "$forecast_before" '.cached')" == "false" ]] || fail "forecast after training is unexpectedly cached"
|
||||||
|
[[ "$(json_get "$forecast_before" '.points | length')" == "168" ]] || fail "forecast after training has wrong point count"
|
||||||
|
ok "live forecast still works: 168 points, cached=false"
|
||||||
|
|
||||||
|
if [[ "$RUN_FORECAST" == "1" ]]; then
|
||||||
|
log "training model=forecast through POST /api/train (multipart CSV)"
|
||||||
|
run_train forecast 300
|
||||||
|
jq -n --argjson submit_http "$JOB_SUBMIT_HTTP" --argjson submit_ms "$JOB_SUBMIT_MS" \
|
||||||
|
--argjson terminal_ms "$JOB_TERMINAL_MS" --arg model forecast --slurpfile job "$WORKDIR/job.json" \
|
||||||
|
'{model:$model, submit_http:$submit_http, submit_ms:$submit_ms, submit_to_terminal_ms:$terminal_ms, job:$job[0]}' \
|
||||||
|
> "$RESULTS_DIR/train-forecast.json"
|
||||||
|
ok "job $JOB_ID succeeded in ${JOB_TERMINAL_MS}ms; mae=$(json_get "$WORKDIR/job.json" '.metrics.mae') wape=$(json_get "$WORKDIR/job.json" '.metrics.wape') version=$JOB_VERSION"
|
||||||
|
|
||||||
|
# The snapshot refresh carries the hot-swapped model version to /api/meta.
|
||||||
|
deadline=$(($(ms) + 30000))
|
||||||
|
while :; do
|
||||||
|
get "/api/meta" "$WORKDIR/meta.json" >/dev/null
|
||||||
|
[[ "$(json_get "$WORKDIR/meta.json" '.model_version')" == "$JOB_VERSION" ]] && break
|
||||||
|
(( $(ms) < deadline )) || fail "Go cache did not pick up the retrained model_version"
|
||||||
|
sleep 0.5
|
||||||
|
done
|
||||||
|
ok "Go /api/meta model_version refreshed to $JOB_VERSION"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------ staffing
|
||||||
|
log "timing staffing plan generation (contract: <2s)"
|
||||||
|
staffing_url="$API_URL/api/staffing?sensor=hn-stadtgalerie-01&scenario=base&standard=standard"
|
||||||
|
staffing_cold=$(curl -sS -m 10 -o "$WORKDIR/staffing-live.json" -w '%{time_total}' "$staffing_url")
|
||||||
|
staffing_warm=$(curl -sS -m 10 -o "$WORKDIR/staffing-warm.json" -w '%{time_total}' "$staffing_url")
|
||||||
|
[[ "$(json_get "$WORKDIR/staffing-live.json" '.hours | length')" == "70" ]] || fail "staffing plan does not cover 7 days x 10 store hours"
|
||||||
|
[[ "$(json_get "$WORKDIR/staffing-live.json" '.cached')" == "false" ]] || fail "live staffing plan is marked cached"
|
||||||
|
staffing_cold_ms=$(awk -v s="$staffing_cold" 'BEGIN { printf "%d", s*1000 }')
|
||||||
|
staffing_warm_ms=$(awk -v s="$staffing_warm" 'BEGIN { printf "%d", s*1000 }')
|
||||||
|
(( staffing_cold_ms < 2000 )) || fail "staffing cold plan took ${staffing_cold_ms}ms (>=2s)"
|
||||||
|
ok "staffing: cold ${staffing_cold_ms}ms, cached ${staffing_warm_ms}ms, 70 hours, coverage=$(json_get "$WORKDIR/staffing-live.json" '.summary.coverage_pct')%"
|
||||||
|
jq -n --argjson cold_ms "$staffing_cold_ms" --argjson warm_ms "$staffing_warm_ms" \
|
||||||
|
--slurpfile plan "$WORKDIR/staffing-live.json" \
|
||||||
|
'{cold_ms:$cold_ms, warm_ms:$warm_ms, hours:($plan[0].hours|length), shifts:($plan[0].shifts|length), coverage_pct:$plan[0].summary.coverage_pct, cost_eur:$plan[0].summary.cost_eur, cached:$plan[0].cached}' \
|
||||||
|
> "$RESULTS_DIR/staffing.json"
|
||||||
|
|
||||||
|
# -------------------------------------------------------------- SSE scenario
|
||||||
|
log "checking SSE scenario propagation, heartbeat and reconnect"
|
||||||
|
sse_log="$RESULTS_DIR/sse-stream.log"
|
||||||
|
: > "$sse_log"
|
||||||
|
curl -sN "$API_URL/api/stream" > "$sse_log" &
|
||||||
|
SSE_PID=$!
|
||||||
|
sleep 0.5
|
||||||
|
grep -q '^retry: 3000$' "$sse_log" || fail "stream does not send retry: 3000"
|
||||||
|
grep -q '^event: scenario$' "$sse_log" || fail "stream does not send the current scenario on connect"
|
||||||
|
ok "connect: retry: 3000 + initial scenario event"
|
||||||
|
|
||||||
|
t0=$(ms)
|
||||||
|
curl -sS -u "$CONTROL_USER:$CONTROL_PASS" -H 'Content-Type: application/json' \
|
||||||
|
-d '{"scenario":"sunny-saturday-market","weather":{"temp_delta_c":4,"precip_mm":0},"event":{"type":"market","uplift":0.2}}' \
|
||||||
|
"$API_URL/api/scenario" > "$WORKDIR/scenario.json"
|
||||||
|
posted_version=$(json_get "$WORKDIR/scenario.json" '.version')
|
||||||
|
[[ "${posted_version:-0}" -gt 1 ]] || fail "scenario POST did not return a new version: $(cat "$WORKDIR/scenario.json")"
|
||||||
|
propagation_ms=""
|
||||||
|
for _ in $(seq 1 200); do
|
||||||
|
if grep -q "^id: ${posted_version}$" "$sse_log"; then propagation_ms=$(( $(ms) - t0 )); break; fi
|
||||||
|
sleep 0.01
|
||||||
|
done
|
||||||
|
[[ -n "$propagation_ms" ]] || fail "scenario event did not reach the SSE client within 2s"
|
||||||
|
(( propagation_ms < 2000 )) || fail "scenario propagation took ${propagation_ms}ms (>2s)"
|
||||||
|
ok "scenario v$posted_version reached the client in ${propagation_ms}ms"
|
||||||
|
|
||||||
|
pings_before=$(grep -c '^event: ping$' "$sse_log" || true)
|
||||||
|
ping_t0=$(ms)
|
||||||
|
for _ in $(seq 1 180); do
|
||||||
|
if [[ "$(grep -c '^event: ping$' "$sse_log" || true)" -gt "$pings_before" ]]; then break; fi
|
||||||
|
sleep 0.1
|
||||||
|
done
|
||||||
|
[[ "$(grep -c '^event: ping$' "$sse_log" || true)" -gt "$pings_before" ]] || fail "no heartbeat within 18s"
|
||||||
|
heartbeat_ms=$(( $(ms) - ping_t0 ))
|
||||||
|
grep -q '^data: {"t":"' "$sse_log" || fail "heartbeat has no RFC3339 timestamp"
|
||||||
|
ok "heartbeat received ${heartbeat_ms}ms after the scenario check; server interval is 15s"
|
||||||
|
|
||||||
|
kill "$SSE_PID" 2>/dev/null || true
|
||||||
|
wait "$SSE_PID" 2>/dev/null || true
|
||||||
|
reconnect_t0=$(ms)
|
||||||
|
: > "$WORKDIR/sse-reconnect.log"
|
||||||
|
curl -sN "$API_URL/api/stream" > "$WORKDIR/sse-reconnect.log" &
|
||||||
|
SSE_PID=$!
|
||||||
|
reconnect_ms=""
|
||||||
|
for _ in $(seq 1 200); do
|
||||||
|
if grep -q "^id: ${posted_version}$" "$WORKDIR/sse-reconnect.log"; then reconnect_ms=$(( $(ms) - reconnect_t0 )); break; fi
|
||||||
|
sleep 0.01
|
||||||
|
done
|
||||||
|
kill "$SSE_PID" 2>/dev/null || true
|
||||||
|
wait "$SSE_PID" 2>/dev/null || true
|
||||||
|
[[ -n "$reconnect_ms" ]] || fail "reconnect did not receive the current scenario immediately"
|
||||||
|
ok "reconnect: current scenario v$posted_version delivered immediately"
|
||||||
|
|
||||||
|
jq -n --argjson version "$posted_version" --argjson propagation_ms "$propagation_ms" \
|
||||||
|
--argjson heartbeat_after_ms "$heartbeat_ms" --argjson reconnect_ms "$reconnect_ms" \
|
||||||
|
'{scenario_version:$version, propagation_ms:$propagation_ms, heartbeat_after_ms:$heartbeat_after_ms, heartbeat_interval_s:15, retry_ms:3000, reconnect_ms:$reconnect_ms, reconnect_current_version:$version}' \
|
||||||
|
> "$RESULTS_DIR/sse.json"
|
||||||
|
|
||||||
|
# ------------------------------------------------------- fallback + recovery
|
||||||
|
log "warming base forecast and staffing, then stopping FastAPI"
|
||||||
|
get "/api/forecast?sensor=hn-stadtgalerie-01&horizon=7d&scenario=base" "$WORKDIR/warm-forecast.json" >/dev/null
|
||||||
|
[[ "$(json_get "$WORKDIR/warm-forecast.json" '.cached')" == "false" ]] || fail "warm forecast is not live"
|
||||||
|
get "/api/meta" "$WORKDIR/meta-warm.json" >/dev/null
|
||||||
|
last_success_before=$(json_get "$WORKDIR/meta-warm.json" '.cache.last_success_at')
|
||||||
|
|
||||||
|
kill -TERM -- "-$ML_PID" 2>/dev/null || kill -TERM "$ML_PID" 2>/dev/null || true
|
||||||
|
fuser -k -TERM "$ML_PORT/tcp" >/dev/null 2>&1 || true
|
||||||
|
for _ in $(seq 1 100); do curl -sf -m 1 "$ML_URL/healthz" >/dev/null 2>&1 || break; sleep 0.1; done
|
||||||
|
curl -sf -m 1 "$ML_URL/healthz" >/dev/null 2>&1 && fail "FastAPI is still reachable after kill"
|
||||||
|
ok "FastAPI stopped"
|
||||||
|
|
||||||
|
deadline=$(($(ms) + 45000))
|
||||||
|
while :; do
|
||||||
|
get "/api/meta" "$WORKDIR/meta-down.json" >/dev/null
|
||||||
|
failures=$(json_get "$WORKDIR/meta-down.json" '.cache.failures')
|
||||||
|
if [[ "${failures:-0}" -ge 1 ]]; then break; fi
|
||||||
|
(( $(ms) < deadline )) || fail "cache did not record a failed refresh within 45s"
|
||||||
|
sleep 0.25
|
||||||
|
done
|
||||||
|
jq -e '.cache.failures >= 1 and .cache.last_error != null and .cache.next_retry_at != null' \
|
||||||
|
"$WORKDIR/meta-down.json" >/dev/null || fail "cache status missing errors/next_retry_at"
|
||||||
|
last_success_down=$(json_get "$WORKDIR/meta-down.json" '.cache.last_success_at')
|
||||||
|
ok "cache failures=$failures last_error=$(json_get "$WORKDIR/meta-down.json" '.cache.last_error') next_retry_at=$(json_get "$WORKDIR/meta-down.json" '.cache.next_retry_at')"
|
||||||
|
|
||||||
|
code=$(curl -sS -o "$WORKDIR/readyz-down.json" -w '%{http_code}' "$API_URL/api/readyz")
|
||||||
|
[[ "$code" == "200" ]] || fail "readyz failed while FastAPI is down (status $code)"
|
||||||
|
ok "readyz still 200 (last good snapshot)"
|
||||||
|
|
||||||
|
code=$(get "/api/sensors" "$WORKDIR/sensors-down.json")
|
||||||
|
[[ "$code" == "200" ]] || fail "/api/sensors returned $code while FastAPI is down"
|
||||||
|
[[ "$(json_get "$WORKDIR/sensors-down.json" 'length')" == "8" ]] || fail "/api/sensors did not return 8 sensors"
|
||||||
|
code=$(get "/api/context" "$WORKDIR/context-down.json")
|
||||||
|
[[ "$code" == "200" ]] || fail "/api/context returned $code while FastAPI is down"
|
||||||
|
[[ "$(json_get "$WORKDIR/context-down.json" '.cached')" == "true" ]] || fail "/api/context is not marked cached"
|
||||||
|
code=$(get "/api/series?sensor=hn-stadtgalerie-01&step=1h" "$WORKDIR/series-down.json")
|
||||||
|
[[ "$code" == "200" ]] || fail "/api/series returned $code while FastAPI is down"
|
||||||
|
ok "sensors/context/series still served from the last good snapshot (cached=true)"
|
||||||
|
|
||||||
|
code=$(get "/api/forecast?sensor=hn-stadtgalerie-01&horizon=7d&scenario=base" "$WORKDIR/forecast-down.json")
|
||||||
|
[[ "$code" == "200" ]] || fail "/api/forecast returned $code while FastAPI is down"
|
||||||
|
[[ "$(json_get "$WORKDIR/forecast-down.json" '.cached')" == "true" ]] || fail "fallback forecast is not marked cached"
|
||||||
|
[[ "$(json_get "$WORKDIR/forecast-down.json" '.points | length')" == "168" ]] || fail "fallback forecast has wrong point count"
|
||||||
|
ok "forecast serves the last good 168 points with cached=true"
|
||||||
|
|
||||||
|
code=$(get "/api/staffing?sensor=hn-stadtgalerie-01&scenario=base&standard=standard" "$WORKDIR/staffing-down.json")
|
||||||
|
[[ "$code" == "200" ]] || fail "/api/staffing returned $code while FastAPI is down"
|
||||||
|
[[ "$(json_get "$WORKDIR/staffing-down.json" '.cached')" == "true" ]] || fail "fallback staffing is not marked cached"
|
||||||
|
ok "staffing builds from the last good forecast with cached=true"
|
||||||
|
|
||||||
|
code=$(get "/api/forecast?sensor=density-01&horizon=15m&scenario=rainy-saturday" "$WORKDIR/forecast-cold.json")
|
||||||
|
[[ "$code" == "503" ]] || fail "uncached forecast returned $code, want 503"
|
||||||
|
ok "uncached forecast correctly returns 503 (no invented data)"
|
||||||
|
|
||||||
|
fallback_failures=$failures
|
||||||
|
fallback_error=$(json_get "$WORKDIR/meta-down.json" '.cache.last_error')
|
||||||
|
fallback_next=$(json_get "$WORKDIR/meta-down.json" '.cache.next_retry_at')
|
||||||
|
|
||||||
|
log "restarting FastAPI and waiting for recovery"
|
||||||
|
(
|
||||||
|
cd "$ML_DIR"
|
||||||
|
env CONTROL_TOKEN="$CONTROL_TOKEN" \
|
||||||
|
PULSECAST_MODELS="$WORKDIR/models" \
|
||||||
|
PULSECAST_UPLOADS="$WORKDIR/uploads" \
|
||||||
|
setsid uv run --frozen uvicorn pulsecast.server:app \
|
||||||
|
--host 127.0.0.1 --port "$ML_PORT" > "$ML_LOG" 2>&1 &
|
||||||
|
echo $! > "$WORKDIR/ml.pid"
|
||||||
|
)
|
||||||
|
ML_PID=$(cat "$WORKDIR/ml.pid")
|
||||||
|
wait_http "$ML_URL/readyz" 90 || fail "FastAPI did not restart; see $ML_LOG"
|
||||||
|
recovery_start=$(ms)
|
||||||
|
deadline=$(($(ms) + 120000))
|
||||||
|
while :; do
|
||||||
|
get "/api/meta" "$WORKDIR/meta-recovered.json" >/dev/null
|
||||||
|
failures=$(json_get "$WORKDIR/meta-recovered.json" '.cache.failures')
|
||||||
|
if [[ "${failures:-1}" == "0" ]]; then break; fi
|
||||||
|
(( $(ms) < deadline )) || fail "cache did not recover within 120s"
|
||||||
|
sleep 0.25
|
||||||
|
done
|
||||||
|
recovery_ms=$(( $(ms) - recovery_start ))
|
||||||
|
recovered_success=$(json_get "$WORKDIR/meta-recovered.json" '.cache.last_success_at')
|
||||||
|
[[ "$recovered_success" != "$last_success_down" ]] || fail "last_success_at did not advance after recovery"
|
||||||
|
code=$(get "/api/forecast?sensor=hn-stadtgalerie-01&horizon=7d&scenario=base" "$WORKDIR/forecast-recovered.json")
|
||||||
|
[[ "$code" == "200" ]] || fail "forecast after recovery returned $code"
|
||||||
|
[[ "$(json_get "$WORKDIR/forecast-recovered.json" '.cached')" == "false" ]] || fail "forecast after recovery is still cached"
|
||||||
|
ok "recovered after ${recovery_ms}ms: failures=0, last_success_at=$recovered_success"
|
||||||
|
|
||||||
|
jq -n \
|
||||||
|
--argjson fallback_failures "$fallback_failures" \
|
||||||
|
--arg fallback_error "$fallback_error" \
|
||||||
|
--arg fallback_next_retry "$fallback_next" \
|
||||||
|
--argjson fallback_forecast_cached "$(json_get "$WORKDIR/forecast-down.json" '.cached')" \
|
||||||
|
--argjson fallback_staffing_cached "$(json_get "$WORKDIR/staffing-down.json" '.cached')" \
|
||||||
|
--argjson recovery_ms "$recovery_ms" \
|
||||||
|
--arg last_success_before "$last_success_before" \
|
||||||
|
--arg last_success_down "$last_success_down" \
|
||||||
|
--arg last_success_after "$recovered_success" \
|
||||||
|
--argjson recovered_cached "$(json_get "$WORKDIR/forecast-recovered.json" '.cached')" \
|
||||||
|
'{fastapi_down:{cache_failures:$fallback_failures, last_error:$fallback_error, next_retry_at:$fallback_next_retry, last_success_at:$last_success_down, forecast_cached:$fallback_forecast_cached, staffing_cached:$fallback_staffing_cached, readyz:200, sensors:8}, recovery:{after_ms:$recovery_ms, last_success_before:$last_success_before, last_success_after:$last_success_after, forecast_cached:$recovered_cached, failures:0}}' \
|
||||||
|
> "$RESULTS_DIR/fallback.json"
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------ load test
|
||||||
|
if [[ "$RUN_LOAD" == "1" ]]; then
|
||||||
|
log "running the 120-client SSE load test (5 minutes)"
|
||||||
|
( cd "$ROOT" && CONTROL_USER="$CONTROL_USER" CONTROL_PASS="$CONTROL_PASS" \
|
||||||
|
go run ./scripts/sse-load -base "$API_URL" -clients 120 -duration 5m \
|
||||||
|
-changes 30s -json "$RESULTS_DIR/load.json" ) | tee "$RESULTS_DIR/load.txt"
|
||||||
|
jq -e '.passed == true' "$RESULTS_DIR/load.json" >/dev/null || fail "SSE load test failed; see $RESULTS_DIR/load.json"
|
||||||
|
ok "load test passed"
|
||||||
|
fi
|
||||||
|
|
||||||
|
log "all integration checks passed"
|
||||||
|
cat > "$RESULTS_DIR/summary.txt" <<EOF
|
||||||
|
PulseCast Go integration $(date -u +%Y-%m-%dT%H:%M:%SZ)
|
||||||
|
FastAPI $ML_URL, Go $API_URL
|
||||||
|
small CSV: $small_rows rows ($SMALL_FROM .. $SMALL_TO)
|
||||||
|
small job: $(jq -r '"status=\(.job.status) version=\(.job.model_version) accuracy=\(.job.metrics.accuracy) roc_auc=\(.job.metrics.roc_auc) submit_to_terminal_ms=\(.submit_to_terminal_ms)"' "$RESULTS_DIR/train-small.json")
|
||||||
|
staffing: $(jq -r '"cold_ms=\(.cold_ms) warm_ms=\(.warm_ms) hours=\(.hours) coverage_pct=\(.coverage_pct) cost_eur=\(.cost_eur)"' "$RESULTS_DIR/staffing.json")
|
||||||
|
SSE: $(jq -c '.' "$RESULTS_DIR/sse.json")
|
||||||
|
fallback: $(jq -c '.' "$RESULTS_DIR/fallback.json")
|
||||||
|
EOF
|
||||||
|
if [[ "$RUN_FORECAST" == "1" ]]; then
|
||||||
|
echo "forecast job: $(jq -r '"status=\(.job.status) version=\(.job.model_version) mae=\(.job.metrics.mae) wape=\(.job.metrics.wape) submit_to_terminal_ms=\(.submit_to_terminal_ms)"' "$RESULTS_DIR/train-forecast.json")" >> "$RESULTS_DIR/summary.txt"
|
||||||
|
fi
|
||||||
|
cat "$RESULTS_DIR/summary.txt"
|
||||||
Executable
+694
@@ -0,0 +1,694 @@
|
|||||||
|
// Command sse-load is the acceptance load test for GET /api/stream.
|
||||||
|
//
|
||||||
|
// It opens N concurrent SSE clients (120 by default) for a fixed duration,
|
||||||
|
// posts scenario changes on an interval, and measures the end-to-end latency
|
||||||
|
// from the POST to each client receiving the matching scenario event. It also
|
||||||
|
// verifies heartbeat cadence, counts connection drops/reconnects, and writes a
|
||||||
|
// machine-readable JSON summary.
|
||||||
|
//
|
||||||
|
// go run ./scripts/sse-load -base http://127.0.0.1:18080 -duration 5m
|
||||||
|
//
|
||||||
|
// Credentials come from -user/-pass or CONTROL_USER/CONTROL_PASS. The tool
|
||||||
|
// exits non-zero when a client drops, an event is not delivered, or the p95
|
||||||
|
// scenario latency exceeds 2 s: the contract's acceptance threshold.
|
||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bufio"
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"flag"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"math"
|
||||||
|
"net/http"
|
||||||
|
"os"
|
||||||
|
"os/signal"
|
||||||
|
"sort"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
"syscall"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
const scenarioThreshold = 2 * time.Second
|
||||||
|
|
||||||
|
// scenarioBodies rotate so consecutive changes differ in payload as well as
|
||||||
|
// version.
|
||||||
|
var scenarioBodies = []string{
|
||||||
|
`{"scenario":"sunny-saturday-market","weather":{"temp_delta_c":4,"precip_mm":0},"event":{"type":"market","uplift":0.2}}`,
|
||||||
|
`{"scenario":"rainy-saturday","weather":{"temp_delta_c":-3,"precip_mm":4},"event":{"type":"rain","uplift":-0.15}}`,
|
||||||
|
`{"scenario":"weekday-event","weather":{"temp_delta_c":0,"precip_mm":0},"event":{"type":"event","uplift":0.1}}`,
|
||||||
|
`{"scenario":"base","weather":{"temp_delta_c":0,"precip_mm":0},"event":{"type":"","uplift":0}}`,
|
||||||
|
}
|
||||||
|
|
||||||
|
type scenarioEvent struct {
|
||||||
|
Version int `json:"version"`
|
||||||
|
At string `json:"at"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type client struct {
|
||||||
|
id int
|
||||||
|
|
||||||
|
mu sync.Mutex
|
||||||
|
connected bool
|
||||||
|
receipts map[int]time.Time // scenario version -> first receipt
|
||||||
|
pings []time.Time
|
||||||
|
|
||||||
|
readyOnce sync.Once
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *client) setConnected(connected bool) {
|
||||||
|
c.mu.Lock()
|
||||||
|
c.connected = connected
|
||||||
|
c.mu.Unlock()
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *client) isConnected() bool {
|
||||||
|
c.mu.Lock()
|
||||||
|
defer c.mu.Unlock()
|
||||||
|
return c.connected
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *client) recordScenario(version int, at time.Time) {
|
||||||
|
c.mu.Lock()
|
||||||
|
if c.receipts == nil {
|
||||||
|
c.receipts = make(map[int]time.Time)
|
||||||
|
}
|
||||||
|
if _, exists := c.receipts[version]; !exists {
|
||||||
|
c.receipts[version] = at
|
||||||
|
}
|
||||||
|
c.mu.Unlock()
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *client) recordPing(at time.Time) {
|
||||||
|
c.mu.Lock()
|
||||||
|
c.pings = append(c.pings, at)
|
||||||
|
c.mu.Unlock()
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *client) receipt(version int) (time.Time, bool) {
|
||||||
|
c.mu.Lock()
|
||||||
|
defer c.mu.Unlock()
|
||||||
|
at, ok := c.receipts[version]
|
||||||
|
return at, ok
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *client) pingCount() int {
|
||||||
|
c.mu.Lock()
|
||||||
|
defer c.mu.Unlock()
|
||||||
|
return len(c.pings)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *client) signalReady(ready chan<- struct{}) {
|
||||||
|
c.readyOnce.Do(func() {
|
||||||
|
select {
|
||||||
|
case ready <- struct{}{}:
|
||||||
|
default:
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
type changeResult struct {
|
||||||
|
Version int `json:"version"`
|
||||||
|
PostAt time.Time `json:"post_at"`
|
||||||
|
Connected int `json:"connected_at_post"`
|
||||||
|
Received int `json:"received"`
|
||||||
|
Missing int `json:"missing"`
|
||||||
|
LatenciesMS []float64 `json:"-"`
|
||||||
|
P50MS float64 `json:"p50_ms"`
|
||||||
|
P95MS float64 `json:"p95_ms"`
|
||||||
|
P99MS float64 `json:"p99_ms"`
|
||||||
|
MaxMS float64 `json:"max_ms"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type summary struct {
|
||||||
|
StartedAt time.Time `json:"started_at"`
|
||||||
|
DurationS float64 `json:"duration_s"`
|
||||||
|
BaseURL string `json:"base_url"`
|
||||||
|
Clients int `json:"clients_configured"`
|
||||||
|
PeakConnected int `json:"clients_peak_connected"`
|
||||||
|
ConnectedAfter int `json:"clients_connected_after_warmup"`
|
||||||
|
|
||||||
|
Drops int `json:"connection_drops"`
|
||||||
|
PostErrors int `json:"scenario_post_errors"`
|
||||||
|
|
||||||
|
ScenarioChanges int `json:"scenario_changes"`
|
||||||
|
Received int `json:"scenario_events_received"`
|
||||||
|
Missing int `json:"scenario_events_missing"`
|
||||||
|
LatencyCount int `json:"scenario_latency_samples"`
|
||||||
|
LatencyP50MS float64 `json:"scenario_p50_ms"`
|
||||||
|
LatencyP95MS float64 `json:"scenario_p95_ms"`
|
||||||
|
LatencyP99MS float64 `json:"scenario_p99_ms"`
|
||||||
|
LatencyMaxMS float64 `json:"scenario_max_ms"`
|
||||||
|
Changes []changeResult `json:"changes"`
|
||||||
|
Ping pingSummary `json:"heartbeat"`
|
||||||
|
Errors []string `json:"errors"`
|
||||||
|
Passed bool `json:"passed"`
|
||||||
|
FailureReasons []string `json:"failure_reasons,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type pingSummary struct {
|
||||||
|
Unique int `json:"unique_server_timestamps"`
|
||||||
|
MinIntervalMS float64 `json:"min_interval_ms"`
|
||||||
|
MedianIntervalMS float64 `json:"median_interval_ms"`
|
||||||
|
P95IntervalMS float64 `json:"p95_interval_ms"`
|
||||||
|
MaxIntervalMS float64 `json:"max_interval_ms"`
|
||||||
|
PerClientMin int `json:"per_client_min"`
|
||||||
|
PerClientMedian int `json:"per_client_median"`
|
||||||
|
PerClientMax int `json:"per_client_max"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type recorder struct {
|
||||||
|
mu sync.Mutex
|
||||||
|
clients []*client
|
||||||
|
connected int
|
||||||
|
peak int
|
||||||
|
drops int
|
||||||
|
postErrs int
|
||||||
|
errors []string
|
||||||
|
pingTimes map[time.Time]struct{}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *recorder) connectedNow() int {
|
||||||
|
r.mu.Lock()
|
||||||
|
defer r.mu.Unlock()
|
||||||
|
return r.connected
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *recorder) addConnected(delta int) {
|
||||||
|
r.mu.Lock()
|
||||||
|
r.connected += delta
|
||||||
|
if r.connected > r.peak {
|
||||||
|
r.peak = r.connected
|
||||||
|
}
|
||||||
|
r.mu.Unlock()
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *recorder) addError(message string, drop bool) {
|
||||||
|
r.mu.Lock()
|
||||||
|
if drop {
|
||||||
|
r.drops++
|
||||||
|
} else {
|
||||||
|
r.postErrs++
|
||||||
|
}
|
||||||
|
if len(r.errors) < 40 {
|
||||||
|
r.errors = append(r.errors, message)
|
||||||
|
}
|
||||||
|
r.mu.Unlock()
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *recorder) addPing(t time.Time) {
|
||||||
|
r.mu.Lock()
|
||||||
|
if r.pingTimes == nil {
|
||||||
|
r.pingTimes = make(map[time.Time]struct{})
|
||||||
|
}
|
||||||
|
r.pingTimes[t] = struct{}{}
|
||||||
|
r.mu.Unlock()
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *recorder) snapshot() (peak, drops, postErrs int, errs []string, pings []time.Time) {
|
||||||
|
r.mu.Lock()
|
||||||
|
defer r.mu.Unlock()
|
||||||
|
peak, drops, postErrs = r.peak, r.drops, r.postErrs
|
||||||
|
errs = append([]string(nil), r.errors...)
|
||||||
|
pings = make([]time.Time, 0, len(r.pingTimes))
|
||||||
|
for t := range r.pingTimes {
|
||||||
|
pings = append(pings, t)
|
||||||
|
}
|
||||||
|
sort.Slice(pings, func(i, j int) bool { return pings[i].Before(pings[j]) })
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
func main() {
|
||||||
|
base := flag.String("base", "http://127.0.0.1:18080", "Go API base URL")
|
||||||
|
clientsN := flag.Int("clients", 120, "concurrent SSE clients")
|
||||||
|
duration := flag.Duration("duration", 5*time.Minute, "load-test duration")
|
||||||
|
changeEvery := flag.Duration("changes", 30*time.Second, "interval between scenario POSTs; 0 disables")
|
||||||
|
warmup := flag.Duration("warmup", 10*time.Second, "max time to wait for all clients to connect")
|
||||||
|
user := flag.String("user", os.Getenv("CONTROL_USER"), "basic-auth user for POST /api/scenario")
|
||||||
|
pass := flag.String("pass", os.Getenv("CONTROL_PASS"), "basic-auth password for POST /api/scenario")
|
||||||
|
jsonOut := flag.String("json", "", "write the JSON summary to this path")
|
||||||
|
flag.Parse()
|
||||||
|
|
||||||
|
if *clientsN < 1 {
|
||||||
|
fatalf("clients must be >= 1")
|
||||||
|
}
|
||||||
|
if *duration <= 0 {
|
||||||
|
fatalf("duration must be > 0")
|
||||||
|
}
|
||||||
|
if *changeEvery > 0 && (*user == "" || *pass == "") {
|
||||||
|
fatalf("POST /api/scenario needs -user/-pass or CONTROL_USER/CONTROL_PASS")
|
||||||
|
}
|
||||||
|
|
||||||
|
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
|
||||||
|
defer stop()
|
||||||
|
ctx, cancel := context.WithTimeout(ctx, *duration)
|
||||||
|
defer cancel()
|
||||||
|
|
||||||
|
rec := &recorder{clients: make([]*client, *clientsN), pingTimes: make(map[time.Time]struct{})}
|
||||||
|
ready := make(chan struct{}, *clientsN)
|
||||||
|
started := time.Now()
|
||||||
|
|
||||||
|
var wg sync.WaitGroup
|
||||||
|
for i := 0; i < *clientsN; i++ {
|
||||||
|
c := &client{id: i}
|
||||||
|
rec.clients[i] = c
|
||||||
|
wg.Add(1)
|
||||||
|
go func() {
|
||||||
|
defer wg.Done()
|
||||||
|
runClient(ctx, *base, c, rec, ready)
|
||||||
|
}()
|
||||||
|
}
|
||||||
|
|
||||||
|
// Wait until every client has received the initial scenario event.
|
||||||
|
connectedAfter := waitReady(ctx, ready, *clientsN, *warmup)
|
||||||
|
fmt.Printf("clients connected: %d/%d after warmup\n", connectedAfter, *clientsN)
|
||||||
|
if connectedAfter < *clientsN {
|
||||||
|
fmt.Printf("warning: only %d/%d clients connected before warmup deadline\n", connectedAfter, *clientsN)
|
||||||
|
}
|
||||||
|
|
||||||
|
changes := []changeResult{}
|
||||||
|
if *changeEvery > 0 {
|
||||||
|
changes = runChanges(ctx, *base, *user, *pass, *changeEvery, rec)
|
||||||
|
}
|
||||||
|
|
||||||
|
<-ctx.Done()
|
||||||
|
cancel()
|
||||||
|
done := make(chan struct{})
|
||||||
|
go func() { wg.Wait(); close(done) }()
|
||||||
|
select {
|
||||||
|
case <-done:
|
||||||
|
case <-time.After(5 * time.Second):
|
||||||
|
fmt.Println("warning: clients did not stop within 5s")
|
||||||
|
}
|
||||||
|
|
||||||
|
elapsed := time.Since(started)
|
||||||
|
sum := buildSummary(rec, changes, *base, *clientsN, connectedAfter, started, elapsed)
|
||||||
|
printSummary(sum)
|
||||||
|
if *jsonOut != "" {
|
||||||
|
if err := writeJSON(*jsonOut, sum); err != nil {
|
||||||
|
fatalf("write %s: %v", *jsonOut, err)
|
||||||
|
}
|
||||||
|
fmt.Printf("json written to %s\n", *jsonOut)
|
||||||
|
}
|
||||||
|
if !sum.Passed {
|
||||||
|
os.Exit(1)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func waitReady(ctx context.Context, ready <-chan struct{}, want int, timeout time.Duration) int {
|
||||||
|
timer := time.NewTimer(timeout)
|
||||||
|
defer timer.Stop()
|
||||||
|
count := 0
|
||||||
|
for count < want {
|
||||||
|
select {
|
||||||
|
case <-ready:
|
||||||
|
count++
|
||||||
|
case <-timer.C:
|
||||||
|
return count
|
||||||
|
case <-ctx.Done():
|
||||||
|
return count
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return count
|
||||||
|
}
|
||||||
|
|
||||||
|
func runClient(ctx context.Context, base string, c *client, rec *recorder, ready chan<- struct{}) {
|
||||||
|
transport := &http.Transport{
|
||||||
|
Proxy: nil,
|
||||||
|
DisableCompression: true,
|
||||||
|
MaxIdleConnsPerHost: 1,
|
||||||
|
IdleConnTimeout: 0,
|
||||||
|
}
|
||||||
|
httpClient := &http.Client{Transport: transport}
|
||||||
|
defer transport.CloseIdleConnections()
|
||||||
|
|
||||||
|
streamURL := strings.TrimRight(base, "/") + "/api/stream"
|
||||||
|
failures := 0
|
||||||
|
for ctx.Err() == nil {
|
||||||
|
req, err := http.NewRequestWithContext(ctx, http.MethodGet, streamURL, nil)
|
||||||
|
if err != nil {
|
||||||
|
rec.addError(fmt.Sprintf("client %d: build request: %v", c.id, err), true)
|
||||||
|
c.signalReady(ready)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
req.Header.Set("Accept", "text/event-stream")
|
||||||
|
resp, err := httpClient.Do(req)
|
||||||
|
if err != nil {
|
||||||
|
if ctx.Err() != nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
failures++
|
||||||
|
rec.addError(fmt.Sprintf("client %d: connect: %v", c.id, err), true)
|
||||||
|
c.signalReady(ready)
|
||||||
|
sleepRetry(ctx)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if resp.StatusCode != http.StatusOK {
|
||||||
|
snippet, _ := io.ReadAll(io.LimitReader(resp.Body, 128))
|
||||||
|
_ = resp.Body.Close()
|
||||||
|
failures++
|
||||||
|
rec.addError(fmt.Sprintf("client %d: status %d: %s", c.id, resp.StatusCode, strings.TrimSpace(string(snippet))), true)
|
||||||
|
c.signalReady(ready)
|
||||||
|
sleepRetry(ctx)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
c.setConnected(true)
|
||||||
|
rec.addConnected(1)
|
||||||
|
readStream(ctx, resp.Body, c, rec, ready)
|
||||||
|
_ = resp.Body.Close()
|
||||||
|
c.setConnected(false)
|
||||||
|
rec.addConnected(-1)
|
||||||
|
|
||||||
|
if ctx.Err() != nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
failures++
|
||||||
|
rec.addError(fmt.Sprintf("client %d: stream ended unexpectedly", c.id), true)
|
||||||
|
c.signalReady(ready)
|
||||||
|
sleepRetry(ctx)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// readStream parses one SSE response. It returns when the body ends or the
|
||||||
|
// context is cancelled.
|
||||||
|
func readStream(ctx context.Context, body io.Reader, c *client, rec *recorder, ready chan<- struct{}) {
|
||||||
|
scanner := bufio.NewScanner(body)
|
||||||
|
scanner.Buffer(make([]byte, 0, 64*1024), 1<<20)
|
||||||
|
|
||||||
|
eventName := ""
|
||||||
|
eventID := ""
|
||||||
|
var data []string
|
||||||
|
for ctx.Err() == nil && scanner.Scan() {
|
||||||
|
line := scanner.Text()
|
||||||
|
if line == "" {
|
||||||
|
now := time.Now()
|
||||||
|
switch eventName {
|
||||||
|
case "scenario":
|
||||||
|
version := 0
|
||||||
|
var event scenarioEvent
|
||||||
|
if err := json.Unmarshal([]byte(strings.Join(data, "\n")), &event); err == nil {
|
||||||
|
version = event.Version
|
||||||
|
}
|
||||||
|
if version == 0 && eventID != "" {
|
||||||
|
_, _ = fmt.Sscanf(eventID, "%d", &version)
|
||||||
|
}
|
||||||
|
if version > 0 {
|
||||||
|
c.recordScenario(version, now)
|
||||||
|
c.signalReady(ready)
|
||||||
|
}
|
||||||
|
case "ping":
|
||||||
|
c.recordPing(now)
|
||||||
|
var payload struct {
|
||||||
|
T string `json:"t"`
|
||||||
|
}
|
||||||
|
if err := json.Unmarshal([]byte(strings.Join(data, "\n")), &payload); err == nil {
|
||||||
|
if t, err := time.Parse(time.RFC3339, payload.T); err == nil {
|
||||||
|
rec.addPing(t.UTC())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
eventName, eventID, data = "", "", nil
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
switch {
|
||||||
|
case strings.HasPrefix(line, "event:"):
|
||||||
|
eventName = strings.TrimSpace(strings.TrimPrefix(line, "event:"))
|
||||||
|
case strings.HasPrefix(line, "id:"):
|
||||||
|
eventID = strings.TrimSpace(strings.TrimPrefix(line, "id:"))
|
||||||
|
case strings.HasPrefix(line, "data:"):
|
||||||
|
data = append(data, strings.TrimPrefix(line, "data:"))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func runChanges(ctx context.Context, base, user, pass string, every time.Duration, rec *recorder) []changeResult {
|
||||||
|
client := &http.Client{Timeout: 10 * time.Second}
|
||||||
|
scenarioURL := strings.TrimRight(base, "/") + "/api/scenario"
|
||||||
|
|
||||||
|
var out []changeResult
|
||||||
|
ticker := time.NewTicker(every)
|
||||||
|
defer ticker.Stop()
|
||||||
|
|
||||||
|
for i := 0; ; i++ {
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return out
|
||||||
|
case <-ticker.C:
|
||||||
|
}
|
||||||
|
body := scenarioBodies[i%len(scenarioBodies)]
|
||||||
|
postAt := time.Now()
|
||||||
|
version, err := postScenario(ctx, client, scenarioURL, user, pass, body)
|
||||||
|
if err != nil {
|
||||||
|
if ctx.Err() != nil {
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
rec.addError("scenario POST: "+err.Error(), false)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
result := changeResult{
|
||||||
|
Version: version,
|
||||||
|
PostAt: postAt.UTC(),
|
||||||
|
Connected: rec.connectedNow(),
|
||||||
|
}
|
||||||
|
|
||||||
|
// Wait up to 2s for every currently connected client to receive it.
|
||||||
|
deadline := time.Now().Add(2 * time.Second)
|
||||||
|
for {
|
||||||
|
result.Received = 0
|
||||||
|
for _, c := range rec.clients {
|
||||||
|
if !c.isConnected() {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if _, ok := c.receipt(version); ok {
|
||||||
|
result.Received++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
result.Missing = result.Connected - result.Received
|
||||||
|
if result.Missing <= 0 || time.Now().After(deadline) {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
time.Sleep(10 * time.Millisecond)
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, c := range rec.clients {
|
||||||
|
if !c.isConnected() {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if at, ok := c.receipt(version); ok {
|
||||||
|
result.LatenciesMS = append(result.LatenciesMS,
|
||||||
|
float64(at.Sub(postAt).Microseconds())/1000.0)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
result.P50MS, result.P95MS, result.P99MS, result.MaxMS = percentiles(result.LatenciesMS)
|
||||||
|
out = append(out, result)
|
||||||
|
fmt.Printf("scenario v%d: %d/%d clients p50=%.1fms p95=%.1fms max=%.1fms\n",
|
||||||
|
version, result.Received, result.Connected, result.P50MS, result.P95MS, result.MaxMS)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func postScenario(ctx context.Context, client *http.Client, url, user, pass, body string) (int, error) {
|
||||||
|
req, err := http.NewRequestWithContext(ctx, http.MethodPost, url, bytes.NewBufferString(body))
|
||||||
|
if err != nil {
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
req.Header.Set("Content-Type", "application/json")
|
||||||
|
req.SetBasicAuth(user, pass)
|
||||||
|
resp, err := client.Do(req)
|
||||||
|
if err != nil {
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
defer resp.Body.Close()
|
||||||
|
raw, _ := io.ReadAll(io.LimitReader(resp.Body, 4096))
|
||||||
|
if resp.StatusCode != http.StatusOK {
|
||||||
|
return 0, fmt.Errorf("status %d: %s", resp.StatusCode, strings.TrimSpace(string(raw)))
|
||||||
|
}
|
||||||
|
var parsed struct {
|
||||||
|
Version int `json:"version"`
|
||||||
|
}
|
||||||
|
if err := json.Unmarshal(raw, &parsed); err != nil {
|
||||||
|
return 0, fmt.Errorf("decode response: %w", err)
|
||||||
|
}
|
||||||
|
if parsed.Version <= 0 {
|
||||||
|
return 0, errors.New("response has no version")
|
||||||
|
}
|
||||||
|
return parsed.Version, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func buildSummary(rec *recorder, changes []changeResult, base string, clientsN, connectedAfter int, started time.Time, elapsed time.Duration) summary {
|
||||||
|
peak, drops, postErrs, errs, pingTimes := rec.snapshot()
|
||||||
|
|
||||||
|
sum := summary{
|
||||||
|
StartedAt: started.UTC(),
|
||||||
|
DurationS: math.Round(elapsed.Seconds()*1000) / 1000,
|
||||||
|
BaseURL: base,
|
||||||
|
Clients: clientsN,
|
||||||
|
PeakConnected: peak,
|
||||||
|
ConnectedAfter: connectedAfter,
|
||||||
|
Drops: drops,
|
||||||
|
PostErrors: postErrs,
|
||||||
|
ScenarioChanges: len(changes),
|
||||||
|
Changes: changes,
|
||||||
|
Errors: errs,
|
||||||
|
}
|
||||||
|
var all []float64
|
||||||
|
for _, change := range changes {
|
||||||
|
all = append(all, change.LatenciesMS...)
|
||||||
|
sum.Received += change.Received
|
||||||
|
sum.Missing += change.Missing
|
||||||
|
}
|
||||||
|
sum.LatencyCount = len(all)
|
||||||
|
sum.LatencyP50MS, sum.LatencyP95MS, sum.LatencyP99MS, sum.LatencyMaxMS = percentiles(all)
|
||||||
|
sum.Ping = buildPingSummary(pingTimes, rec.clients)
|
||||||
|
|
||||||
|
// Pass criteria from the API contract: 120 clients for 5 minutes, no dropped
|
||||||
|
// connections, p95 scenario latency below 2 s.
|
||||||
|
if peak < clientsN {
|
||||||
|
sum.FailureReasons = append(sum.FailureReasons, fmt.Sprintf("peak connections %d < %d", peak, clientsN))
|
||||||
|
}
|
||||||
|
if drops > 0 {
|
||||||
|
sum.FailureReasons = append(sum.FailureReasons, fmt.Sprintf("%d dropped/reconnected connections", drops))
|
||||||
|
}
|
||||||
|
if postErrs > 0 {
|
||||||
|
sum.FailureReasons = append(sum.FailureReasons, fmt.Sprintf("%d scenario POST errors", postErrs))
|
||||||
|
}
|
||||||
|
if sum.Missing > 0 {
|
||||||
|
sum.FailureReasons = append(sum.FailureReasons, fmt.Sprintf("%d scenario events missing", sum.Missing))
|
||||||
|
}
|
||||||
|
if len(all) > 0 && sum.LatencyP95MS > scenarioThreshold.Seconds()*1000 {
|
||||||
|
sum.FailureReasons = append(sum.FailureReasons, fmt.Sprintf("scenario p95 %.1fms > 2000ms", sum.LatencyP95MS))
|
||||||
|
}
|
||||||
|
if sum.Ping.P95IntervalMS > 17_000 {
|
||||||
|
sum.FailureReasons = append(sum.FailureReasons, fmt.Sprintf("heartbeat p95 interval %.0fms > 17000ms", sum.Ping.P95IntervalMS))
|
||||||
|
}
|
||||||
|
sum.Passed = len(sum.FailureReasons) == 0
|
||||||
|
return sum
|
||||||
|
}
|
||||||
|
|
||||||
|
func buildPingSummary(pingTimes []time.Time, clients []*client) pingSummary {
|
||||||
|
out := pingSummary{Unique: len(pingTimes)}
|
||||||
|
if len(pingTimes) > 1 {
|
||||||
|
intervals := make([]float64, 0, len(pingTimes)-1)
|
||||||
|
for i := 1; i < len(pingTimes); i++ {
|
||||||
|
intervals = append(intervals, float64(pingTimes[i].Sub(pingTimes[i-1]).Microseconds())/1000.0)
|
||||||
|
}
|
||||||
|
out.MinIntervalMS = minFloat(intervals)
|
||||||
|
out.MedianIntervalMS = percentile(intervals, 50)
|
||||||
|
out.P95IntervalMS = percentile(intervals, 95)
|
||||||
|
out.MaxIntervalMS = maxFloat(intervals)
|
||||||
|
}
|
||||||
|
counts := make([]int, 0, len(clients))
|
||||||
|
for _, c := range clients {
|
||||||
|
counts = append(counts, c.pingCount())
|
||||||
|
}
|
||||||
|
if len(counts) > 0 {
|
||||||
|
sort.Ints(counts)
|
||||||
|
out.PerClientMin = counts[0]
|
||||||
|
out.PerClientMedian = counts[len(counts)/2]
|
||||||
|
out.PerClientMax = counts[len(counts)-1]
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func percentiles(values []float64) (p50, p95, p99, max float64) {
|
||||||
|
if len(values) == 0 {
|
||||||
|
return 0, 0, 0, 0
|
||||||
|
}
|
||||||
|
return percentile(values, 50), percentile(values, 95), percentile(values, 99), maxFloat(values)
|
||||||
|
}
|
||||||
|
|
||||||
|
// percentile uses the nearest-rank method on a sorted copy.
|
||||||
|
func percentile(values []float64, p float64) float64 {
|
||||||
|
if len(values) == 0 {
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
sorted := append([]float64(nil), values...)
|
||||||
|
sort.Float64s(sorted)
|
||||||
|
rank := int(math.Ceil(p/100*float64(len(sorted)))) - 1
|
||||||
|
if rank < 0 {
|
||||||
|
rank = 0
|
||||||
|
}
|
||||||
|
if rank >= len(sorted) {
|
||||||
|
rank = len(sorted) - 1
|
||||||
|
}
|
||||||
|
return math.Round(sorted[rank]*10) / 10
|
||||||
|
}
|
||||||
|
|
||||||
|
func minFloat(values []float64) float64 {
|
||||||
|
m := values[0]
|
||||||
|
for _, v := range values[1:] {
|
||||||
|
if v < m {
|
||||||
|
m = v
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return m
|
||||||
|
}
|
||||||
|
|
||||||
|
func maxFloat(values []float64) float64 {
|
||||||
|
m := values[0]
|
||||||
|
for _, v := range values[1:] {
|
||||||
|
if v > m {
|
||||||
|
m = v
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return m
|
||||||
|
}
|
||||||
|
|
||||||
|
func sleepRetry(ctx context.Context) {
|
||||||
|
timer := time.NewTimer(3 * time.Second)
|
||||||
|
defer timer.Stop()
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
case <-timer.C:
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func printSummary(sum summary) {
|
||||||
|
fmt.Println()
|
||||||
|
fmt.Println("=== SSE load test ===")
|
||||||
|
fmt.Printf("duration: %.1fs\n", sum.DurationS)
|
||||||
|
fmt.Printf("clients: %d configured, %d peak connected, %d after warmup\n",
|
||||||
|
sum.Clients, sum.PeakConnected, sum.ConnectedAfter)
|
||||||
|
fmt.Printf("connections: %d drops/reconnects, %d scenario POST errors\n", sum.Drops, sum.PostErrors)
|
||||||
|
fmt.Printf("scenario changes: %d (%d events received, %d missing)\n",
|
||||||
|
sum.ScenarioChanges, sum.Received, sum.Missing)
|
||||||
|
fmt.Printf("scenario latency: p50=%.1fms p95=%.1fms p99=%.1fms max=%.1fms (n=%d)\n",
|
||||||
|
sum.LatencyP50MS, sum.LatencyP95MS, sum.LatencyP99MS, sum.LatencyMaxMS, sum.LatencyCount)
|
||||||
|
fmt.Printf("heartbeat: %d server pings, interval min=%.0fms median=%.0fms p95=%.0fms max=%.0fms\n",
|
||||||
|
sum.Ping.Unique, sum.Ping.MinIntervalMS, sum.Ping.MedianIntervalMS,
|
||||||
|
sum.Ping.P95IntervalMS, sum.Ping.MaxIntervalMS)
|
||||||
|
fmt.Printf("heartbeat/client: min=%d median=%d max=%d\n",
|
||||||
|
sum.Ping.PerClientMin, sum.Ping.PerClientMedian, sum.Ping.PerClientMax)
|
||||||
|
if len(sum.Errors) > 0 {
|
||||||
|
fmt.Println("errors:")
|
||||||
|
for _, err := range sum.Errors {
|
||||||
|
fmt.Printf(" - %s\n", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if sum.Passed {
|
||||||
|
fmt.Println("result: PASS")
|
||||||
|
} else {
|
||||||
|
fmt.Println("result: FAIL")
|
||||||
|
for _, reason := range sum.FailureReasons {
|
||||||
|
fmt.Printf(" - %s\n", reason)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeJSON(path string, sum summary) error {
|
||||||
|
raw, err := json.MarshalIndent(sum, "", " ")
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
raw = append(raw, '\n')
|
||||||
|
return os.WriteFile(path, raw, 0o644)
|
||||||
|
}
|
||||||
|
|
||||||
|
func fatalf(format string, args ...any) {
|
||||||
|
fmt.Fprintf(os.Stderr, "sse-load: "+format+"\n", args...)
|
||||||
|
os.Exit(2)
|
||||||
|
}
|
||||||
@@ -0,0 +1,124 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"math"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestPercentileNearestRank(t *testing.T) {
|
||||||
|
values := []float64{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}
|
||||||
|
cases := []struct {
|
||||||
|
p float64
|
||||||
|
want float64
|
||||||
|
}{
|
||||||
|
{50, 5},
|
||||||
|
{95, 10},
|
||||||
|
{99, 10},
|
||||||
|
{0, 1},
|
||||||
|
{100, 10},
|
||||||
|
}
|
||||||
|
for _, tc := range cases {
|
||||||
|
if got := percentile(values, tc.p); got != tc.want {
|
||||||
|
t.Fatalf("percentile(%v, %v)=%v, want %v", values, tc.p, got, tc.want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if got := percentile(nil, 95); got != 0 {
|
||||||
|
t.Fatalf("empty percentile=%v, want 0", got)
|
||||||
|
}
|
||||||
|
|
||||||
|
p50, p95, p99, max := percentiles(values)
|
||||||
|
if p50 != 5 || p95 != 10 || p99 != 10 || max != 10 {
|
||||||
|
t.Fatalf("percentiles=(%v,%v,%v,%v)", p50, p95, p99, max)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPercentileDoesNotMutateInput(t *testing.T) {
|
||||||
|
values := []float64{3, 1, 2}
|
||||||
|
_ = percentile(values, 50)
|
||||||
|
if values[0] != 3 || values[1] != 1 || values[2] != 2 {
|
||||||
|
t.Fatalf("input was reordered: %v", values)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBuildPingSummary(t *testing.T) {
|
||||||
|
start := time.Date(2026, 11, 14, 10, 0, 0, 0, time.UTC)
|
||||||
|
var pings []time.Time
|
||||||
|
for i := 0; i < 5; i++ {
|
||||||
|
pings = append(pings, start.Add(time.Duration(i)*15*time.Second))
|
||||||
|
}
|
||||||
|
clients := []*client{
|
||||||
|
{pings: pings[:3]},
|
||||||
|
{pings: pings[:4]},
|
||||||
|
{pings: pings},
|
||||||
|
}
|
||||||
|
got := buildPingSummary(pings, clients)
|
||||||
|
if got.Unique != 5 {
|
||||||
|
t.Fatalf("unique=%d, want 5", got.Unique)
|
||||||
|
}
|
||||||
|
if math.Abs(got.MedianIntervalMS-15000) > 0.1 {
|
||||||
|
t.Fatalf("median interval=%v, want 15000", got.MedianIntervalMS)
|
||||||
|
}
|
||||||
|
if got.MaxIntervalMS != 15000 || got.MinIntervalMS != 15000 {
|
||||||
|
t.Fatalf("interval range=[%v,%v], want [15000,15000]", got.MinIntervalMS, got.MaxIntervalMS)
|
||||||
|
}
|
||||||
|
if got.PerClientMin != 3 || got.PerClientMedian != 4 || got.PerClientMax != 5 {
|
||||||
|
t.Fatalf("per-client=(%d,%d,%d), want (3,4,5)",
|
||||||
|
got.PerClientMin, got.PerClientMedian, got.PerClientMax)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSummaryPassAndFail(t *testing.T) {
|
||||||
|
started := time.Now()
|
||||||
|
makeRecorder := func(peak, drops, postErrs int) *recorder {
|
||||||
|
return &recorder{
|
||||||
|
peak: peak,
|
||||||
|
drops: drops,
|
||||||
|
postErrs: postErrs,
|
||||||
|
pingTimes: make(map[time.Time]struct{}),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
changes := []changeResult{{
|
||||||
|
Version: 2,
|
||||||
|
Connected: 120,
|
||||||
|
Received: 120,
|
||||||
|
LatenciesMS: []float64{4, 6, 8},
|
||||||
|
}}
|
||||||
|
pass := buildSummary(makeRecorder(120, 0, 0), changes, "http://example", 120, 120, started, 5*time.Minute)
|
||||||
|
if !pass.Passed {
|
||||||
|
t.Fatalf("expected pass, reasons=%v", pass.FailureReasons)
|
||||||
|
}
|
||||||
|
|
||||||
|
fail := buildSummary(makeRecorder(119, 1, 1), changes, "http://example", 120, 120, started, 5*time.Minute)
|
||||||
|
if fail.Passed {
|
||||||
|
t.Fatal("expected failure for low peak and drops")
|
||||||
|
}
|
||||||
|
if len(fail.FailureReasons) < 3 {
|
||||||
|
t.Fatalf("failure reasons=%v, want at least 3", fail.FailureReasons)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSlowLatencyFailsThreshold(t *testing.T) {
|
||||||
|
rec := &recorder{peak: 120, pingTimes: make(map[time.Time]struct{})}
|
||||||
|
changes := []changeResult{{
|
||||||
|
Version: 3,
|
||||||
|
Connected: 120,
|
||||||
|
Received: 120,
|
||||||
|
LatenciesMS: []float64{2500},
|
||||||
|
}}
|
||||||
|
sum := buildSummary(rec, changes, "http://example", 120, 120, time.Now(), 5*time.Minute)
|
||||||
|
if sum.Passed {
|
||||||
|
t.Fatal("expected p95 latency over 2s to fail")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestScenarioReceiptKeepsFirst(t *testing.T) {
|
||||||
|
c := &client{}
|
||||||
|
first := time.Unix(100, 0)
|
||||||
|
c.recordScenario(4, first)
|
||||||
|
c.recordScenario(4, first.Add(time.Second))
|
||||||
|
got, ok := c.receipt(4)
|
||||||
|
if !ok || !got.Equal(first) {
|
||||||
|
t.Fatalf("receipt=%v ok=%v, want %v", got, ok, first)
|
||||||
|
}
|
||||||
|
}
|
||||||
Vendored
+1
@@ -0,0 +1 @@
|
|||||||
|
{"data_version":"2026-11-07T23:51:00Z","model_version":"histgb-2026-11-14","context_version":"2026-11-14T03:00:00Z","sensors":[{"id":"hn-bahnhof-01","name":"Bahnhof","lat":49.1429,"lon":9.2076,"mean":24.2,"p95":88.2,"battery":71},{"id":"hn-deutschhof-01","name":"Deutschhof","lat":49.1397,"lon":9.2149,"mean":14.0,"p95":49.6,"battery":null},{"id":"hn-kaiserstrasse-01","name":"Kaiserstra\u00dfe","lat":49.1419,"lon":9.2096,"mean":21.1,"p95":74.1,"battery":64},{"id":"hn-marktplatz-01","name":"Marktplatz","lat":49.1426,"lon":9.212,"mean":17.9,"p95":63.8,"battery":83},{"id":"hn-neckar-01","name":"Neckarufer","lat":49.1448,"lon":9.2087,"mean":9.4,"p95":35.4,"battery":92},{"id":"hn-rathaus-01","name":"Rathaus","lat":49.1422,"lon":9.2131,"mean":15.6,"p95":55.9,"battery":58},{"id":"hn-stadtgalerie-01","name":"Stadtgalerie","lat":49.1427,"lon":9.2109,"mean":22.6,"p95":67.2,"battery":87},{"id":"hn-theater-01","name":"Theater","lat":49.1402,"lon":9.2124,"mean":11.7,"p95":41.7,"battery":49}],"series":{"hn-bahnhof-01":[{"t":"2026-10-25T00:00:00Z","v":3.6},{"t":"2026-10-25T00:15:00Z","v":4.1},{"t":"2026-10-25T00:30:00Z","v":4.4},{"t":"2026-10-25T00:45:00Z","v":4.0},{"t":"2026-10-25T01:00:00Z","v":3.7},{"t":"2026-10-25T01:15:00Z","v":3.6},{"t":"2026-10-25T01:30:00Z","v":4.3},{"t":"2026-10-25T01:45:00Z","v":3.8},{"t":"2026-10-25T02:00:00Z","v":4.5},{"t":"2026-10-25T02:15:00Z","v":3.7},{"t":"2026-10-25T02:30:00Z","v":4.4},{"t":"2026-10-25T02:45:00Z","v":4.3},{"t":"2026-10-25T03:00:00Z","v":4.2},{"t":"2026-10-25T03:15:00Z","v":4.1},{"t":"2026-10-25T03:30:00Z","v":3.9},{"t":"2026-10-25T03:45:00Z","v":3.8},{"t":"2026-10-25T04:00:00Z","v":3.8},{"t":"2026-10-25T04:15:00Z","v":3.7},{"t":"2026-10-25T04:30:00Z","v":4.4},{"t":"2026-10-25T04:45:00Z","v":4.1},{"t":"2026-10-25T05:00:00Z","v":12.8},{"t":"2026-10-25T05:15:00Z","v":12.7},{"t":"2026-10-25T05:30:00Z","v":12.9},{"t":"2026-10-25T05:45:00Z","v":11.4},{"t":"2026-10-25T06:00:00Z","v":16.3},{"t":"2026-10-25T06:15:00Z","v":17.5},{"t":"2026-10-25T06:30:00Z","v":17.3},{"t":"2026-10-25T06:45:00Z","v":17.0},{"t":"2026-10-25T07:00:00Z","v":23.5},{"t":"2026-10-25T07:15:00Z","v":23.5},{"t":"2026-10-25T07:30:00Z","v":20.4},{"t":"2026-10-25T07:45:00Z","v":19.5},{"t":"2026-10-25T08:00:00Z","v":26.7},{"t":"2026-10-25T08:15:00Z","v":25.3},{"t":"2026-10-25T08:30:00Z","v":28.2},{"t":"2026-10-25T08:45:00Z","v":23.6},{"t":"2026-10-25T09:00:00Z","v":30.1},{"t":"2026-10-25T09:15:00Z","v":31.2},{"t":"2026-10-25T09:30:00Z","v":30.4},{"t":"2026-10-25T09:45:00Z","v":33.7},{"t":"2026-10-25T10:00:00Z","v":37.2},{"t":"2026-10-25T10:15:00Z","v":32.1},{"t":"2026-10-25T10:30:00Z","v":36.9},{"t":"2026-10-25T10:45:00Z","v":30.6},{"t":"2026-10-25T11:00:00Z","v":35.9},{"t":"2026-10-25T11:15:00Z","v":31.7},{"t":"2026-10-25T11:30:00Z","v":30.7},{"t":"2026-10-25T11:45:00Z","v":35.4},{"t":"2026-10-25T12:00:00Z","v":36.0},{"t":"2026-10-25T12:15:00Z","v":31.3},{"t":"2026-10-25T12:30:00Z","v":30.0},{"t":"2026-10-25T12:45:00Z","v":32.9},{"t":"2026-10-25T13:00:00Z","v":27.5},{"t":"2026-10-25T13:15:00Z","v":26.5},{"t":"2026-10-25T13:30:00Z","v":28.1},{"t":"2026-10-25T13:45:00Z","v":29.8},{"t":"2026-10-25T14:00:00Z","v":26.4},{"t":"2026-10-25T14:15:00Z","v":26.5},{"t":"2026-10-25T14:30:00Z","v":29.2},{"t":"2026-10-25T14:45:00Z","v":29.3},{"t":"2026-10-25T15:00:00Z","v":30.5},{"t":"2026-10-25T15:15:00Z","v":30.7},{"t":"2026-10-25T15:30:00Z","v":27.4},{"t":"2026-10-25T15:45:00Z","v":25.2},{"t":"2026-10-25T16:00:00Z","v":31.8},{"t":"2026-10-25T16:15:00Z","v":28.9},{"t":"2026-10-25T16:30:00Z","v":33.3},{"t":"2026-10-25T16:45:00Z","v":32.8},{"t":"2026-10-25T17:00:00Z","v":33.7},{"t":"2026-10-25T17:15:00Z","v":31.5},{"t":"2026-10-25T17:30:00Z","v":33.8},{"t":"2026-10-25T17:45:00Z","v":31.9},{"t":"2026-10-25T18:00:00Z","v":29.9},{"t":"2026-10-25T18:15:00Z","v":32.6},{"t":"2026-10-25T18:30:00Z","v":31.8},{"t":"2026-10-25T18:45:00Z","v":32.6},{"t":"2026-10-25T19:00:00Z","v":16.7},{"t":"2026-10-25T19:15:00Z","v":15.7},{"t":"2026-10-25T19:30:00Z","v":16.2},{"t":"2026-10-25T19:45:00Z","v":15.8},{"t":"2026-10-25T20:00:00Z","v":15.8},{"t":"2026-10-25T20:15:00Z","v":14.4},{"t":"2026-10-25T20:30:00Z","v":16.5},{"t":"2026-10-25T20:45:00Z","v":15.7},{"t":"2026-10-25T21:00:00Z","v":16.7},{"t":"2026-10-25T21:15:00Z","v":13.8},{"t":"2026-10-25T21:30:00Z","v":16.7},{"t":"2026-10-25T21:45:00Z","v":14.2},{"t":"2026-10-25T22:00:00Z","v":7.2},{"t":"2026-10-25T22:15:00Z","v":6.5},{"t":"2026-10-25T22:30:00Z","v":7.1},{"t":"2026-10-25T22:45:00Z","v":5.9},{"t":"2026-10-25T23:00:00Z","v":3.6},{"t":"2026-10-25T23:15:00Z","v":4.1},{"t":"2026-10-25T23:30:00Z","v":3.9},{"t":"2026-10-25T23:45:00Z","v":3.7},{"t":"2026-10-26T00:00:00Z","v":3.4},{"t":"2026-10-26T00:15:00Z","v":3.5},{"t":"2026-10-26T00:30:00Z","v":4.0},{"t":"2026-10-26T00:45:00Z","v":3.7},{"t":"2026-10-26T01:00:00Z","v":3.9},{"t":"2026-10-26T01:15:00Z","v":3.7},{"t":"2026-10-26T01:30:00Z","v":4.0},{"t":"2026-10-26T01:45:00Z","v":3.9},{"t":"2026-10-26T02:00:00Z","v":3.5},{"t":"2026-10-26T02:15:00Z"Line truncated
|
||||||
Reference in new issue
Block a user