feat: prezero priority snapshot and internal API
ci / check (push) Successful in 30s
ci / check (pull_request) Successful in 29s

Build artifacts/prezero_priority.json from the anonymized glass fill-level export (11,701 rows, 449 containers, 1,338 location/material series). The contract tier rule yields high 88, medium 248, low 113 with 1,192 overfull readings.

Add /prezero/summary, /prezero/priority, and /prezero/containers/{id} to the internal FastAPI service, with startup validation, filtering/pagination, recent reading history, and 503 isolation when the artifact is missing. Raw CSV stays gitignored; the derived 1.1 MB artifact is the source of truth. This is a priority proof, not a route: the dataset has no coordinates.
This commit is contained in:
Noel Sabia committed 2026-11-14 18:06:02 +01:00
1 parent 507f33b7ea
commit ad63d1caf3
9 files changed
+46198 -4

No files matched your search

+60 -3
View File
@@ -57,6 +57,7 @@ To rebuild the artifacts from raw data:
make data # needs the raw CSV; first run also fetches Open-Meteo weather
make train # writes artifacts/model.joblib, metrics.json, forecasts.json
make small # prints the busy-hour classifier metrics + confusion matrix
make prezero # rebuilds artifacts/prezero_priority.json from the fill-level CSV
make figures # regenerates docs/figures/*.svg from metrics.json
make test
```
@@ -67,6 +68,11 @@ The raw challenge CSV is **never committed**. The default path is
git-ignored. The committed `artifacts/` (JSON + joblib + compact weather frames)
let the API and Docker image run without the raw file or network.
The PreZero fill-level export is also git-ignored. For `make prezero`, put
`glass_fill_levels_anonymized_f.csv` at `data/raw/` or point
`PULSECAST_PREZERO_CSV` at it. Only the derived
`artifacts/prezero_priority.json` is committed.
## Artifacts
| File | Purpose |
@@ -78,6 +84,7 @@ let the API and Docker image run without the raw file or network.
| `artifacts/small_metrics.json` | Busy classifier: metrics, confusion matrix, per-sensor, thresholds |
| `artifacts/context.json` | Current + 7-day weather, BW holidays |
| `artifacts/data_audit.json` | Row/device checks for the raw CSV |
| `artifacts/prezero_priority.json` | 449-container priority snapshot + last 10 readings each |
| `artifacts/model.joblib` | Baked HistGB forecast bundle |
| `artifacts/small_model.joblib` | Baked busy classifier |
| `artifacts/weather_*.parquet` | Baked weather frames for retraining/inference |
@@ -93,6 +100,9 @@ exposes it publicly.
| `GET` | `/readyz` | `{"ok":true,"model":true}`, 503 if no model |
| `GET` | `/meta` | Versions, coverage, metrics, features, holiday effect |
| `GET` | `/bootstrap` | Snapshot for Go: sensors, hourly series, context, meta |
| `GET` | `/prezero/summary` | Priority counts and material totals |
| `GET` | `/prezero/priority` | Ranked containers; `priority`, `limit`, `offset` |
| `GET` | `/prezero/containers/{id}` | One container + its 10 newest readings |
| `POST` | `/predict` | `{sensor,horizon,scenario,weather?,is_holiday?}` → `points` |
| `POST` | `/simulate` | Scenario forecast for all or selected sensors |
| `POST` | `/train` | Control token; multipart CSV or JSON path; 202 job |
@@ -125,6 +135,47 @@ and persists to `PULSECAST_MODELS` (default `/tmp/pulsecast/models`). A restart
falls back to the baked model. `model=small` is the fast demo; `model=forecast`
retrains the global HistGB (slower, still background).
## PreZero container priority
`artifacts/prezero_priority.json` ranks the 449 anonymous container locations in
the 11,701-row glass fill-level export (2025-10-17 → 2026-09-30, 1,338
location/material series). The rule uses only reported states:
- **high:** `overfull_count >= 5` or (`overfull_count >= 2` and
`overfull_rate >= 0.25`)
- **medium:** `overfull_count >= 1`, not high
- **low:** `overfull_count == 0`
The generated snapshot has **high 88, medium 248, low 113**, with 1,192 overfull
readings. Sorted by tier, overfull count/rate, staleness (`reference_date` is
the dataset max date, 2026-09-30, not today), then location id. The artifact
also holds each container's 10 newest readings for the history route.
**The dataset has no coordinates.** This is a service-priority proof, not a
route; there is no capacity or depot data, and `overfull` is a reported state,
not a measured volume. The committed artifact is the API's source of truth. If
it is missing, only `/prezero/*` returns 503; the forecast/training routes keep
working.
Rebuild from the raw, git-ignored export:
```bash
cp glass_fill_levels_anonymized_f.csv data/raw/ # or set PULSECAST_PREZERO_CSV
make prezero
# equivalently:
uv run python scripts/build_prezero.py \
--csv data/raw/glass_fill_levels_anonymized_f.csv \
--out artifacts/prezero_priority.json
```
```bash
curl -s localhost:8000/prezero/summary
curl -s 'localhost:8000/prezero/priority?priority=high&limit=3'
curl -s localhost:8000/prezero/containers/<location_id>
```
Detail, counts, and limitations: [`docs/prezero.md`](docs/prezero.md).
## End-to-end check
`scripts/e2e_train.py` starts a real uvicorn process on port 18000 with a
@@ -145,6 +196,7 @@ The unit and contract tests are `make test`; `make check` is the CI gate.
| `docs/metrics.md` | Paper tables/figures: protocol, headline, ablation, per sensor, uncertainty, small model, claim limits |
| `docs/small-model.md` | Busy-hour target, thresholds, split, metrics, confusion matrix, per sensor, limits |
| `docs/train-e2e.md` | Exact live training-job evidence (small + forecast, hot-swap, errors) |
| `docs/prezero.md` | Container-priority rule, counts, artifact/API shape, no-coordinates limits |
| `docs/figures/*.svg` | Ablation MAE, per-sensor MAE, small-model confusion matrix (generated) |
| `model_card.md` | Model card: features, split, baseline, ablation, leakage, limitations |
@@ -167,6 +219,8 @@ artifacts baked in, writable paths only under `/tmp`.
| `PULSECAST_ARTIFACTS` | `./artifacts` | Baked JSON/model output |
| `PULSECAST_CACHE` | `./data/cache` | Parquet/weather cache |
| `PULSECAST_WEATHER` | `PULSECAST_ARTIFACTS` | Baked weather frame location |
| `PULSECAST_PREZERO_ARTIFACT` | `./artifacts/prezero_priority.json` | PreZero priority snapshot |
| `PULSECAST_PREZERO_CSV` | `./data/raw/glass_fill_levels_anonymized_f.csv` | Raw fill-level export for `make prezero` |
| `PULSECAST_MODELS` | `/tmp/pulsecast/models` | Retrained model persistence |
| `PULSECAST_UPLOADS` | `/tmp/pulsecast/uploads` | Multipart upload landing |
| `CONTROL_TOKEN` | unset | Required by `/train`; service returns 503 if unset |
@@ -184,10 +238,11 @@ src/pulsecast/
train.py `make train`, ablation, artifacts
small.py busy classifier + flexible CSV parser
retrain.py runtime forecast retraining from upload
prezero.py load/validate/filter/page/lookup the priority snapshot
server.py FastAPI app and training job manager
scripts/ train_small.py, plot_metrics.py, e2e_train.py
docs/ metrics.md, small-model.md, train-e2e.md, generated figures
tests/ schema, bootstrap, health, prediction, small-training job
scripts/ train_small.py, plot_metrics.py, build_prezero.py, e2e_train.py
docs/ metrics.md, small-model.md, train-e2e.md, prezero.md, figures
tests/ schema, bootstrap, health, prediction, prezero, small-training job
model_card.md features, split, baseline, ablation, limits, leakage
```
@@ -200,3 +255,5 @@ model_card.md features, split, baseline, ablation, limits, leakage
- Holidays are context and a small feature, not a large accuracy claim.
- No POS, sales, queue, or labor-cost data; staffing is decision support with
explicit assumptions.
- PreZero has no coordinates, capacity, or depot data: it ranks collection
priority, it does not produce or optimize routes.