Compare commits
3
Commits
feat/ci-docs
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
603b1c8cee | ||
|
|
ad63d1caf3 | ||
|
|
507f33b7ea |
No files matched your search
@@ -1,12 +1,13 @@
|
||||
UV ?= uv
|
||||
|
||||
.PHONY: help data train small figures test lint check serve docker clean
|
||||
.PHONY: help data train small prezero figures test lint check serve docker clean
|
||||
|
||||
help:
|
||||
@printf '%s\n' \
|
||||
'make data parse/clean the raw CSV, join weather + holidays, export JSON' \
|
||||
'make train fit the 28-day holdout model, export metrics/forecasts/model' \
|
||||
'make small train + print the busy-hour classifier metrics from the raw CSV' \
|
||||
'make prezero build artifacts/prezero_priority.json from the fill-level CSV' \
|
||||
'make figures regenerate docs/figures/*.svg from artifacts/metrics.json' \
|
||||
'make test run pytest' \
|
||||
'make lint run ruff static checks' \
|
||||
@@ -24,6 +25,9 @@ train:
|
||||
small:
|
||||
$(UV) run python scripts/train_small.py
|
||||
|
||||
prezero:
|
||||
$(UV) run python scripts/build_prezero.py
|
||||
|
||||
figures:
|
||||
$(UV) run python scripts/plot_metrics.py
|
||||
|
||||
|
||||
@@ -57,6 +57,7 @@ To rebuild the artifacts from raw data:
|
||||
make data # needs the raw CSV; first run also fetches Open-Meteo weather
|
||||
make train # writes artifacts/model.joblib, metrics.json, forecasts.json
|
||||
make small # prints the busy-hour classifier metrics + confusion matrix
|
||||
make prezero # rebuilds artifacts/prezero_priority.json from the fill-level CSV
|
||||
make figures # regenerates docs/figures/*.svg from metrics.json
|
||||
make test
|
||||
```
|
||||
@@ -67,6 +68,11 @@ The raw challenge CSV is **never committed**. The default path is
|
||||
git-ignored. The committed `artifacts/` (JSON + joblib + compact weather frames)
|
||||
let the API and Docker image run without the raw file or network.
|
||||
|
||||
The PreZero fill-level export is also git-ignored. For `make prezero`, put
|
||||
`glass_fill_levels_anonymized_f.csv` at `data/raw/` or point
|
||||
`PULSECAST_PREZERO_CSV` at it. Only the derived
|
||||
`artifacts/prezero_priority.json` is committed.
|
||||
|
||||
## Artifacts
|
||||
|
||||
| File | Purpose |
|
||||
@@ -78,6 +84,7 @@ let the API and Docker image run without the raw file or network.
|
||||
| `artifacts/small_metrics.json` | Busy classifier: metrics, confusion matrix, per-sensor, thresholds |
|
||||
| `artifacts/context.json` | Current + 7-day weather, BW holidays |
|
||||
| `artifacts/data_audit.json` | Row/device checks for the raw CSV |
|
||||
| `artifacts/prezero_priority.json` | 449-container priority snapshot + last 10 readings each |
|
||||
| `artifacts/model.joblib` | Baked HistGB forecast bundle |
|
||||
| `artifacts/small_model.joblib` | Baked busy classifier |
|
||||
| `artifacts/weather_*.parquet` | Baked weather frames for retraining/inference |
|
||||
@@ -93,6 +100,9 @@ exposes it publicly.
|
||||
| `GET` | `/readyz` | `{"ok":true,"model":true}`, 503 if no model |
|
||||
| `GET` | `/meta` | Versions, coverage, metrics, features, holiday effect |
|
||||
| `GET` | `/bootstrap` | Snapshot for Go: sensors, hourly series, context, meta |
|
||||
| `GET` | `/prezero/summary` | Priority counts and material totals |
|
||||
| `GET` | `/prezero/priority` | Ranked containers; `priority`, `limit`, `offset` |
|
||||
| `GET` | `/prezero/containers/{id}` | One container + its 10 newest readings |
|
||||
| `POST` | `/predict` | `{sensor,horizon,scenario,weather?,is_holiday?}` → `points` |
|
||||
| `POST` | `/simulate` | Scenario forecast for all or selected sensors |
|
||||
| `POST` | `/train` | Control token; multipart CSV or JSON path; 202 job |
|
||||
@@ -125,6 +135,47 @@ and persists to `PULSECAST_MODELS` (default `/tmp/pulsecast/models`). A restart
|
||||
falls back to the baked model. `model=small` is the fast demo; `model=forecast`
|
||||
retrains the global HistGB (slower, still background).
|
||||
|
||||
## PreZero container priority
|
||||
|
||||
`artifacts/prezero_priority.json` ranks the 449 anonymous container locations in
|
||||
the 11,701-row glass fill-level export (2025-10-17 → 2026-09-30, 1,338
|
||||
location/material series). The rule uses only reported states:
|
||||
|
||||
- **high:** `overfull_count >= 5` or (`overfull_count >= 2` and
|
||||
`overfull_rate >= 0.25`)
|
||||
- **medium:** `overfull_count >= 1`, not high
|
||||
- **low:** `overfull_count == 0`
|
||||
|
||||
The generated snapshot has **high 88, medium 248, low 113**, with 1,192 overfull
|
||||
readings. Sorted by tier, overfull count/rate, staleness (`reference_date` is
|
||||
the dataset max date, 2026-09-30, not today), then location id. The artifact
|
||||
also holds each container's 10 newest readings for the history route.
|
||||
|
||||
**The dataset has no coordinates.** This is a service-priority proof, not a
|
||||
route; there is no capacity or depot data, and `overfull` is a reported state,
|
||||
not a measured volume. The committed artifact is the API's source of truth. If
|
||||
it is missing, only `/prezero/*` returns 503; the forecast/training routes keep
|
||||
working.
|
||||
|
||||
Rebuild from the raw, git-ignored export:
|
||||
|
||||
```bash
|
||||
cp glass_fill_levels_anonymized_f.csv data/raw/ # or set PULSECAST_PREZERO_CSV
|
||||
make prezero
|
||||
# equivalently:
|
||||
uv run python scripts/build_prezero.py \
|
||||
--csv data/raw/glass_fill_levels_anonymized_f.csv \
|
||||
--out artifacts/prezero_priority.json
|
||||
```
|
||||
|
||||
```bash
|
||||
curl -s localhost:8000/prezero/summary
|
||||
curl -s 'localhost:8000/prezero/priority?priority=high&limit=3'
|
||||
curl -s localhost:8000/prezero/containers/<location_id>
|
||||
```
|
||||
|
||||
Detail, counts, and limitations: [`docs/prezero.md`](docs/prezero.md).
|
||||
|
||||
## End-to-end check
|
||||
|
||||
`scripts/e2e_train.py` starts a real uvicorn process on port 18000 with a
|
||||
@@ -145,6 +196,7 @@ The unit and contract tests are `make test`; `make check` is the CI gate.
|
||||
| `docs/metrics.md` | Paper tables/figures: protocol, headline, ablation, per sensor, uncertainty, small model, claim limits |
|
||||
| `docs/small-model.md` | Busy-hour target, thresholds, split, metrics, confusion matrix, per sensor, limits |
|
||||
| `docs/train-e2e.md` | Exact live training-job evidence (small + forecast, hot-swap, errors) |
|
||||
| `docs/prezero.md` | Container-priority rule, counts, artifact/API shape, no-coordinates limits |
|
||||
| `docs/figures/*.svg` | Ablation MAE, per-sensor MAE, small-model confusion matrix (generated) |
|
||||
| `model_card.md` | Model card: features, split, baseline, ablation, leakage, limitations |
|
||||
|
||||
@@ -167,6 +219,8 @@ artifacts baked in, writable paths only under `/tmp`.
|
||||
| `PULSECAST_ARTIFACTS` | `./artifacts` | Baked JSON/model output |
|
||||
| `PULSECAST_CACHE` | `./data/cache` | Parquet/weather cache |
|
||||
| `PULSECAST_WEATHER` | `PULSECAST_ARTIFACTS` | Baked weather frame location |
|
||||
| `PULSECAST_PREZERO_ARTIFACT` | `./artifacts/prezero_priority.json` | PreZero priority snapshot |
|
||||
| `PULSECAST_PREZERO_CSV` | `./data/raw/glass_fill_levels_anonymized_f.csv` | Raw fill-level export for `make prezero` |
|
||||
| `PULSECAST_MODELS` | `/tmp/pulsecast/models` | Retrained model persistence |
|
||||
| `PULSECAST_UPLOADS` | `/tmp/pulsecast/uploads` | Multipart upload landing |
|
||||
| `CONTROL_TOKEN` | unset | Required by `/train`; service returns 503 if unset |
|
||||
@@ -184,10 +238,11 @@ src/pulsecast/
|
||||
train.py `make train`, ablation, artifacts
|
||||
small.py busy classifier + flexible CSV parser
|
||||
retrain.py runtime forecast retraining from upload
|
||||
prezero.py load/validate/filter/page/lookup the priority snapshot
|
||||
server.py FastAPI app and training job manager
|
||||
scripts/ train_small.py, plot_metrics.py, e2e_train.py
|
||||
docs/ metrics.md, small-model.md, train-e2e.md, generated figures
|
||||
tests/ schema, bootstrap, health, prediction, small-training job
|
||||
scripts/ train_small.py, plot_metrics.py, build_prezero.py, e2e_train.py
|
||||
docs/ metrics.md, small-model.md, train-e2e.md, prezero.md, figures
|
||||
tests/ schema, bootstrap, health, prediction, prezero, small-training job
|
||||
model_card.md features, split, baseline, ablation, limits, leakage
|
||||
```
|
||||
|
||||
@@ -200,3 +255,5 @@ model_card.md features, split, baseline, ablation, limits, leakage
|
||||
- Holidays are context and a small feature, not a large accuracy claim.
|
||||
- No POS, sales, queue, or labor-cost data; staffing is decision support with
|
||||
explicit assumptions.
|
||||
- PreZero has no coordinates, capacity, or depot data: it ranks collection
|
||||
priority, it does not produce or optimize routes.
|
||||
File diff suppressed because it is too large.
Load diff
+179
@@ -0,0 +1,179 @@
|
||||
# PreZero container priority
|
||||
|
||||
A transparent collection-priority snapshot built from the anonymized glass
|
||||
fill-level export. The committed artifact is
|
||||
`artifacts/prezero_priority.json`; the raw CSV is **not** committed.
|
||||
|
||||
**This is a priority proof, not a route.** The dataset has no coordinates, no
|
||||
container capacity, and no depot data. `overfull` is a reported state, not a
|
||||
measured volume.
|
||||
|
||||
## Provenance
|
||||
|
||||
| Field | Value |
|
||||
|---|---:|
|
||||
| Input | `glass_fill_levels_anonymized_f.csv` (UTF-8 with BOM) |
|
||||
| Rows | 11,701 |
|
||||
| Distinct `Location_id` | 449 |
|
||||
| `Location_id` + `material_type` series | 1,338 |
|
||||
| Date range | 2025-10-17 → 2026-09-30 |
|
||||
| `overfull` readings | 1,192 |
|
||||
| Material readings | white 4,432 · green 4,169 · brown 3,100 |
|
||||
| Material overfull | white 731 · green 357 · brown 104 |
|
||||
|
||||
The export is read with `utf-8-sig`; header names and cells are stripped. Every
|
||||
row must have `Location_id`, `material_type`, a `YYYY-MM-DD` date, and a
|
||||
`fill_level_perc` of either `overfull` or an integer 0–100.
|
||||
|
||||
The raw file stays git-ignored under `data/raw/`. Put it there or set
|
||||
`PULSECAST_PREZERO_CSV`; the generated artifact is committed so the API and
|
||||
Docker image do not need the CSV.
|
||||
|
||||
## Rule
|
||||
|
||||
Each `Location_id` is one container location. For all rows at that location:
|
||||
|
||||
- `readings`, `overfull_count`, `overfull_rate = overfull_count / readings`
|
||||
- `numeric_count` = rows with a numeric 0–100 fill (excludes `overfull`)
|
||||
- `p95_fill` = 95th percentile of numeric readings, `null` if none
|
||||
- `max_fill`, `mean_fill` = numeric readings only
|
||||
- `last_fill`, `last_is_overfull`, `last_date` = most recent row
|
||||
- `days_since_last` = `reference_date - last_date`; reference is the dataset
|
||||
max date (2026-09-30), not today
|
||||
- `trend` = least-squares slope over the last up to 3 numeric readings
|
||||
(percentage points per reading); `null` with fewer than 2
|
||||
- `priority_score = round(overfull_count + 2 * overfull_rate, 4)` — a display
|
||||
and sorting convenience, not the tier
|
||||
- `materials` = per material: readings, overfull count, last fill/state/date
|
||||
|
||||
Tier:
|
||||
|
||||
- **high:** `overfull_count >= 5` OR (`overfull_count >= 2` AND
|
||||
`overfull_rate >= 0.25`)
|
||||
- **medium:** `overfull_count >= 1` and not high
|
||||
- **low:** `overfull_count == 0`
|
||||
|
||||
Result on the attached CSV: **high 88, medium 248, low 113**.
|
||||
|
||||
Order: tier (`high` first), then overfull count desc, overfull rate desc,
|
||||
`days_since_last` asc, `location_id` asc.
|
||||
|
||||
## Artifact shape
|
||||
|
||||
```json
|
||||
{
|
||||
"meta": {
|
||||
"source": "glass_fill_levels_anonymized_f.csv",
|
||||
"generated_at": "2026-11-14T17:00:00Z",
|
||||
"rows": 11701,
|
||||
"containers": 449,
|
||||
"material_series": 1338,
|
||||
"date_from": "2025-10-17",
|
||||
"date_to": "2026-09-30",
|
||||
"reference_date": "2026-09-30",
|
||||
"overfull_readings": 1192,
|
||||
"priority_rule": "high: ...; medium: ...; low: 0",
|
||||
"limitations": ["No coordinates ...", "..."]
|
||||
},
|
||||
"summary": {
|
||||
"priority_counts": {"high": 88, "medium": 248, "low": 113},
|
||||
"by_material": {
|
||||
"white": {"readings": 4432, "overfull": 731},
|
||||
"green": {"readings": 4169, "overfull": 357},
|
||||
"brown": {"readings": 3100, "overfull": 104}
|
||||
}
|
||||
},
|
||||
"containers": [
|
||||
{
|
||||
"location_id": "...",
|
||||
"priority": "high",
|
||||
"priority_score": 21.525,
|
||||
"readings": 80,
|
||||
"overfull_count": 21,
|
||||
"overfull_rate": 0.2625,
|
||||
"numeric_count": 59,
|
||||
"p95_fill": 100.0,
|
||||
"max_fill": 100,
|
||||
"mean_fill": 64.24,
|
||||
"last_fill": null,
|
||||
"last_is_overfull": true,
|
||||
"last_date": "2026-09-29",
|
||||
"days_since_last": 1,
|
||||
"trend": 30.0,
|
||||
"materials": {"white": {"readings": 29, "...": "..."}},
|
||||
"recent_readings": [
|
||||
{"date": "2026-09-29", "material_type": "white",
|
||||
"fill_level_perc": "overfull", "is_overfull": true}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
The artifact holds at most 10 recent readings per container, not the full
|
||||
11,701-row table. `last_fill` is `null` when the newest reading is `overfull`.
|
||||
`p95_fill`, `max_fill`, `mean_fill`, and `trend` are `null` when the location
|
||||
has no numeric readings or too few for a slope.
|
||||
|
||||
## Generation
|
||||
|
||||
```bash
|
||||
# Raw CSV is git-ignored; copy it or set PULSECAST_PREZERO_CSV.
|
||||
cp glass_fill_levels_anonymized_f.csv data/raw/
|
||||
make prezero
|
||||
|
||||
# Equivalent explicit command:
|
||||
uv run python scripts/build_prezero.py \
|
||||
--csv data/raw/glass_fill_levels_anonymized_f.csv \
|
||||
--out artifacts/prezero_priority.json
|
||||
```
|
||||
|
||||
The script prints rows, containers, material series, overfull readings, date
|
||||
range, and the three tier counts. It fails loudly on missing columns, blank
|
||||
cells, bad dates, or invalid fill values instead of guessing.
|
||||
|
||||
## Internal API
|
||||
|
||||
The ML service loads and validates the artifact at startup. If it is missing or
|
||||
malformed, only these routes return 503; `/healthz`, `/readyz`, `/bootstrap`,
|
||||
`/predict`, `/simulate`, and `/meta` keep working.
|
||||
|
||||
| Method | Path | Behavior |
|
||||
|---|---|---|
|
||||
| GET | `/prezero/summary` | `meta` + `summary` |
|
||||
| GET | `/prezero/priority` | Ranked containers; `priority` (`high\|medium\|low`), `limit` 1–500 (default 50), `offset` >= 0 |
|
||||
| GET | `/prezero/containers/{location_id}` | One container + 10 newest readings; unknown → 404 |
|
||||
|
||||
Invalid `priority`, `limit`, or `offset` → 400 in the existing error envelope.
|
||||
The priority list omits `recent_readings`; the container route returns it as
|
||||
`readings`. `/prezero/priority` reports `total` before pagination and `count`
|
||||
after it.
|
||||
|
||||
```bash
|
||||
curl -s localhost:8000/prezero/summary
|
||||
curl -s 'localhost:8000/prezero/priority?priority=high&limit=3'
|
||||
curl -s localhost:8000/prezero/containers/<known-id>
|
||||
```
|
||||
|
||||
## Verification
|
||||
|
||||
```bash
|
||||
uv run pytest -q tests/test_prezero.py
|
||||
make check
|
||||
```
|
||||
|
||||
The tests assert the dataset counts above (11,701 rows, 449 containers, 1,338
|
||||
series, 1,192 overfull, 88/248/113 tiers), filtering/pagination, known and
|
||||
unknown containers, invalid queries, artifact schema and tier consistency, and
|
||||
the missing-artifact 503 path.
|
||||
|
||||
## Limits
|
||||
|
||||
- No coordinates: cannot compute distance, travel time, or a route.
|
||||
- No capacity, depot, vehicle, or shift data.
|
||||
- `overfull` is a reported state; fill percentages are coarse, user-reported
|
||||
values (0/10/.../100), not continuous sensor volumes.
|
||||
- The reference date is the dataset max date, so staleness is relative to the
|
||||
snapshot, not to when the API is called.
|
||||
- Counts and rates describe this export only; they are not a forecast of future
|
||||
fill levels.
|
||||
@@ -0,0 +1,329 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Build ``artifacts/prezero_priority.json`` from the raw fill-level export.
|
||||
|
||||
The raw CSV is challenge data and stays git-ignored. The derived artifact is
|
||||
committed and is the source of truth for the internal ``/prezero`` routes.
|
||||
|
||||
Priority rule (contract, transparent, no invented labels):
|
||||
|
||||
high: overfull_count >= 5 or (overfull_count >= 2 and overfull_rate >= 0.25)
|
||||
medium: overfull_count >= 1 and not high
|
||||
low: overfull_count == 0
|
||||
|
||||
There are no coordinates in the dataset: this is a service-priority proof, not
|
||||
a route. ``priority_score`` is a sorting/display convenience; the tier is the
|
||||
contract.
|
||||
|
||||
Examples:
|
||||
uv run python scripts/build_prezero.py
|
||||
uv run python scripts/build_prezero.py --csv data/raw/glass_fill_levels_anonymized_f.csv
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import csv
|
||||
import json
|
||||
import sys
|
||||
from collections import Counter, defaultdict
|
||||
from datetime import UTC, date, datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import numpy as np
|
||||
|
||||
from pulsecast import config
|
||||
from pulsecast.prezero import container_sort_key, priority_tier
|
||||
|
||||
REQUIRED_COLUMNS = ("Location_id", "material_type", "date", "fill_level_perc")
|
||||
MATERIAL_ORDER = ("white", "green", "brown")
|
||||
PRIORITY_RULE = (
|
||||
"high: overfull_count >= 5 or (overfull_count >= 2 and overfull_rate >= 0.25); "
|
||||
"medium: overfull_count >= 1 and not high; low: 0"
|
||||
)
|
||||
LIMITATIONS = [
|
||||
"No coordinates in the dataset; this is a priority proof, not a route.",
|
||||
"No container capacity or depot data.",
|
||||
"Overfull is a reported state, not a measured volume.",
|
||||
]
|
||||
|
||||
|
||||
class BuildError(ValueError):
|
||||
"""The input CSV does not satisfy the PreZero contract."""
|
||||
|
||||
|
||||
def _normalized_columns(fieldnames: list[str] | None, path: Path) -> dict[str, str]:
|
||||
"""Map BOM/whitespace-normalized names back to the CSV's raw names."""
|
||||
if not fieldnames:
|
||||
raise BuildError(f"{path}: missing header row")
|
||||
mapping: dict[str, str] = {}
|
||||
for raw in fieldnames:
|
||||
name = str(raw).strip().lstrip("\ufeff")
|
||||
if not name:
|
||||
raise BuildError(f"{path}: blank column name")
|
||||
if name in mapping:
|
||||
raise BuildError(f"{path}: duplicate column after trimming: {name}")
|
||||
mapping[name] = raw
|
||||
missing = [name for name in REQUIRED_COLUMNS if name not in mapping]
|
||||
if missing:
|
||||
raise BuildError(
|
||||
f"{path}: missing required columns: {', '.join(missing)}; "
|
||||
f"found: {', '.join(mapping)}"
|
||||
)
|
||||
return mapping
|
||||
|
||||
|
||||
def _read_rows(path: Path) -> list[dict[str, Any]]:
|
||||
"""Read the export as strings, normalize headers/cells, validate every row."""
|
||||
try:
|
||||
handle = path.open("r", encoding="utf-8-sig", newline="")
|
||||
except FileNotFoundError as exc:
|
||||
raise BuildError(
|
||||
f"CSV not found: {path}. Copy the raw export there or pass --csv."
|
||||
) from exc
|
||||
except OSError as exc:
|
||||
raise BuildError(f"cannot read {path}: {exc}") from exc
|
||||
|
||||
rows: list[dict[str, Any]] = []
|
||||
with handle:
|
||||
reader = csv.DictReader(handle)
|
||||
columns = _normalized_columns(reader.fieldnames, path)
|
||||
for line_number, raw_row in enumerate(reader, start=2):
|
||||
row = {
|
||||
name: str(raw_row.get(source) or "").strip()
|
||||
for name, source in columns.items()
|
||||
}
|
||||
blank = [name for name in REQUIRED_COLUMNS if not row[name]]
|
||||
if blank:
|
||||
raise BuildError(f"{path}:{line_number}: blank values in {', '.join(blank)}")
|
||||
|
||||
try:
|
||||
reading_date = date.fromisoformat(row["date"])
|
||||
except ValueError as exc:
|
||||
raise BuildError(
|
||||
f"{path}:{line_number}: invalid date {row['date']!r}"
|
||||
) from exc
|
||||
|
||||
fill_text = row["fill_level_perc"]
|
||||
if fill_text == "overfull":
|
||||
fill: int | None = None
|
||||
is_overfull = True
|
||||
else:
|
||||
try:
|
||||
fill = int(fill_text)
|
||||
except ValueError as exc:
|
||||
raise BuildError(
|
||||
f"{path}:{line_number}: fill_level_perc must be an integer "
|
||||
f"0-100 or 'overfull', got {fill_text!r}"
|
||||
) from exc
|
||||
if not 0 <= fill <= 100:
|
||||
raise BuildError(
|
||||
f"{path}:{line_number}: fill_level_perc out of range: {fill}"
|
||||
)
|
||||
is_overfull = False
|
||||
|
||||
rows.append(
|
||||
{
|
||||
"index": len(rows),
|
||||
"location_id": row["Location_id"],
|
||||
"material_type": row["material_type"],
|
||||
"date": row["date"],
|
||||
"date_value": reading_date,
|
||||
"fill_level_perc": fill_text,
|
||||
"fill": fill,
|
||||
"is_overfull": is_overfull,
|
||||
}
|
||||
)
|
||||
if not rows:
|
||||
raise BuildError(f"{path}: no data rows")
|
||||
return rows
|
||||
|
||||
|
||||
def _ordered(rows: list[dict[str, Any]]) -> list[dict[str, Any]]:
|
||||
"""Chronological; input order breaks ties on the same date."""
|
||||
return sorted(rows, key=lambda row: (row["date_value"], row["index"]))
|
||||
|
||||
|
||||
def _trend(numeric: list[int]) -> float | None:
|
||||
"""Least-squares slope over the last up to 3 numeric readings (points/read)."""
|
||||
if len(numeric) < 2:
|
||||
return None
|
||||
tail = numeric[-3:]
|
||||
x_mean = (len(tail) - 1) / 2
|
||||
y_mean = sum(tail) / len(tail)
|
||||
numerator = sum((x - x_mean) * (y - y_mean) for x, y in enumerate(tail))
|
||||
denominator = sum((x - x_mean) ** 2 for x in range(len(tail)))
|
||||
slope = numerator / denominator
|
||||
return round(slope, 2) if slope else 0.0
|
||||
|
||||
|
||||
def _material_names(materials: set[str]) -> list[str]:
|
||||
known = [name for name in MATERIAL_ORDER if name in materials]
|
||||
return known + sorted(materials - set(known))
|
||||
|
||||
|
||||
def _material_summary(rows: list[dict[str, Any]]) -> dict[str, dict[str, Any]]:
|
||||
ordered = _ordered(rows)
|
||||
last = ordered[-1]
|
||||
return {
|
||||
"readings": len(ordered),
|
||||
"overfull_count": sum(row["is_overfull"] for row in ordered),
|
||||
"last_fill": last["fill"],
|
||||
"last_is_overfull": last["is_overfull"],
|
||||
"last_date": last["date"],
|
||||
}
|
||||
|
||||
|
||||
def _container(rows: list[dict[str, Any]], reference: date) -> dict[str, Any]:
|
||||
ordered = _ordered(rows)
|
||||
readings = len(ordered)
|
||||
overfull_count = sum(row["is_overfull"] for row in ordered)
|
||||
overfull_rate = overfull_count / readings
|
||||
numeric = [row["fill"] for row in ordered if row["fill"] is not None]
|
||||
last = ordered[-1]
|
||||
|
||||
by_material: dict[str, list[dict[str, Any]]] = defaultdict(list)
|
||||
for row in ordered:
|
||||
by_material[row["material_type"]].append(row)
|
||||
materials = {
|
||||
name: _material_summary(by_material[name])
|
||||
for name in _material_names(set(by_material))
|
||||
}
|
||||
|
||||
p95_fill = round(float(np.percentile(numeric, 95)), 2) if numeric else None
|
||||
mean_fill = round(sum(numeric) / len(numeric), 2) if numeric else None
|
||||
recent = [
|
||||
{
|
||||
"date": row["date"],
|
||||
"material_type": row["material_type"],
|
||||
"fill_level_perc": row["fill_level_perc"],
|
||||
"is_overfull": row["is_overfull"],
|
||||
}
|
||||
for row in reversed(ordered[-10:])
|
||||
]
|
||||
|
||||
return {
|
||||
"location_id": ordered[0]["location_id"],
|
||||
"priority": priority_tier(overfull_count, overfull_rate),
|
||||
"priority_score": round(overfull_count + 2 * overfull_rate, 4),
|
||||
"readings": readings,
|
||||
"overfull_count": overfull_count,
|
||||
"overfull_rate": overfull_rate,
|
||||
"numeric_count": len(numeric),
|
||||
"p95_fill": p95_fill,
|
||||
"max_fill": max(numeric) if numeric else None,
|
||||
"mean_fill": mean_fill,
|
||||
"last_fill": last["fill"],
|
||||
"last_is_overfull": last["is_overfull"],
|
||||
"last_date": last["date"],
|
||||
"days_since_last": (reference - last["date_value"]).days,
|
||||
"trend": _trend(numeric),
|
||||
"materials": materials,
|
||||
"recent_readings": recent,
|
||||
}
|
||||
|
||||
|
||||
def build(csv_path: Path) -> dict[str, Any]:
|
||||
rows = _read_rows(csv_path)
|
||||
reference = max(row["date_value"] for row in rows)
|
||||
date_from = min(row["date_value"] for row in rows)
|
||||
|
||||
grouped: dict[str, list[dict[str, Any]]] = defaultdict(list)
|
||||
for row in rows:
|
||||
grouped[row["location_id"]].append(row)
|
||||
containers = [
|
||||
_container(grouped[location_id], reference)
|
||||
for location_id in sorted(grouped)
|
||||
]
|
||||
containers.sort(key=container_sort_key)
|
||||
|
||||
material_counts = Counter(row["material_type"] for row in rows)
|
||||
material_overfull = Counter(
|
||||
row["material_type"] for row in rows if row["is_overfull"]
|
||||
)
|
||||
priority_counts = Counter(container["priority"] for container in containers)
|
||||
by_material = {
|
||||
name: {
|
||||
"readings": material_counts[name],
|
||||
"overfull": material_overfull[name],
|
||||
}
|
||||
for name in _material_names(set(material_counts))
|
||||
}
|
||||
|
||||
meta = {
|
||||
"source": csv_path.name,
|
||||
"generated_at": datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%SZ"),
|
||||
"rows": len(rows),
|
||||
"containers": len(containers),
|
||||
"material_series": len(
|
||||
{(row["location_id"], row["material_type"]) for row in rows}
|
||||
),
|
||||
"date_from": date_from.isoformat(),
|
||||
"date_to": reference.isoformat(),
|
||||
"reference_date": reference.isoformat(),
|
||||
"overfull_readings": sum(row["is_overfull"] for row in rows),
|
||||
"priority_rule": PRIORITY_RULE,
|
||||
"limitations": list(LIMITATIONS),
|
||||
}
|
||||
summary = {
|
||||
"priority_counts": {
|
||||
"high": priority_counts["high"],
|
||||
"medium": priority_counts["medium"],
|
||||
"low": priority_counts["low"],
|
||||
},
|
||||
"by_material": by_material,
|
||||
}
|
||||
return {"meta": meta, "summary": summary, "containers": containers}
|
||||
|
||||
|
||||
def _parse_args(argv: list[str] | None = None) -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(
|
||||
description=__doc__,
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
)
|
||||
parser.add_argument(
|
||||
"--csv",
|
||||
type=Path,
|
||||
default=config.prezero_csv(),
|
||||
help="raw fill-level CSV (git-ignored; default: %(default)s)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--out",
|
||||
type=Path,
|
||||
default=config.prezero_artifact(),
|
||||
help="artifact destination (default: %(default)s)",
|
||||
)
|
||||
return parser.parse_args(argv)
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
args = _parse_args(argv)
|
||||
try:
|
||||
payload = build(args.csv)
|
||||
except BuildError as exc:
|
||||
print(f"build_prezero: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
args.out.parent.mkdir(parents=True, exist_ok=True)
|
||||
args.out.write_text(
|
||||
json.dumps(payload, indent=2, ensure_ascii=False) + "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
meta = payload["meta"]
|
||||
counts = payload["summary"]["priority_counts"]
|
||||
print(f"wrote {args.out} ({args.out.stat().st_size:,} bytes)")
|
||||
print(
|
||||
f"rows={meta['rows']} containers={meta['containers']} "
|
||||
f"material_series={meta['material_series']} "
|
||||
f"overfull={meta['overfull_readings']}"
|
||||
)
|
||||
print(
|
||||
f"date_range={meta['date_from']}..{meta['date_to']} "
|
||||
f"high={counts['high']} medium={counts['medium']} low={counts['low']}"
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -45,6 +45,26 @@ def raw_csv_path() -> Path:
|
||||
)
|
||||
|
||||
|
||||
def prezero_artifact() -> Path:
|
||||
"""Committed PreZero priority snapshot served by the /prezero routes."""
|
||||
return Path(
|
||||
os.environ.get(
|
||||
"PULSECAST_PREZERO_ARTIFACT",
|
||||
artifacts_dir() / "prezero_priority.json",
|
||||
)
|
||||
).resolve()
|
||||
|
||||
|
||||
def prezero_csv() -> Path:
|
||||
"""Raw fill-level export used only to rebuild the PreZero artifact."""
|
||||
return Path(
|
||||
os.environ.get(
|
||||
"PULSECAST_PREZERO_CSV",
|
||||
_root() / "data" / "raw" / "glass_fill_levels_anonymized_f.csv",
|
||||
)
|
||||
).resolve()
|
||||
|
||||
|
||||
def weather_dir() -> Path:
|
||||
"""Baked weather frames live beside the other exported artifacts."""
|
||||
return Path(os.environ.get("PULSECAST_WEATHER", artifacts_dir())).resolve()
|
||||
|
||||
@@ -0,0 +1,384 @@
|
||||
"""PreZero container-priority snapshot: load, validate, filter, page, lookup.
|
||||
|
||||
The committed ``artifacts/prezero_priority.json`` is the API's source of truth.
|
||||
The dataset has no coordinates, so this ranks collection priority; it does not
|
||||
produce routes. Tiers follow the contract exactly:
|
||||
|
||||
- high: overfull_count >= 5 or (overfull_count >= 2 and overfull_rate >= 0.25)
|
||||
- medium: overfull_count >= 1 and not high
|
||||
- low: overfull_count == 0
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from collections import Counter
|
||||
from collections.abc import Mapping
|
||||
from datetime import date
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
PRIORITIES = ("high", "medium", "low")
|
||||
PRIORITY_RANK = {name: rank for rank, name in enumerate(PRIORITIES)}
|
||||
MAX_LIMIT = 500
|
||||
DEFAULT_LIMIT = 50
|
||||
|
||||
META_KEYS = (
|
||||
"source",
|
||||
"generated_at",
|
||||
"rows",
|
||||
"containers",
|
||||
"material_series",
|
||||
"date_from",
|
||||
"date_to",
|
||||
"reference_date",
|
||||
"overfull_readings",
|
||||
"priority_rule",
|
||||
"limitations",
|
||||
)
|
||||
SUMMARY_KEYS = ("priority_counts", "by_material")
|
||||
MATERIAL_KEYS = (
|
||||
"readings",
|
||||
"overfull_count",
|
||||
"last_fill",
|
||||
"last_is_overfull",
|
||||
"last_date",
|
||||
)
|
||||
READING_KEYS = ("date", "material_type", "fill_level_perc", "is_overfull")
|
||||
CONTAINER_KEYS = (
|
||||
"location_id",
|
||||
"priority",
|
||||
"priority_score",
|
||||
"readings",
|
||||
"overfull_count",
|
||||
"overfull_rate",
|
||||
"numeric_count",
|
||||
"p95_fill",
|
||||
"max_fill",
|
||||
"mean_fill",
|
||||
"last_fill",
|
||||
"last_is_overfull",
|
||||
"last_date",
|
||||
"days_since_last",
|
||||
"trend",
|
||||
"materials",
|
||||
"recent_readings",
|
||||
)
|
||||
PUBLIC_CONTAINER_KEYS = tuple(
|
||||
key for key in CONTAINER_KEYS if key != "recent_readings"
|
||||
)
|
||||
|
||||
|
||||
class PrezeroArtifactError(ValueError):
|
||||
"""The PreZero artifact is missing, unreadable, or inconsistent."""
|
||||
|
||||
|
||||
def priority_tier(overfull_count: int, overfull_rate: float) -> str:
|
||||
"""Return the contract tier. ``priority_score`` is display-only."""
|
||||
if overfull_count >= 5 or (overfull_count >= 2 and overfull_rate >= 0.25):
|
||||
return "high"
|
||||
if overfull_count >= 1:
|
||||
return "medium"
|
||||
return "low"
|
||||
|
||||
|
||||
def container_sort_key(container: Mapping[str, Any]) -> tuple[int, int, float, int, str]:
|
||||
"""Priority first, then overfull count/rate, staleness, and location id."""
|
||||
return (
|
||||
PRIORITY_RANK[container["priority"]],
|
||||
-int(container["overfull_count"]),
|
||||
-float(container["overfull_rate"]),
|
||||
int(container["days_since_last"]),
|
||||
str(container["location_id"]),
|
||||
)
|
||||
|
||||
|
||||
def _fail(where: str, message: str) -> None:
|
||||
raise PrezeroArtifactError(f"{where}: {message}")
|
||||
|
||||
|
||||
def _require(mapping: Mapping[str, Any], keys: tuple[str, ...], where: str) -> None:
|
||||
if not isinstance(mapping, Mapping):
|
||||
_fail(where, "must be an object")
|
||||
missing = [key for key in keys if key not in mapping]
|
||||
if missing:
|
||||
_fail(where, f"missing keys: {', '.join(missing)}")
|
||||
|
||||
|
||||
def _integer(value: Any, where: str, *, minimum: int = 0) -> int:
|
||||
if isinstance(value, bool) or not isinstance(value, int):
|
||||
_fail(where, "must be an integer")
|
||||
if value < minimum:
|
||||
_fail(where, f"must be >= {minimum}")
|
||||
return value
|
||||
|
||||
|
||||
def _number(value: Any, where: str) -> float:
|
||||
if isinstance(value, bool) or not isinstance(value, (int, float)):
|
||||
_fail(where, "must be a number")
|
||||
return float(value)
|
||||
|
||||
|
||||
def _optional_number(value: Any, where: str) -> float | None:
|
||||
return None if value is None else _number(value, where)
|
||||
|
||||
|
||||
def _optional_fill(value: Any, where: str) -> int | None:
|
||||
if value is None:
|
||||
return None
|
||||
fill = _integer(value, where)
|
||||
if fill > 100:
|
||||
_fail(where, "must be between 0 and 100")
|
||||
return fill
|
||||
|
||||
|
||||
def _date(value: Any, where: str) -> date:
|
||||
if not isinstance(value, str):
|
||||
_fail(where, "must be a YYYY-MM-DD string")
|
||||
try:
|
||||
return date.fromisoformat(value)
|
||||
except ValueError as exc:
|
||||
raise PrezeroArtifactError(f"{where}: invalid date {value!r}") from exc
|
||||
|
||||
|
||||
def _validate_material(item: Any, where: str) -> None:
|
||||
_require(item, MATERIAL_KEYS, where)
|
||||
readings = _integer(item["readings"], f"{where}.readings")
|
||||
overfull = _integer(item["overfull_count"], f"{where}.overfull_count")
|
||||
if overfull > readings:
|
||||
_fail(where, "overfull_count exceeds readings")
|
||||
_optional_fill(item["last_fill"], f"{where}.last_fill")
|
||||
if not isinstance(item["last_is_overfull"], bool):
|
||||
_fail(f"{where}.last_is_overfull", "must be a boolean")
|
||||
_date(item["last_date"], f"{where}.last_date")
|
||||
|
||||
|
||||
def _validate_reading(item: Any, where: str) -> None:
|
||||
_require(item, READING_KEYS, where)
|
||||
_date(item["date"], f"{where}.date")
|
||||
if not isinstance(item["material_type"], str) or not item["material_type"]:
|
||||
_fail(f"{where}.material_type", "must be a non-empty string")
|
||||
if not isinstance(item["fill_level_perc"], str):
|
||||
_fail(f"{where}.fill_level_perc", "must be a string")
|
||||
if not isinstance(item["is_overfull"], bool):
|
||||
_fail(f"{where}.is_overfull", "must be a boolean")
|
||||
if item["is_overfull"] is not (item["fill_level_perc"] == "overfull"):
|
||||
_fail(f"{where}.is_overfull", "does not match fill_level_perc")
|
||||
|
||||
|
||||
def _validate_container(item: Any, where: str) -> None:
|
||||
_require(item, CONTAINER_KEYS, where)
|
||||
location_id = item["location_id"]
|
||||
if not isinstance(location_id, str) or not location_id.strip():
|
||||
_fail(f"{where}.location_id", "must be a non-empty string")
|
||||
priority = item["priority"]
|
||||
if priority not in PRIORITIES:
|
||||
_fail(f"{where}.priority", f"must be one of {', '.join(PRIORITIES)}")
|
||||
|
||||
readings = _integer(item["readings"], f"{where}.readings")
|
||||
overfull = _integer(item["overfull_count"], f"{where}.overfull_count")
|
||||
numeric = _integer(item["numeric_count"], f"{where}.numeric_count")
|
||||
if overfull > readings:
|
||||
_fail(where, "overfull_count exceeds readings")
|
||||
if numeric > readings:
|
||||
_fail(where, "numeric_count exceeds readings")
|
||||
rate = _number(item["overfull_rate"], f"{where}.overfull_rate")
|
||||
expected_rate = overfull / readings if readings else 0.0
|
||||
if abs(rate - expected_rate) > 1e-9:
|
||||
_fail(f"{where}.overfull_rate", "does not match overfull_count / readings")
|
||||
if priority_tier(overfull, rate) != priority:
|
||||
_fail(f"{where}.priority", "does not match the contract tier rule")
|
||||
|
||||
score = _number(item["priority_score"], f"{where}.priority_score")
|
||||
if abs(score - round(overfull + 2 * rate, 4)) > 1e-9:
|
||||
_fail(f"{where}.priority_score", "does not match the contract score")
|
||||
for key in ("p95_fill", "max_fill", "mean_fill"):
|
||||
_optional_number(item[key], f"{where}.{key}")
|
||||
_optional_fill(item["last_fill"], f"{where}.last_fill")
|
||||
if not isinstance(item["last_is_overfull"], bool):
|
||||
_fail(f"{where}.last_is_overfull", "must be a boolean")
|
||||
_date(item["last_date"], f"{where}.last_date")
|
||||
_integer(item["days_since_last"], f"{where}.days_since_last")
|
||||
_optional_number(item["trend"], f"{where}.trend")
|
||||
|
||||
materials = item["materials"]
|
||||
if not isinstance(materials, Mapping) or not materials:
|
||||
_fail(f"{where}.materials", "must be a non-empty object")
|
||||
material_readings = 0
|
||||
material_overfull = 0
|
||||
for material, entry in materials.items():
|
||||
if not isinstance(material, str) or not material:
|
||||
_fail(f"{where}.materials", "keys must be non-empty material names")
|
||||
_validate_material(entry, f"{where}.materials.{material}")
|
||||
material_readings += entry["readings"]
|
||||
material_overfull += entry["overfull_count"]
|
||||
if material_readings != readings or material_overfull != overfull:
|
||||
_fail(f"{where}.materials", "per-material totals do not match the container")
|
||||
|
||||
recent = item["recent_readings"]
|
||||
if not isinstance(recent, list):
|
||||
_fail(f"{where}.recent_readings", "must be an array")
|
||||
if len(recent) > 10:
|
||||
_fail(f"{where}.recent_readings", "holds at most 10 rows")
|
||||
for index, reading in enumerate(recent):
|
||||
_validate_reading(reading, f"{where}.recent_readings[{index}]")
|
||||
|
||||
|
||||
def _validate_meta(meta: Any, container_count: int) -> None:
|
||||
_require(meta, META_KEYS, "meta")
|
||||
if not isinstance(meta["source"], str) or not meta["source"]:
|
||||
_fail("meta.source", "must be a non-empty string")
|
||||
if not isinstance(meta["generated_at"], str) or not meta["generated_at"]:
|
||||
_fail("meta.generated_at", "must be a non-empty string")
|
||||
for key in ("rows", "containers", "material_series", "overfull_readings"):
|
||||
_integer(meta[key], f"meta.{key}")
|
||||
if meta["containers"] != container_count:
|
||||
_fail("meta.containers", "does not match the container array")
|
||||
date_from = _date(meta["date_from"], "meta.date_from")
|
||||
date_to = _date(meta["date_to"], "meta.date_to")
|
||||
reference = _date(meta["reference_date"], "meta.reference_date")
|
||||
if date_from > date_to:
|
||||
_fail("meta.date_from", "is after meta.date_to")
|
||||
if reference != date_to:
|
||||
_fail("meta.reference_date", "must equal the dataset max date")
|
||||
if not isinstance(meta["priority_rule"], str) or not meta["priority_rule"]:
|
||||
_fail("meta.priority_rule", "must be a non-empty string")
|
||||
limitations = meta["limitations"]
|
||||
if not isinstance(limitations, list) or not all(
|
||||
isinstance(line, str) and line for line in limitations
|
||||
):
|
||||
_fail("meta.limitations", "must be an array of non-empty strings")
|
||||
|
||||
|
||||
def _validate_summary(summary: Any, meta: Mapping[str, Any]) -> None:
|
||||
_require(summary, SUMMARY_KEYS, "summary")
|
||||
counts = summary["priority_counts"]
|
||||
_require(counts, PRIORITIES, "summary.priority_counts")
|
||||
total = 0
|
||||
for priority in PRIORITIES:
|
||||
total += _integer(counts[priority], f"summary.priority_counts.{priority}")
|
||||
if total != meta["containers"]:
|
||||
_fail("summary.priority_counts", "does not add up to meta.containers")
|
||||
|
||||
by_material = summary["by_material"]
|
||||
if not isinstance(by_material, Mapping) or not by_material:
|
||||
_fail("summary.by_material", "must be a non-empty object")
|
||||
readings = 0
|
||||
overfull = 0
|
||||
for material, entry in by_material.items():
|
||||
where = f"summary.by_material.{material}"
|
||||
_require(entry, ("readings", "overfull"), where)
|
||||
entry_readings = _integer(entry["readings"], f"{where}.readings")
|
||||
entry_overfull = _integer(entry["overfull"], f"{where}.overfull")
|
||||
if entry_overfull > entry_readings:
|
||||
_fail(where, "overfull exceeds readings")
|
||||
readings += entry_readings
|
||||
overfull += entry_overfull
|
||||
if readings != meta["rows"] or overfull != meta["overfull_readings"]:
|
||||
_fail("summary.by_material", "totals do not match meta rows/overfull")
|
||||
|
||||
|
||||
class PrezeroStore:
|
||||
"""Validated, in-memory view of the committed priority artifact."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
meta: dict[str, Any],
|
||||
summary: dict[str, Any],
|
||||
containers: list[dict[str, Any]],
|
||||
) -> None:
|
||||
self.meta = meta
|
||||
self.summary = summary
|
||||
self.containers = tuple(sorted(containers, key=container_sort_key))
|
||||
self._by_id = {container["location_id"]: container for container in self.containers}
|
||||
|
||||
@classmethod
|
||||
def from_payload(cls, payload: Any) -> PrezeroStore:
|
||||
_require(payload, ("meta", "summary", "containers"), "artifact")
|
||||
containers = payload["containers"]
|
||||
if not isinstance(containers, list) or not containers:
|
||||
_fail("containers", "must be a non-empty array")
|
||||
for index, container in enumerate(containers):
|
||||
_validate_container(container, f"containers[{index}]")
|
||||
|
||||
meta = payload["meta"]
|
||||
_validate_meta(meta, len(containers))
|
||||
_validate_summary(payload["summary"], meta)
|
||||
|
||||
ids = [container["location_id"] for container in containers]
|
||||
duplicates = sorted(
|
||||
location for location, count in Counter(ids).items() if count > 1
|
||||
)
|
||||
if duplicates:
|
||||
_fail("containers", f"duplicate location_id values: {', '.join(duplicates)}")
|
||||
|
||||
actual = Counter(container["priority"] for container in containers)
|
||||
for priority in PRIORITIES:
|
||||
if actual[priority] != payload["summary"]["priority_counts"][priority]:
|
||||
_fail(
|
||||
"summary.priority_counts",
|
||||
f"{priority} count does not match the container array",
|
||||
)
|
||||
return cls(meta=meta, summary=payload["summary"], containers=containers)
|
||||
|
||||
def summary_payload(self) -> dict[str, Any]:
|
||||
return {"meta": self.meta, "summary": self.summary}
|
||||
|
||||
def priority_payload(
|
||||
self,
|
||||
priority: str | None = None,
|
||||
limit: int = DEFAULT_LIMIT,
|
||||
offset: int = 0,
|
||||
) -> dict[str, Any]:
|
||||
if priority is not None and priority not in PRIORITIES:
|
||||
raise ValueError(f"priority must be one of {', '.join(PRIORITIES)}")
|
||||
if (
|
||||
isinstance(limit, bool)
|
||||
or not isinstance(limit, int)
|
||||
or not 1 <= limit <= MAX_LIMIT
|
||||
):
|
||||
raise ValueError(f"limit must be between 1 and {MAX_LIMIT}")
|
||||
if isinstance(offset, bool) or not isinstance(offset, int) or offset < 0:
|
||||
raise ValueError("offset must be >= 0")
|
||||
|
||||
matching = [
|
||||
container
|
||||
for container in self.containers
|
||||
if priority is None or container["priority"] == priority
|
||||
]
|
||||
page = matching[offset : offset + limit]
|
||||
return {
|
||||
"meta": self.meta,
|
||||
"total": len(matching),
|
||||
"count": len(page),
|
||||
"offset": offset,
|
||||
"limit": limit,
|
||||
"containers": [
|
||||
{key: container[key] for key in PUBLIC_CONTAINER_KEYS}
|
||||
for container in page
|
||||
],
|
||||
}
|
||||
|
||||
def container_payload(self, location_id: str) -> dict[str, Any] | None:
|
||||
container = self._by_id.get(location_id)
|
||||
if container is None:
|
||||
return None
|
||||
return {
|
||||
"container": {key: container[key] for key in PUBLIC_CONTAINER_KEYS},
|
||||
"readings": container["recent_readings"],
|
||||
}
|
||||
|
||||
|
||||
def load_prezero(path: Path) -> PrezeroStore:
|
||||
"""Read and validate the artifact. Raises FileNotFoundError when absent."""
|
||||
try:
|
||||
raw = path.read_text(encoding="utf-8")
|
||||
except FileNotFoundError:
|
||||
raise
|
||||
except (OSError, UnicodeDecodeError) as exc:
|
||||
raise PrezeroArtifactError(f"cannot read {path}: {exc}") from exc
|
||||
try:
|
||||
payload = json.loads(raw)
|
||||
except json.JSONDecodeError as exc:
|
||||
raise PrezeroArtifactError(f"invalid JSON in {path}: {exc}") from exc
|
||||
return PrezeroStore.from_payload(payload)
|
||||
@@ -29,6 +29,7 @@ from pydantic import BaseModel, ConfigDict
|
||||
from . import config
|
||||
from .features import future_calendar
|
||||
from .model import event_multiplier, future_frame, points, predict
|
||||
from .prezero import PrezeroStore, load_prezero
|
||||
from .retrain import train_forecast_from_csv
|
||||
from .small import train_small_from_csv
|
||||
|
||||
@@ -54,6 +55,7 @@ class Runtime:
|
||||
context: dict | None = None
|
||||
small_bundle: dict | None = None
|
||||
small_metrics: dict | None = None
|
||||
prezero: PrezeroStore | None = None
|
||||
model_source: str = "baked"
|
||||
jobs: dict[str, dict[str, Any]] = field(default_factory=dict)
|
||||
jobs_lock: threading.Lock = field(default_factory=threading.Lock)
|
||||
@@ -117,6 +119,24 @@ def _load_runtime(runtime: Runtime) -> None:
|
||||
if small_metrics_path.exists():
|
||||
runtime.small_metrics = json.loads(small_metrics_path.read_text())
|
||||
|
||||
# PreZero is an optional proof surface: a missing or malformed artifact
|
||||
# must disable only /prezero/*, never the forecast/training routes.
|
||||
prezero_path = config.prezero_artifact()
|
||||
try:
|
||||
runtime.prezero = load_prezero(prezero_path)
|
||||
log.info(
|
||||
"loaded prezero artifact with %d containers",
|
||||
len(runtime.prezero.containers),
|
||||
)
|
||||
except FileNotFoundError:
|
||||
log.warning("prezero artifact missing: %s; /prezero routes will return 503", prezero_path)
|
||||
except Exception as exc:
|
||||
log.error(
|
||||
"prezero artifact invalid (%s); /prezero routes will return 503: %s",
|
||||
prezero_path,
|
||||
exc,
|
||||
)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- schemas
|
||||
|
||||
@@ -153,6 +173,12 @@ class SimulateRequest(BaseModel):
|
||||
# --------------------------------------------------------------------------- helpers
|
||||
|
||||
|
||||
def _prezero(runtime: Runtime) -> PrezeroStore:
|
||||
if runtime.prezero is None:
|
||||
raise HTTPException(503, "PreZero priority artifact is not loaded")
|
||||
return runtime.prezero
|
||||
|
||||
|
||||
def _require_token(authorization: str | None) -> None:
|
||||
expected = config.control_token()
|
||||
if not expected:
|
||||
@@ -536,6 +562,31 @@ def create_app() -> FastAPI:
|
||||
},
|
||||
}
|
||||
|
||||
@app.get("/prezero/summary")
|
||||
def prezero_summary() -> dict:
|
||||
"""Priority counts and material totals for the PreZero proof."""
|
||||
return _prezero(runtime()).summary_payload()
|
||||
|
||||
@app.get("/prezero/priority")
|
||||
def prezero_priority(
|
||||
priority: str | None = None,
|
||||
limit: int = 50,
|
||||
offset: int = 0,
|
||||
) -> dict:
|
||||
"""Ranked container locations, optionally filtered by priority tier."""
|
||||
try:
|
||||
return _prezero(runtime()).priority_payload(priority, limit, offset)
|
||||
except ValueError as exc:
|
||||
raise HTTPException(400, str(exc)) from exc
|
||||
|
||||
@app.get("/prezero/containers/{location_id}")
|
||||
def prezero_container(location_id: str) -> dict:
|
||||
"""One container location plus its most recent 10 readings."""
|
||||
payload = _prezero(runtime()).container_payload(location_id)
|
||||
if payload is None:
|
||||
raise HTTPException(404, f"unknown location: {location_id}")
|
||||
return payload
|
||||
|
||||
@app.post("/predict")
|
||||
def predict_endpoint(body: PredictRequest) -> dict:
|
||||
state = runtime()
|
||||
|
||||
@@ -0,0 +1,335 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import copy
|
||||
import json
|
||||
from collections import Counter
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from pulsecast import config
|
||||
from pulsecast.prezero import (
|
||||
CONTAINER_KEYS,
|
||||
MATERIAL_KEYS,
|
||||
META_KEYS,
|
||||
PRIORITIES,
|
||||
READING_KEYS,
|
||||
PrezeroArtifactError,
|
||||
PrezeroStore,
|
||||
priority_tier,
|
||||
)
|
||||
|
||||
EXPECTED = {
|
||||
"rows": 11_701,
|
||||
"containers": 449,
|
||||
"material_series": 1_338,
|
||||
"overfull_readings": 1_192,
|
||||
"high": 88,
|
||||
"medium": 248,
|
||||
"low": 113,
|
||||
"date_from": "2025-10-17",
|
||||
"date_to": "2026-09-30",
|
||||
}
|
||||
PUBLIC_KEYS = tuple(key for key in CONTAINER_KEYS if key != "recent_readings")
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def artifact() -> dict:
|
||||
return json.loads(config.prezero_artifact().read_text(encoding="utf-8"))
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def store(artifact) -> PrezeroStore:
|
||||
return PrezeroStore.from_payload(artifact)
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def summary(client) -> dict:
|
||||
response = client.get("/prezero/summary")
|
||||
assert response.status_code == 200, response.text
|
||||
return response.json()
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- rule
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("overfull_count", "overfull_rate", "expected"),
|
||||
[
|
||||
(0, 0.0, "low"),
|
||||
(1, 0.01, "medium"),
|
||||
(4, 0.99, "high"),
|
||||
(2, 0.25, "high"),
|
||||
(2, 0.2499, "medium"),
|
||||
(5, 0.0, "high"),
|
||||
(1, 1.0, "medium"),
|
||||
],
|
||||
)
|
||||
def test_priority_rule(overfull_count, overfull_rate, expected):
|
||||
assert priority_tier(overfull_count, overfull_rate) == expected
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- summary
|
||||
|
||||
|
||||
def test_summary_counts(summary):
|
||||
assert set(summary) == {"meta", "summary"}
|
||||
meta = summary["meta"]
|
||||
for key in META_KEYS:
|
||||
assert key in meta, key
|
||||
assert meta["rows"] == EXPECTED["rows"]
|
||||
assert meta["containers"] == EXPECTED["containers"]
|
||||
assert meta["material_series"] == EXPECTED["material_series"]
|
||||
assert meta["overfull_readings"] == EXPECTED["overfull_readings"]
|
||||
assert meta["date_from"] == EXPECTED["date_from"]
|
||||
assert meta["date_to"] == EXPECTED["date_to"]
|
||||
assert meta["reference_date"] == EXPECTED["date_to"]
|
||||
assert meta["source"] == "glass_fill_levels_anonymized_f.csv"
|
||||
assert "overfull_count >= 5" in meta["priority_rule"]
|
||||
assert any("not a route" in line for line in meta["limitations"])
|
||||
|
||||
counts = summary["summary"]["priority_counts"]
|
||||
assert counts == {
|
||||
"high": EXPECTED["high"],
|
||||
"medium": EXPECTED["medium"],
|
||||
"low": EXPECTED["low"],
|
||||
}
|
||||
assert summary["summary"]["by_material"] == {
|
||||
"white": {"readings": 4_432, "overfull": 731},
|
||||
"green": {"readings": 4_169, "overfull": 357},
|
||||
"brown": {"readings": 3_100, "overfull": 104},
|
||||
}
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- priority
|
||||
|
||||
|
||||
def test_priority_defaults_and_filter_totals(client, summary):
|
||||
payload = client.get("/prezero/priority").json()
|
||||
assert set(payload) == {"meta", "total", "count", "offset", "limit", "containers"}
|
||||
assert payload["meta"] == summary["meta"]
|
||||
assert payload["total"] == EXPECTED["containers"]
|
||||
assert payload["limit"] == 50
|
||||
assert payload["offset"] == 0
|
||||
assert payload["count"] == 50
|
||||
assert all(container["priority"] == "high" for container in payload["containers"])
|
||||
for priority in PRIORITIES:
|
||||
filtered = client.get("/prezero/priority", params={"priority": priority}).json()
|
||||
assert filtered["total"] == EXPECTED[priority]
|
||||
assert filtered["count"] == min(50, EXPECTED[priority])
|
||||
assert all(row["priority"] == priority for row in filtered["containers"])
|
||||
|
||||
|
||||
def test_priority_limit_offset_and_count(client, store, artifact):
|
||||
payload = client.get(
|
||||
"/prezero/priority", params={"priority": "high", "limit": 3}
|
||||
).json()
|
||||
assert payload["total"] == EXPECTED["high"]
|
||||
assert payload["count"] == 3
|
||||
high_ids = [
|
||||
row["location_id"]
|
||||
for row in artifact["containers"]
|
||||
if row["priority"] == "high"
|
||||
]
|
||||
expected_ids = high_ids[:3]
|
||||
assert [row["location_id"] for row in payload["containers"]] == expected_ids
|
||||
for row in payload["containers"]:
|
||||
assert set(row) == set(PUBLIC_KEYS)
|
||||
assert "recent_readings" not in row
|
||||
|
||||
paged = client.get(
|
||||
"/prezero/priority",
|
||||
params={"priority": "high", "limit": 2, "offset": 2},
|
||||
).json()
|
||||
assert paged["total"] == EXPECTED["high"]
|
||||
assert paged["count"] == 2
|
||||
assert paged["offset"] == 2
|
||||
assert [row["location_id"] for row in paged["containers"]] == high_ids[2:4]
|
||||
|
||||
beyond = client.get(
|
||||
"/prezero/priority", params={"priority": "high", "limit": 10, "offset": 88}
|
||||
).json()
|
||||
assert beyond["total"] == EXPECTED["high"]
|
||||
assert beyond["count"] == 0
|
||||
assert beyond["containers"] == []
|
||||
|
||||
|
||||
def test_priority_is_sorted_by_contract(client, store):
|
||||
payload = client.get("/prezero/priority", params={"limit": 500}).json()
|
||||
assert payload["total"] == EXPECTED["containers"]
|
||||
assert payload["count"] == EXPECTED["containers"]
|
||||
rows = payload["containers"]
|
||||
assert [row["location_id"] for row in rows] == [
|
||||
container["location_id"] for container in store.containers
|
||||
]
|
||||
rank = {priority: index for index, priority in enumerate(PRIORITIES)}
|
||||
keys = [
|
||||
(
|
||||
rank[row["priority"]],
|
||||
-row["overfull_count"],
|
||||
-row["overfull_rate"],
|
||||
row["days_since_last"],
|
||||
row["location_id"],
|
||||
)
|
||||
for row in rows
|
||||
]
|
||||
assert keys == sorted(keys)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- container
|
||||
|
||||
|
||||
def test_known_container_and_recent_readings(client, artifact):
|
||||
known = next(row for row in artifact["containers"] if row["priority"] == "high")
|
||||
location_id = known["location_id"]
|
||||
response = client.get(f"/prezero/containers/{location_id}")
|
||||
assert response.status_code == 200, response.text
|
||||
payload = response.json()
|
||||
|
||||
assert set(payload) == {"container", "readings"}
|
||||
container = payload["container"]
|
||||
assert set(container) == set(PUBLIC_KEYS)
|
||||
for key in PUBLIC_KEYS:
|
||||
assert container[key] == known[key]
|
||||
assert len(payload["readings"]) == min(10, known["readings"])
|
||||
dates = [reading["date"] for reading in payload["readings"]]
|
||||
assert dates == sorted(dates, reverse=True)
|
||||
for reading in payload["readings"]:
|
||||
assert set(reading) == set(READING_KEYS)
|
||||
assert isinstance(reading["fill_level_perc"], str)
|
||||
assert reading["is_overfull"] is (reading["fill_level_perc"] == "overfull")
|
||||
|
||||
|
||||
def test_sparse_container_uses_null_numeric_fields(client, artifact):
|
||||
sparse = next(row for row in artifact["containers"] if row["numeric_count"] == 0)
|
||||
payload = client.get(f"/prezero/containers/{sparse['location_id']}").json()
|
||||
assert payload["container"]["p95_fill"] is None
|
||||
assert payload["container"]["max_fill"] is None
|
||||
assert payload["container"]["mean_fill"] is None
|
||||
assert payload["container"]["trend"] is None
|
||||
assert payload["container"]["last_is_overfull"] is True
|
||||
assert payload["container"]["last_fill"] is None
|
||||
|
||||
|
||||
def test_unknown_container_404(client):
|
||||
response = client.get("/prezero/containers/does-not-exist")
|
||||
assert response.status_code == 404
|
||||
assert response.json() == {
|
||||
"error": {
|
||||
"code": "not_found",
|
||||
"message": "unknown location: does-not-exist",
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("params", "message"),
|
||||
[
|
||||
({"priority": "urgent"}, "priority must be one of high, medium, low"),
|
||||
({"priority": "HIGH"}, "priority must be one of high, medium, low"),
|
||||
({"limit": 0}, "limit must be between 1 and 500"),
|
||||
({"limit": 501}, "limit must be between 1 and 500"),
|
||||
({"offset": -1}, "offset must be >= 0"),
|
||||
],
|
||||
)
|
||||
def test_invalid_queries_400(client, params, message):
|
||||
response = client.get("/prezero/priority", params=params)
|
||||
assert response.status_code == 400
|
||||
assert response.json() == {"error": {"code": "bad_request", "message": message}}
|
||||
|
||||
|
||||
def test_non_integer_query_400(client):
|
||||
response = client.get("/prezero/priority", params={"limit": "many"})
|
||||
assert response.status_code == 400
|
||||
assert response.json()["error"]["code"] == "bad_request"
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- artifact
|
||||
|
||||
|
||||
def test_artifact_schema(artifact, store):
|
||||
assert set(artifact) == {"meta", "summary", "containers"}
|
||||
meta = artifact["meta"]
|
||||
assert meta["rows"] == EXPECTED["rows"]
|
||||
assert meta["containers"] == EXPECTED["containers"]
|
||||
assert meta["material_series"] == EXPECTED["material_series"]
|
||||
assert meta["overfull_readings"] == EXPECTED["overfull_readings"]
|
||||
assert meta["date_from"] == EXPECTED["date_from"]
|
||||
assert meta["date_to"] == EXPECTED["date_to"]
|
||||
assert meta["reference_date"] == EXPECTED["date_to"]
|
||||
|
||||
containers = artifact["containers"]
|
||||
assert len(containers) == EXPECTED["containers"]
|
||||
assert len(store.containers) == EXPECTED["containers"]
|
||||
assert len({row["location_id"] for row in containers}) == EXPECTED["containers"]
|
||||
assert Counter(row["priority"] for row in containers) == {
|
||||
"high": EXPECTED["high"],
|
||||
"medium": EXPECTED["medium"],
|
||||
"low": EXPECTED["low"],
|
||||
}
|
||||
|
||||
for row in containers:
|
||||
assert set(row) == set(CONTAINER_KEYS)
|
||||
assert row["last_date"].count("-") == 2
|
||||
assert len(row["recent_readings"]) == min(10, row["readings"])
|
||||
assert row["readings"] == sum(
|
||||
material["readings"] for material in row["materials"].values()
|
||||
)
|
||||
assert row["overfull_count"] == sum(
|
||||
material["overfull_count"] for material in row["materials"].values()
|
||||
)
|
||||
for material in row["materials"].values():
|
||||
assert set(material) == set(MATERIAL_KEYS)
|
||||
|
||||
material_totals = Counter()
|
||||
material_overfull = Counter()
|
||||
for row in containers:
|
||||
for name, material in row["materials"].items():
|
||||
material_totals[name] += material["readings"]
|
||||
material_overfull[name] += material["overfull_count"]
|
||||
assert material_totals == {
|
||||
"white": 4_432,
|
||||
"green": 4_169,
|
||||
"brown": 3_100,
|
||||
}
|
||||
assert material_overfull == {"white": 731, "green": 357, "brown": 104}
|
||||
assert sum(row["readings"] for row in containers) == EXPECTED["rows"]
|
||||
assert sum(row["overfull_count"] for row in containers) == EXPECTED[
|
||||
"overfull_readings"
|
||||
]
|
||||
|
||||
|
||||
def test_artifact_rejects_inconsistent_tier(artifact):
|
||||
payload = copy.deepcopy(artifact)
|
||||
payload["containers"][0]["priority"] = "urgent"
|
||||
with pytest.raises(PrezeroArtifactError, match="priority"):
|
||||
PrezeroStore.from_payload(payload)
|
||||
|
||||
|
||||
def test_artifact_rejects_wrong_summary_count(artifact):
|
||||
payload = copy.deepcopy(artifact)
|
||||
payload["summary"]["priority_counts"]["high"] += 1
|
||||
with pytest.raises(PrezeroArtifactError, match="priority_counts"):
|
||||
PrezeroStore.from_payload(payload)
|
||||
|
||||
|
||||
def test_missing_artifact_returns_503(tmp_path, monkeypatch):
|
||||
monkeypatch.setenv(
|
||||
"PULSECAST_PREZERO_ARTIFACT", str(Path(tmp_path) / "missing.json")
|
||||
)
|
||||
from pulsecast.server import create_app
|
||||
|
||||
with TestClient(create_app()) as isolated:
|
||||
assert isolated.get("/healthz").status_code == 200
|
||||
assert isolated.get("/readyz").status_code == 200
|
||||
assert isolated.get("/meta").status_code == 200
|
||||
assert isolated.get("/bootstrap").status_code == 200
|
||||
for path in (
|
||||
"/prezero/summary",
|
||||
"/prezero/priority",
|
||||
"/prezero/containers/anything",
|
||||
):
|
||||
response = isolated.get(path)
|
||||
assert response.status_code == 503, path
|
||||
assert response.json()["error"]["code"] == "unavailable"
|
||||
Reference in new issue
Block a user