feat: prezero priority snapshot and internal API
ci / check (push) Successful in 30s
ci / check (pull_request) Successful in 29s

Build artifacts/prezero_priority.json from the anonymized glass fill-level export (11,701 rows, 449 containers, 1,338 location/material series). The contract tier rule yields high 88, medium 248, low 113 with 1,192 overfull readings.

Add /prezero/summary, /prezero/priority, and /prezero/containers/{id} to the internal FastAPI service, with startup validation, filtering/pagination, recent reading history, and 503 isolation when the artifact is missing. Raw CSV stays gitignored; the derived 1.1 MB artifact is the source of truth. This is a priority proof, not a route: the dataset has no coordinates.
This commit is contained in:
Noel Sabia committed 2026-11-14 18:06:02 +01:00
1 parent 507f33b7ea
commit ad63d1caf3
9 files changed
+46198 -4

No files matched your search

+179
View File
@@ -0,0 +1,179 @@
# PreZero container priority
A transparent collection-priority snapshot built from the anonymized glass
fill-level export. The committed artifact is
`artifacts/prezero_priority.json`; the raw CSV is **not** committed.
**This is a priority proof, not a route.** The dataset has no coordinates, no
container capacity, and no depot data. `overfull` is a reported state, not a
measured volume.
## Provenance
| Field | Value |
|---|---:|
| Input | `glass_fill_levels_anonymized_f.csv` (UTF-8 with BOM) |
| Rows | 11,701 |
| Distinct `Location_id` | 449 |
| `Location_id` + `material_type` series | 1,338 |
| Date range | 2025-10-17 → 2026-09-30 |
| `overfull` readings | 1,192 |
| Material readings | white 4,432 · green 4,169 · brown 3,100 |
| Material overfull | white 731 · green 357 · brown 104 |
The export is read with `utf-8-sig`; header names and cells are stripped. Every
row must have `Location_id`, `material_type`, a `YYYY-MM-DD` date, and a
`fill_level_perc` of either `overfull` or an integer 0–100.
The raw file stays git-ignored under `data/raw/`. Put it there or set
`PULSECAST_PREZERO_CSV`; the generated artifact is committed so the API and
Docker image do not need the CSV.
## Rule
Each `Location_id` is one container location. For all rows at that location:
- `readings`, `overfull_count`, `overfull_rate = overfull_count / readings`
- `numeric_count` = rows with a numeric 0–100 fill (excludes `overfull`)
- `p95_fill` = 95th percentile of numeric readings, `null` if none
- `max_fill`, `mean_fill` = numeric readings only
- `last_fill`, `last_is_overfull`, `last_date` = most recent row
- `days_since_last` = `reference_date - last_date`; reference is the dataset
max date (2026-09-30), not today
- `trend` = least-squares slope over the last up to 3 numeric readings
(percentage points per reading); `null` with fewer than 2
- `priority_score = round(overfull_count + 2 * overfull_rate, 4)` — a display
and sorting convenience, not the tier
- `materials` = per material: readings, overfull count, last fill/state/date
Tier:
- **high:** `overfull_count >= 5` OR (`overfull_count >= 2` AND
`overfull_rate >= 0.25`)
- **medium:** `overfull_count >= 1` and not high
- **low:** `overfull_count == 0`
Result on the attached CSV: **high 88, medium 248, low 113**.
Order: tier (`high` first), then overfull count desc, overfull rate desc,
`days_since_last` asc, `location_id` asc.
## Artifact shape
```json
{
"meta": {
"source": "glass_fill_levels_anonymized_f.csv",
"generated_at": "2026-11-14T17:00:00Z",
"rows": 11701,
"containers": 449,
"material_series": 1338,
"date_from": "2025-10-17",
"date_to": "2026-09-30",
"reference_date": "2026-09-30",
"overfull_readings": 1192,
"priority_rule": "high: ...; medium: ...; low: 0",
"limitations": ["No coordinates ...", "..."]
},
"summary": {
"priority_counts": {"high": 88, "medium": 248, "low": 113},
"by_material": {
"white": {"readings": 4432, "overfull": 731},
"green": {"readings": 4169, "overfull": 357},
"brown": {"readings": 3100, "overfull": 104}
}
},
"containers": [
{
"location_id": "...",
"priority": "high",
"priority_score": 21.525,
"readings": 80,
"overfull_count": 21,
"overfull_rate": 0.2625,
"numeric_count": 59,
"p95_fill": 100.0,
"max_fill": 100,
"mean_fill": 64.24,
"last_fill": null,
"last_is_overfull": true,
"last_date": "2026-09-29",
"days_since_last": 1,
"trend": 30.0,
"materials": {"white": {"readings": 29, "...": "..."}},
"recent_readings": [
{"date": "2026-09-29", "material_type": "white",
"fill_level_perc": "overfull", "is_overfull": true}
]
}
]
}
```
The artifact holds at most 10 recent readings per container, not the full
11,701-row table. `last_fill` is `null` when the newest reading is `overfull`.
`p95_fill`, `max_fill`, `mean_fill`, and `trend` are `null` when the location
has no numeric readings or too few for a slope.
## Generation
```bash
# Raw CSV is git-ignored; copy it or set PULSECAST_PREZERO_CSV.
cp glass_fill_levels_anonymized_f.csv data/raw/
make prezero
# Equivalent explicit command:
uv run python scripts/build_prezero.py \
--csv data/raw/glass_fill_levels_anonymized_f.csv \
--out artifacts/prezero_priority.json
```
The script prints rows, containers, material series, overfull readings, date
range, and the three tier counts. It fails loudly on missing columns, blank
cells, bad dates, or invalid fill values instead of guessing.
## Internal API
The ML service loads and validates the artifact at startup. If it is missing or
malformed, only these routes return 503; `/healthz`, `/readyz`, `/bootstrap`,
`/predict`, `/simulate`, and `/meta` keep working.
| Method | Path | Behavior |
|---|---|---|
| GET | `/prezero/summary` | `meta` + `summary` |
| GET | `/prezero/priority` | Ranked containers; `priority` (`high\|medium\|low`), `limit` 1–500 (default 50), `offset` >= 0 |
| GET | `/prezero/containers/{location_id}` | One container + 10 newest readings; unknown → 404 |
Invalid `priority`, `limit`, or `offset` → 400 in the existing error envelope.
The priority list omits `recent_readings`; the container route returns it as
`readings`. `/prezero/priority` reports `total` before pagination and `count`
after it.
```bash
curl -s localhost:8000/prezero/summary
curl -s 'localhost:8000/prezero/priority?priority=high&limit=3'
curl -s localhost:8000/prezero/containers/<known-id>
```
## Verification
```bash
uv run pytest -q tests/test_prezero.py
make check
```
The tests assert the dataset counts above (11,701 rows, 449 containers, 1,338
series, 1,192 overfull, 88/248/113 tiers), filtering/pagination, known and
unknown containers, invalid queries, artifact schema and tier consistency, and
the missing-artifact 503 path.
## Limits
- No coordinates: cannot compute distance, travel time, or a route.
- No capacity, depot, vehicle, or shift data.
- `overfull` is a reported state; fill percentages are coarse, user-reported
values (0/10/.../100), not continuous sensor volumes.
- The reference date is the dataset max date, so staleness is relative to the
snapshot, not to when the API is called.
- Counts and rates describe this export only; they are not a forecast of future
fill levels.