feat: prezero priority snapshot and internal API
Build artifacts/prezero_priority.json from the anonymized glass fill-level export (11,701 rows, 449 containers, 1,338 location/material series). The contract tier rule yields high 88, medium 248, low 113 with 1,192 overfull readings.
Add /prezero/summary, /prezero/priority, and /prezero/containers/{id} to the internal FastAPI service, with startup validation, filtering/pagination, recent reading history, and 503 isolation when the artifact is missing. Raw CSV stays gitignored; the derived 1.1 MB artifact is the source of truth. This is a priority proof, not a route: the dataset has no coordinates.
This commit is contained in:
1 parent
507f33b7ea
commit
ad63d1caf3
9 files changed
+46198
-4
No files matched your search
+179
@@ -0,0 +1,179 @@
|
||||
# PreZero container priority
|
||||
|
||||
A transparent collection-priority snapshot built from the anonymized glass
|
||||
fill-level export. The committed artifact is
|
||||
`artifacts/prezero_priority.json`; the raw CSV is **not** committed.
|
||||
|
||||
**This is a priority proof, not a route.** The dataset has no coordinates, no
|
||||
container capacity, and no depot data. `overfull` is a reported state, not a
|
||||
measured volume.
|
||||
|
||||
## Provenance
|
||||
|
||||
| Field | Value |
|
||||
|---|---:|
|
||||
| Input | `glass_fill_levels_anonymized_f.csv` (UTF-8 with BOM) |
|
||||
| Rows | 11,701 |
|
||||
| Distinct `Location_id` | 449 |
|
||||
| `Location_id` + `material_type` series | 1,338 |
|
||||
| Date range | 2025-10-17 → 2026-09-30 |
|
||||
| `overfull` readings | 1,192 |
|
||||
| Material readings | white 4,432 · green 4,169 · brown 3,100 |
|
||||
| Material overfull | white 731 · green 357 · brown 104 |
|
||||
|
||||
The export is read with `utf-8-sig`; header names and cells are stripped. Every
|
||||
row must have `Location_id`, `material_type`, a `YYYY-MM-DD` date, and a
|
||||
`fill_level_perc` of either `overfull` or an integer 0–100.
|
||||
|
||||
The raw file stays git-ignored under `data/raw/`. Put it there or set
|
||||
`PULSECAST_PREZERO_CSV`; the generated artifact is committed so the API and
|
||||
Docker image do not need the CSV.
|
||||
|
||||
## Rule
|
||||
|
||||
Each `Location_id` is one container location. For all rows at that location:
|
||||
|
||||
- `readings`, `overfull_count`, `overfull_rate = overfull_count / readings`
|
||||
- `numeric_count` = rows with a numeric 0–100 fill (excludes `overfull`)
|
||||
- `p95_fill` = 95th percentile of numeric readings, `null` if none
|
||||
- `max_fill`, `mean_fill` = numeric readings only
|
||||
- `last_fill`, `last_is_overfull`, `last_date` = most recent row
|
||||
- `days_since_last` = `reference_date - last_date`; reference is the dataset
|
||||
max date (2026-09-30), not today
|
||||
- `trend` = least-squares slope over the last up to 3 numeric readings
|
||||
(percentage points per reading); `null` with fewer than 2
|
||||
- `priority_score = round(overfull_count + 2 * overfull_rate, 4)` — a display
|
||||
and sorting convenience, not the tier
|
||||
- `materials` = per material: readings, overfull count, last fill/state/date
|
||||
|
||||
Tier:
|
||||
|
||||
- **high:** `overfull_count >= 5` OR (`overfull_count >= 2` AND
|
||||
`overfull_rate >= 0.25`)
|
||||
- **medium:** `overfull_count >= 1` and not high
|
||||
- **low:** `overfull_count == 0`
|
||||
|
||||
Result on the attached CSV: **high 88, medium 248, low 113**.
|
||||
|
||||
Order: tier (`high` first), then overfull count desc, overfull rate desc,
|
||||
`days_since_last` asc, `location_id` asc.
|
||||
|
||||
## Artifact shape
|
||||
|
||||
```json
|
||||
{
|
||||
"meta": {
|
||||
"source": "glass_fill_levels_anonymized_f.csv",
|
||||
"generated_at": "2026-11-14T17:00:00Z",
|
||||
"rows": 11701,
|
||||
"containers": 449,
|
||||
"material_series": 1338,
|
||||
"date_from": "2025-10-17",
|
||||
"date_to": "2026-09-30",
|
||||
"reference_date": "2026-09-30",
|
||||
"overfull_readings": 1192,
|
||||
"priority_rule": "high: ...; medium: ...; low: 0",
|
||||
"limitations": ["No coordinates ...", "..."]
|
||||
},
|
||||
"summary": {
|
||||
"priority_counts": {"high": 88, "medium": 248, "low": 113},
|
||||
"by_material": {
|
||||
"white": {"readings": 4432, "overfull": 731},
|
||||
"green": {"readings": 4169, "overfull": 357},
|
||||
"brown": {"readings": 3100, "overfull": 104}
|
||||
}
|
||||
},
|
||||
"containers": [
|
||||
{
|
||||
"location_id": "...",
|
||||
"priority": "high",
|
||||
"priority_score": 21.525,
|
||||
"readings": 80,
|
||||
"overfull_count": 21,
|
||||
"overfull_rate": 0.2625,
|
||||
"numeric_count": 59,
|
||||
"p95_fill": 100.0,
|
||||
"max_fill": 100,
|
||||
"mean_fill": 64.24,
|
||||
"last_fill": null,
|
||||
"last_is_overfull": true,
|
||||
"last_date": "2026-09-29",
|
||||
"days_since_last": 1,
|
||||
"trend": 30.0,
|
||||
"materials": {"white": {"readings": 29, "...": "..."}},
|
||||
"recent_readings": [
|
||||
{"date": "2026-09-29", "material_type": "white",
|
||||
"fill_level_perc": "overfull", "is_overfull": true}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
The artifact holds at most 10 recent readings per container, not the full
|
||||
11,701-row table. `last_fill` is `null` when the newest reading is `overfull`.
|
||||
`p95_fill`, `max_fill`, `mean_fill`, and `trend` are `null` when the location
|
||||
has no numeric readings or too few for a slope.
|
||||
|
||||
## Generation
|
||||
|
||||
```bash
|
||||
# Raw CSV is git-ignored; copy it or set PULSECAST_PREZERO_CSV.
|
||||
cp glass_fill_levels_anonymized_f.csv data/raw/
|
||||
make prezero
|
||||
|
||||
# Equivalent explicit command:
|
||||
uv run python scripts/build_prezero.py \
|
||||
--csv data/raw/glass_fill_levels_anonymized_f.csv \
|
||||
--out artifacts/prezero_priority.json
|
||||
```
|
||||
|
||||
The script prints rows, containers, material series, overfull readings, date
|
||||
range, and the three tier counts. It fails loudly on missing columns, blank
|
||||
cells, bad dates, or invalid fill values instead of guessing.
|
||||
|
||||
## Internal API
|
||||
|
||||
The ML service loads and validates the artifact at startup. If it is missing or
|
||||
malformed, only these routes return 503; `/healthz`, `/readyz`, `/bootstrap`,
|
||||
`/predict`, `/simulate`, and `/meta` keep working.
|
||||
|
||||
| Method | Path | Behavior |
|
||||
|---|---|---|
|
||||
| GET | `/prezero/summary` | `meta` + `summary` |
|
||||
| GET | `/prezero/priority` | Ranked containers; `priority` (`high\|medium\|low`), `limit` 1–500 (default 50), `offset` >= 0 |
|
||||
| GET | `/prezero/containers/{location_id}` | One container + 10 newest readings; unknown → 404 |
|
||||
|
||||
Invalid `priority`, `limit`, or `offset` → 400 in the existing error envelope.
|
||||
The priority list omits `recent_readings`; the container route returns it as
|
||||
`readings`. `/prezero/priority` reports `total` before pagination and `count`
|
||||
after it.
|
||||
|
||||
```bash
|
||||
curl -s localhost:8000/prezero/summary
|
||||
curl -s 'localhost:8000/prezero/priority?priority=high&limit=3'
|
||||
curl -s localhost:8000/prezero/containers/<known-id>
|
||||
```
|
||||
|
||||
## Verification
|
||||
|
||||
```bash
|
||||
uv run pytest -q tests/test_prezero.py
|
||||
make check
|
||||
```
|
||||
|
||||
The tests assert the dataset counts above (11,701 rows, 449 containers, 1,338
|
||||
series, 1,192 overfull, 88/248/113 tiers), filtering/pagination, known and
|
||||
unknown containers, invalid queries, artifact schema and tier consistency, and
|
||||
the missing-artifact 503 path.
|
||||
|
||||
## Limits
|
||||
|
||||
- No coordinates: cannot compute distance, travel time, or a route.
|
||||
- No capacity, depot, vehicle, or shift data.
|
||||
- `overfull` is a reported state; fill percentages are coarse, user-reported
|
||||
values (0/10/.../100), not continuous sensor volumes.
|
||||
- The reference date is the dataset max date, so staleness is relative to the
|
||||
snapshot, not to when the API is called.
|
||||
- Counts and rates describe this export only; they are not a forecast of future
|
||||
fill levels.
|
||||
Reference in new issue
Block a user