Files
Noel Sabia ad63d1caf3
ci / check (push) Successful in 30s
ci / check (pull_request) Successful in 29s
feat: prezero priority snapshot and internal API
Build artifacts/prezero_priority.json from the anonymized glass fill-level export (11,701 rows, 449 containers, 1,338 location/material series). The contract tier rule yields high 88, medium 248, low 113 with 1,192 overfull readings.

Add /prezero/summary, /prezero/priority, and /prezero/containers/{id} to the internal FastAPI service, with startup validation, filtering/pagination, recent reading history, and 503 isolation when the artifact is missing. Raw CSV stays gitignored; the derived 1.1 MB artifact is the source of truth. This is a priority proof, not a route: the dataset has no coordinates.
2026-11-14 18:06:02 +01:00

6.2 KiB
Raw Permalink Blame History

PreZero container priority

A transparent collection-priority snapshot built from the anonymized glass fill-level export. The committed artifact is artifacts/prezero_priority.json; the raw CSV is not committed.

This is a priority proof, not a route. The dataset has no coordinates, no container capacity, and no depot data. overfull is a reported state, not a measured volume.

Provenance

Field Value
Input glass_fill_levels_anonymized_f.csv (UTF-8 with BOM)
Rows 11,701
Distinct Location_id 449
Location_id + material_type series 1,338
Date range 2025-10-17 → 2026-09-30
overfull readings 1,192
Material readings white 4,432 · green 4,169 · brown 3,100
Material overfull white 731 · green 357 · brown 104

The export is read with utf-8-sig; header names and cells are stripped. Every row must have Location_id, material_type, a YYYY-MM-DD date, and a fill_level_perc of either overfull or an integer 0–100.

The raw file stays git-ignored under data/raw/. Put it there or set PULSECAST_PREZERO_CSV; the generated artifact is committed so the API and Docker image do not need the CSV.

Rule

Each Location_id is one container location. For all rows at that location:

  • readings, overfull_count, overfull_rate = overfull_count / readings
  • numeric_count = rows with a numeric 0–100 fill (excludes overfull)
  • p95_fill = 95th percentile of numeric readings, null if none
  • max_fill, mean_fill = numeric readings only
  • last_fill, last_is_overfull, last_date = most recent row
  • days_since_last = reference_date - last_date; reference is the dataset max date (2026-09-30), not today
  • trend = least-squares slope over the last up to 3 numeric readings (percentage points per reading); null with fewer than 2
  • priority_score = round(overfull_count + 2 * overfull_rate, 4) — a display and sorting convenience, not the tier
  • materials = per material: readings, overfull count, last fill/state/date

Tier:

  • high: overfull_count >= 5 OR (overfull_count >= 2 AND overfull_rate >= 0.25)
  • medium: overfull_count >= 1 and not high
  • low: overfull_count == 0

Result on the attached CSV: high 88, medium 248, low 113.

Order: tier (high first), then overfull count desc, overfull rate desc, days_since_last asc, location_id asc.

Artifact shape

{
  "meta": {
    "source": "glass_fill_levels_anonymized_f.csv",
    "generated_at": "2026-11-14T17:00:00Z",
    "rows": 11701,
    "containers": 449,
    "material_series": 1338,
    "date_from": "2025-10-17",
    "date_to": "2026-09-30",
    "reference_date": "2026-09-30",
    "overfull_readings": 1192,
    "priority_rule": "high: ...; medium: ...; low: 0",
    "limitations": ["No coordinates ...", "..."]
  },
  "summary": {
    "priority_counts": {"high": 88, "medium": 248, "low": 113},
    "by_material": {
      "white": {"readings": 4432, "overfull": 731},
      "green": {"readings": 4169, "overfull": 357},
      "brown": {"readings": 3100, "overfull": 104}
    }
  },
  "containers": [
    {
      "location_id": "...",
      "priority": "high",
      "priority_score": 21.525,
      "readings": 80,
      "overfull_count": 21,
      "overfull_rate": 0.2625,
      "numeric_count": 59,
      "p95_fill": 100.0,
      "max_fill": 100,
      "mean_fill": 64.24,
      "last_fill": null,
      "last_is_overfull": true,
      "last_date": "2026-09-29",
      "days_since_last": 1,
      "trend": 30.0,
      "materials": {"white": {"readings": 29, "...": "..."}},
      "recent_readings": [
        {"date": "2026-09-29", "material_type": "white",
         "fill_level_perc": "overfull", "is_overfull": true}
      ]
    }
  ]
}

The artifact holds at most 10 recent readings per container, not the full 11,701-row table. last_fill is null when the newest reading is overfull. p95_fill, max_fill, mean_fill, and trend are null when the location has no numeric readings or too few for a slope.

Generation

# Raw CSV is git-ignored; copy it or set PULSECAST_PREZERO_CSV.
cp glass_fill_levels_anonymized_f.csv data/raw/
make prezero

# Equivalent explicit command:
uv run python scripts/build_prezero.py \
  --csv data/raw/glass_fill_levels_anonymized_f.csv \
  --out artifacts/prezero_priority.json

The script prints rows, containers, material series, overfull readings, date range, and the three tier counts. It fails loudly on missing columns, blank cells, bad dates, or invalid fill values instead of guessing.

Internal API

The ML service loads and validates the artifact at startup. If it is missing or malformed, only these routes return 503; /healthz, /readyz, /bootstrap, /predict, /simulate, and /meta keep working.

Method Path Behavior
GET /prezero/summary meta + summary
GET /prezero/priority Ranked containers; priority (high|medium|low), limit 1–500 (default 50), offset >= 0
GET /prezero/containers/{location_id} One container + 10 newest readings; unknown → 404

Invalid priority, limit, or offset → 400 in the existing error envelope. The priority list omits recent_readings; the container route returns it as readings. /prezero/priority reports total before pagination and count after it.

curl -s localhost:8000/prezero/summary
curl -s 'localhost:8000/prezero/priority?priority=high&limit=3'
curl -s localhost:8000/prezero/containers/<known-id>

Verification

uv run pytest -q tests/test_prezero.py
make check

The tests assert the dataset counts above (11,701 rows, 449 containers, 1,338 series, 1,192 overfull, 88/248/113 tiers), filtering/pagination, known and unknown containers, invalid queries, artifact schema and tier consistency, and the missing-artifact 503 path.

Limits

  • No coordinates: cannot compute distance, travel time, or a route.
  • No capacity, depot, vehicle, or shift data.
  • overfull is a reported state; fill percentages are coarse, user-reported values (0/10/.../100), not continuous sensor volumes.
  • The reference date is the dataset max date, so staleness is relative to the snapshot, not to when the API is called.
  • Counts and rates describe this export only; they are not a forecast of future fill levels.