Justice Centered Edge AI · Cal Poly Food Pantry · Amogh Somisetty, Aug 2026

Phase Zero Report

Pre-semester progress on the pantry shelf-monitoring prototype: system design, site characterization, baseline training, and initial domain-adaptation experiments. Open decisions are listed at the end.

Summary. All work below used public data, a laptop, free Colab GPU time, and the pantry's public video:

Working baseline model: a gap detector at 0.90 mAP50 on retail validation data, trained on 4,860 pooled public shelf images.
Two experiments: EXP-001 characterizes how the retail-trained model fails on pantry imagery; EXP-002 shows that fine-tuning on 15 labeled frames from the site removes the observed failures. Before/after figures below.
Site characterized from video: dimensioned floor plan and shelf elevation, camera coverage geometry, power locations. The week 1 visit is verification.
Lit review drafted (food insecurity, design justice, pantry operations), the paper's novelty claim verified at evidence grade, and the Cal Poly IRB path mapped with a ready-to-send inquiry email.
Five open decisions are listed at the end.

1 · What the system does

A fixed camera watches the pantry's dry-goods shelf wall. An NVIDIA Jetson Orin Nano runs a small detection model (YOLO11n, 2.6M parameters) that finds empty shelf regions, not products: an emptying shelf exposes visually distinctive bare board, which turns "how full is this shelf" into simple arithmetic per shelf region instead of a hard product-recognition problem. Occupancy per cubby feeds a debounced state ("low for N hours"), which feeds a morning digest for staff. All inference is on-device: frames are analyzed and immediately discarded, only stock numbers are ever stored. No microphone; no person detection.

flowchart LR
  A[camera frame] --> B[on-device gap detection\nJetson, ~10 ms class]
  B --> C[frame destroyed]
  B --> D[occupancy per cubby ROI]
  D --> E[persistence filter\nlow for N hours?]
  E --> F[morning digest to staff]
  style C stroke-dasharray: 5 5

2 · Site characterization from video

From the pantry's own promotional video (250 frames analyzed, cross-checked against known-size references: brick coursing, can diameters, the SABA reach-in fridge), we reconstructed a dimensioned floor plan and elevation of the room. Findings with design consequences:

Finding	Design consequence
Galley room ~8-9 x 18-20 ft; cubby openings ~22-24 x 13-14 in, 2 rows x 8-10 columns	Per-cubby ROI grid built; ~300 px per cubby at 1080p, ample resolution
Max camera standoff ~7-8 ft; a 78 degree 1080p camera covers ~11-13 ft of a 16-20 ft wall	Camera decision: corner mount, 4K, two cameras, or a 6-column prototype scope
Staff-only backstock cabinets below the display shelves	"Empty on camera" does not mean "out of stock" - alerts must be persistence-based restock prompts, not scarcity claims
Restocking is continuous through the week, mid-open-hours, with shoppers present	Occupancy dips are normal churn; the camera must tolerate people and hands (frames discarded regardless)
Stock is bulk-purchased from donated funds (per the coordinator on camera), not irregular donations	Core stock is more retail-like than the pantry literature assumes - good for model transfer; the irregular stream is the Thu/Fri food rescue
Visits grew 131 (2014-15) to 9,367 (2021-22) to 10,100 in Fall 2022 alone; ~$250k annual operating cost	Motivation numbers for the report, from the pantry's own material

The full survey (floor plan, elevation, camera coverage geometry, annotated frames, and the trainable-feature analysis) is a separate reference: see links at the end.

3 · Data and model

Training data: six public empty-shelf datasets pooled into one 4,860-image set with a single class, gap (sources and licenses documented in the repo). SKU-110K (11,743 dense retail images) is staged as an optional pretraining comparison for the model-comparison task.

Baseline: YOLO11n fine-tuned from COCO weights, 40 epochs at 960 px on a free Colab T4: mAP50 0.898, mAP50-95 0.639, precision 0.87, recall 0.82 on the retail validation split. Inference is ~80 ms per frame on a laptop CPU alone, well within budget at the intended duty cycle of one frame every few seconds.

Novelty check (evidence-graded): a 2025 systematic review (301 papers screened) found zero camera-based shelf monitoring in pantries or food banks, and no occupancy-annotated pantry imagery exists publicly. Small pantry item-classification datasets do exist, so the defensible claim is scoped to shelf-occupancy deployment and occupancy-labeled data, with citations compiled in the project notes.

4 · The two experiments

EXP-001: baseline transfer test

We ran the 0.90-mAP50 baseline on real pantry imagery it had never seen. Its single most confident detection anywhere (0.70) was wall decoration: the painted fruit cutouts above the shelves. It also fired on empty counter space and a shopper's dark hair, and missed genuinely bare shelves. The failure has one shape: the model learned "dark flat region = gap" without learning "must be inside a shelf." Notably, every false positive fell outside the per-cubby ROI grid, so per-cubby ROI cropping alone removes the observed false positives.

EXP-002: site-specific fine-tuning

We hand-labeled 15 frames from the pantry video (7 gap boxes, 8 negatives including the decor wall and frames with people) and fine-tuned for 2 minutes on a laptop, backbone frozen. Same test, held-out images:

Before and after: retail model fires on wall decoration at 0.70 confidence; fine-tuned model produces zero detections on the same frame — Figure 1. Decor-heavy wall, held-out negative frame. Left: false positives on wall decoration (0.70) and counter (0.35). Right: no detections.

Before and after: retail model false-alarms on counter and misses bare case shelves; fine-tuned model finds the bare shelves with no false alarms — Figure 2. Left wall; frame excluded from training. Left: counter false positive; the bare case shelves are missed. Right: the bare shelves detected at 0.29, no false positives.

Before and after: retail model fires three times on video player interface elements over a fully stocked wall; fine-tuned model produces zero detections — Figure 3. Fully stocked wall. Left: three false positives on video-player interface elements, the same dark-region error. Right: none. Freezer gaps visible through glass were detected both before (0.45, 0.59) and after (0.41, 0.36) fine-tuning; no regression.

Interpretation: site-specific adaptation data, even at n=15, removed all observed false positives and recovered a missed gap class without regression. This is the quantitative case for on-site data collection (proposal task 3).

Limitations: all frames share one video's lighting conditions; confidence on newly detected gaps is modest; EXP-003 repeats this protocol with a larger set from the installed camera in Phase 2.

5 · Groundwork: literature, IRB, safeguards

Area	State
Lit review, 3 legs	Drafted with verified citations: campus food insecurity (prevalence + measurement caveats; the 39.5% figure traced to ACHA-NCHA 2024), Design Justice (with a concrete checklist the project must satisfy: MOU, co-design before hardware, no person detection, data owned by the pantry, exit plan), pantry operations + tech landscape (62% of pantries manage inventory by visual assessment; no incumbent does passive sensing)
IRB path	Cal Poly-specific research done: likely exempt category 2 for staff interviews, camera likely exempt or not human-subjects IF no raw frames are ever retained; determination inquiry email to hs-irb@calpoly.edu drafted and ready; CITI training and PI-of-record sit with the PI
Runtime safeguards	Designed in: out-of-distribution guard (abnormal frames produce "no reading", never a fabricated number), stale-state fallback on the digest, plausibility bounds
Privacy architecture	On-device inference, frames discarded immediately, no audio hardware, only shelf-level numbers stored; the room's own video and frames never leave the local machine

6 · Open decisions

1. Camera purchase. Recommendation: order a Logitech C920s (~$65) now as the guaranteed-working baseline; decide between a 4K camera, corner mount, or 6-column prototype scope after the site visit (the coverage math is in the survey, Sheet 3). Verify current Amazon prices before ordering.

2. Training-image retention. If we ever retain labeled training images that contain people, the IRB analysis changes materially. Recommendation: retain no frames with people; collect training data during closed hours or with person-blur. This must be decided before the IRB email goes out.

3. IRB inquiry. The determination email is drafted; it should go from you (or from Amogh with you copied) to hs-irb@calpoly.edu early, since minimal review is 1-2 weeks and expedited is 2-4.

4. Scope: dry goods only for v1. Recommendation: monitor the cubby wall only; the glass-door fridge and freezer are a different vision problem (reflections, LED lighting) and a different privacy conversation. Please confirm before the pantry meeting.

5. Pantry contact. The Basic Needs Coordinator was Kari Howell circa 2023; 2026 reporting names Taffy Gonzalez. Whoever is current, the first meeting should include the MOU conversation (co-design authority, unplug rights, data ownership) from the design-justice checklist.

Deep-dive links

Site survey (floor plan, elevation, camera geometry, annotated frames, trainable features): Building 27 Pantry Survey
Code, configs, labels, and experiment scripts: github.com/Asomisetty27/pantry-vision (private; access on request)