Research Log

Project scoped done

The JDEI proposal was read and its scope locked: a fixed camera watches Cal Poly Food Pantry shelves, an on-device model estimates occupancy and flags low stock, all inference stays on the device. Public shelf-detection datasets were surveyed and a two-stage data strategy was chosen (pretrain on dense retail imagery, fine-tune on empty-shelf sets), and the detector architecture was chosen (YOLO11n: small enough for the Jetson, accurate enough for the task). Three camera options were priced for the PI.

The deployment site, measured from video done

No site visit had happened yet, so a promotional video the pantry had already published was analyzed frame by frame: 250 frames, cross-referenced against known object sizes (brick coursing, can dimensions, a commercial reach-in fridge of known make) to reconstruct a dimensioned floor plan and shelf elevation. This is the Building 27 Pantry Survey: room geometry, camera coverage math, power locations, and an inventory of what the camera would actually need to recognize.

Baseline model trained on Colab done

Six public empty-shelf datasets were pooled (4,860 training images, one class: gap) and YOLO11n was fine-tuned on a free Colab GPU. The first run was interrupted twice by Colab reclaiming the VM mid-training. The fix: point training output at a mounted Google Drive folder so every epoch's checkpoint survives a disconnect, and make the script auto-resume from the last checkpoint instead of restarting. After that fix, disconnects cost at most one epoch instead of the whole run.

Result: mAP50 0.898, mAP50-95 0.639 on the retail validation split.

EXP-001: the retail model meets the real pantry done

The trained model was run on real pantry photos it had never seen. Its single most confident detection anywhere (0.70) was the painted fruit decoration on the wall above the shelves. It also flagged empty counter space and, in one frame, a shopper's hair, while missing shelves that were genuinely empty. The pattern: the model had learned "dark flat region," not "empty region inside a shelf." Full write-up and images: Phase Zero Report, EXP-001.

EXP-002: 15 labeled frames from the site done

Fifteen frames from the pantry's own video were hand-labeled (7 gap boxes, 8 negatives including the decor wall) and used to fine-tune the model for two minutes on a laptop CPU. Every false positive observed in EXP-001 disappeared, and the model began detecting gap types it had missed before, including a freezer shelf it had never seen labeled. This became the deployment model, v2. Before and after images: Phase Zero Report, EXP-002.

On-device monitoring loop built and tested done

The full runtime pipeline was written: a guard that refuses to report a number when the camera view looks wrong (lights off, camera bumped or blocked) rather than guessing, per-shelf occupancy from the gap detections, a filter that requires a shelf to read low for a sustained period before it is reported (a single odd frame should not trigger an alert), and a plain-text summary for staff. It was tested against video frames standing in for a camera: fed frames from different viewpoints, it correctly refused 29 of 30 as "camera moved"; fed two frames of the same fully-stocked view, it read all shelves correctly and produced a clean summary. Diagram and test output: Building 27 Survey, Sheet 7.

Progress shared with the PI done

This site was built and a progress email sent summarizing the baseline model, both experiments, and the site survey, along with the camera purchase options.

The PI agreed to order the camera and asked what else to include in that order; a small hardware list was sent back (NVMe SSD, microSD card, an active USB extension cable sized from the room measurements, and a wall mount). The PI is separately locating the pantry's admin contact for installation permissions. A plain-language one-page description of the system (what the camera sees, what is stored, who is in charge) was drafted for that conversation.

EXP-004: does dense-retail pretraining help done

With the Kaggle recipe working, the originally planned comparison ran: pretrain on SKU-110K (a dense retail-shelf dataset, 11,743 images) before the gap fine-tune, versus fine-tuning directly. This was pre-registered before the run: the decision rule for adopting the pretrained model was fixed in writing in advance, so the result could not be interpreted to fit a preference.

Pretraining won on the benchmark, and also produced fewer false positives on real pantry photos with zero pantry-specific data, meaning the improvement was not just curve-fitting to the validation set. Per the pre-registered rule, it replaced the previous approach. Re-applying the same 15-frame pantry fine-tune on top of this stronger base produced v3, the current deployment model, which detects the hardest real-world test case (a bare shelf) at meaningfully higher confidence than v2 did (0.43 versus 0.29).

Casting a wider net for training data done

A search was run across public dataset platforms for additional empty-shelf imagery and, separately, for any real food-pantry photos (the closest match to the actual deployment domain). Four sizable new detection datasets were found and downloaded. One risk surfaced immediately: several of the new sources and the sources already pooled appeared to be different exported versions of the same underlying photographs by the same authors, which would silently duplicate images between the training set and the validation set and make later results look artificially good.

Every image was fingerprinted (a perceptual hash, robust to resizing and re-compression) and checked against the frozen validation and test sets before being added. 809 duplicates were caught and dropped, including 56 that would have leaked into validation.

The pantry-domain search came back thin: the closest match was a small (~220 image) academic pantry photo set, labeled for identifying items rather than empty shelves, of limited direct use but a useful data point that pantry-specific imagery is genuinely scarce in public datasets.

EXP-006: does the model's percentage match reality done

Every result so far used mAP, a score for bounding boxes. But the system's real output is a percentage on a staff summary ("shelf 3 is 30% full"), so a separate question matters: how far off is that percentage? A held-out dataset reserved since the first day of planning answers it: 180 shelf images from three sources where every gap pixel was hand-traced by researchers, giving exact ground-truth gap fractions to compare against.

An error of ~12 occupancy points is compatible with the digest's design, which reports coarse levels (EMPTY / LOW / OK, each spanning 25 to 40 points) rather than precise percentages, with a persistence filter absorbing flicker near level boundaries.

Near-miss worth recording: the first run of this evaluation silently paired only 59 of the 180 images. One source names its files differently (image 100jpg.jpg for mask 100.tiff) and the pairing code missed the pattern, dropping two-thirds of the evaluation without any error message. On the truncated set the model ranking was inverted: v3 looked worst. Only checking the image count against the expected total exposed it. The conclusion an evaluation produces can depend entirely on data that silently went missing; every eval table now gets its n verified first.

EXP-005 concluded: quality beats quantity done

The retrain on the cleaned 11,624-image pool finished at 0.902 mAP50: cleaning recovered 24 of the 26 points the poisoned run lost, but the expanded pool still came in below the smaller curated pool's 0.920. The full battery agreed. Occupancy error: 12.3 points versus 11.8. And on the most important real-world check, the candidate model missed the bare case shelves that the current model detects. The current model (v3) stays.

The finding: doubling the training data produced a strictly worse model, even with the label poisoning removed. The added sources' differing annotation styles and heavy augmentation diluted the target distribution rather than reinforcing it. This closes the public-data avenue with a measured answer: the remaining way to improve the model is real imagery from the pantry itself, which is exactly what the first weeks of the semester will collect. A negative result, but a decision-shaping one: effort now goes where the data is scarce and valuable instead of where it is plentiful and cheap.

The data question, asked harder and closed done

A challenge worth recording: was the search for training data actually thorough? A second sweep answered it properly. Thirteen previously unprobed dataset leads were checked by API; four were downloaded and rendered for inspection, and three were rejected on sight (one "empty space" dataset turned out to be photographs of bolts on a parts tray). Every non-Roboflow platform - Kaggle, HuggingFace, Zenodo, Dataverse, Figshare, IEEE DataPort, GitHub, synthetic-data pipelines - was swept and documented empty for this task. Two ideal academic datasets exist but are locked behind author permission. The search is now exhausted in the strong sense: every thread named, every dead end recorded.

The sweep also surfaced one real hypothesis: maybe EXP-005 failed because the added data was bulk and sloppy, and carefully verified additions would still help. EXP-007 tested exactly that, with the rule fixed in advance: the winning pool plus only inspection-passed additions, including researcher-grade labels derived from hand-traced masks. Result: 0.892, below both the original pool (0.920) and the cleaned bulk pool (0.902). Three experiments now form one line: every public-data addition, careful or not, made the model worse on the fixed benchmark.

One honest caveat, stated in the record: the benchmark's validation images come from the same sources as the original pool, which structurally favors it. What these experiments prove is that public data is spent as a lever. What actually improves the deployed system will be decided by imagery from the pantry itself, which is precisely what the first weeks of the semester collect. The current model (v3) enters the semester as the final Phase 0 model.

EXP-008: manufacturing pantry data done

With public data closed, one lever remained reachable before any camera exists: synthesizing pantry-domain training images by compositing verified gap crops from the labeled frames into other pantry frames. The first attempt failed its own inspection gate twice before training even started (one donor crop contained a volunteer's arm; another was freezer interior that looked alien pasted on wood shelving), which is the inspection process working as designed.

Two training rounds against the pre-registered bar: the synthetic-augmented model doubled its confidence on the hardest real gap (0.43 to 0.86) with near-clean false-positive behavior, but its occupancy calibration regressed slightly both rounds (11.9 vs 10.9 points), so by the rule it was not adopted. The mechanistic read: pasted crops teach what pantry gaps look like but their arbitrary sizes distort how much area the model attributes to them.

The takeaway is a prediction: if artificial pantry data moves recognition this much, real on-site imagery, with correct sizes in correct contexts, should capture the gain without the calibration cost. That is now the strongest quantitative argument for the data collection planned in the first weeks of the semester. The deployment model remains v3; the week-2 hardware bring-up is now scripted end to end (flash, build engine, calibrate, run as an auto-restarting service).

A third data sweep, and a citation retracted done

An observation about the deployment site opened one last search angle: the pantry buys its core stock in bulk, a few consistent brands restocked repeatedly. A small fixed catalog makes per-item recognition tractable in a way open-ended retail recognition is not. So a third sweep asked a new question of the public data: not "are there empty-shelf boxes" but "are there brand-item datasets worth training on." Thirty-five queries against the dataset platform's search API returned 493 distinct projects, each screened by class list and sample imagery.

The answer is again no, and again useful. Every brand-item pantry dataset found is tabletop close-up photography, a hand holding a soup can at arm's length, which does not transfer to a fixed camera viewing a shelf wall. One 893-image candidate whose class names suggested fill-level annotation was downloaded in full and turned out to be supermarket egg-carton displays, unrelated to its own title. But two small student projects demonstrated the recipe that matters: working closed-set detection of five to seven fixed products from a few hundred self-collected images. That is precisely the shape of the on-site option this fall, applied to the pantry's own bulk catalog, and it is now recorded as the validated path should item-level detail prove wanted.

The sweep also forced a correction. An earlier literature pass had recorded a pantry-domain labeling dataset attributed to a university lab. Verification across the dataset platform's search, its API, and the open web found no trace of it, so the citation was retracted from the project record; the datasets that anchor the novelty claim were re-verified live the same day and stand. A record that catches its own unverifiable citation is doing its job: every claim in this log is meant to survive exactly this kind of audit.

Advisor sync: scope locked, hardware in motion done

First live meeting with the advisor since the Phase Zero report. The scope question that shaped five open decisions resolved simply: the research goal is a proof of concept and a minimal working prototype, with the written report as the priority deliverable. That confirms the dry-goods-wall scoping and keeps the semester plan intact.

Logistics: the camera and remaining hardware will be ordered shortly; the advisor has the Jetson in hand midweek, with physical handoff when the semester starts. The storage drive he already holds (a 500 GB M.2 NVMe module) was checked against the developer kit's slot spec during the call and is compatible. He also has an Arduino Nicla Vision, a microcontroller-class camera board. It cannot run this project's detector, but it is recorded as a future option for an affordability comparison: what fraction of the monitoring task survives on a board costing half as much as the Jetson? That question fits the project's cost-minimization framing and is deliberately parked to protect scope.

Two process decisions: meetings will run as-needed rather than on a fixed cadence, and no pantry imagery will be collected until the site survey happens with the pantry's own staff at the table, expected early in the semester. The image-retention policy and the IRB determination inquiry both wait for that conversation. The pantry contact who enables it is expected in the first week of September.

Dress rehearsal: four defects the benchmarks could not see done

Before the hardware arrives, the full monitoring loop was run against a live camera in daylight rather than the simulated frames used until now. The model was never the point of the exercise. The point was everything around it: opening a camera, waiting for it to expose correctly, pacing readings, and writing the message staff actually read. Four defects surfaced, none of which existed in simulation, and all four were fixed.

The most serious one is worth stating plainly. The out-of-distribution guard did its job perfectly: pointed at the wrong scene, it refused every single frame and recorded why. But the daily staff digest rendered that state as "All 0 shelves OK. Nothing needs attention." A monitor that had measured absolutely nothing was reporting good news. In deployment, a bumped camera or a drifted calibration would leave pantry staff reassured while the system sat blind. The digest now says "NO READINGS YET" and explains what to check. Blindness has to be louder than an empty shelf.

The other three were quieter but would each have stalled the hardware bring-up. The interval setting that is supposed to take one reading per minute was never applied, so the loop ran the detector on every incoming frame, roughly a thousandfold more work than intended and a continuous load on a device meant to idle between readings. The camera was never released after use, which leaves the device locked and makes the next start hang forever, precisely the calibrate-then-start sequence the deployment script uses. And calibration captured its reference from the very first frame, before automatic exposure had settled, recording a brightness of 35 for a scene that stabilizes at 143 a second later; that single early frame then caused every subsequent reading to be rejected as out of range.

The lesson generalizes past this project. Accuracy metrics measure the model; they say nothing about the seams between a camera and a model, and those seams are where an unattended system actually fails. For a deployment whose entire premise is running quietly for months without supervision, a monitor that reports its own blindness as normal is a more dangerous defect than any accuracy shortfall, and no benchmark would ever have caught it.

Writing the report, and auditing its own numbers done

With the hardware a week away and site access gated on a partner introduction, the highest value work left was the deliverable itself. Every section of the final report that does not depend on fall data is now written as prose rather than outline: motivation and the measurement caveats around campus food insecurity statistics, related work and the corrected novelty position, the justice-centered design commitments, the system design, the site characterization method, the full arc of data experiments, and the evaluation protocol with its results. What remains genuinely requires the deployment to exist.

Writing it surfaced something worth recording. A systematic sweep of every underlying page found the same quantities stated slightly differently in different places, because pages were written weeks apart as results arrived. The occupancy error figure appeared without the confidence threshold it depends on, and that threshold had itself moved. The pantry adaptation set was described once by its labelled size and once by its training split. Dataset growth was described with multipliers that referenced different baselines. None of these were errors of fact, and every one of them was a chance to publish two numbers that contradict each other.

They are now fixed to one convention each in a single reference page, which the report, the poster, and this site all defer to. Two stale lines in the experiment records were corrected at the source. The exercise also forced a more honest statement of one claim: the promise that raw images are never stored is enforced in code for the deployed monitor, but it cannot cover the training-image collection that necessarily retains labelled frames. The report now says so explicitly rather than letting a clean sentence do work the architecture does not do.