Date on Master's Thesis/Doctoral Dissertation

8-2020

Document Type

Master's Thesis

Degree Name

M.S.

Department

Electrical and Computer Engineering

Degree Program

Electrical Engineering, MS

Committee Chair

Welch, Karla

Committee Co-Chair (if applicable)

Nasraoui, Olfa

Committee Member

Nasraoui, Olfa

Committee Member

Zurada, Jacek

Author's Keywords

computer vision; hand-object interaction; inventory management; hand detection; produce classifier; background removal

Abstract

Tracking the inventory of one’s refrigerator has been a mission for consumers since the advent of the refrigerator. With the improvement of computer vision capabilities, automatic inventory systems are within reach. One inventory area with many potential benefits is the fresh food produce bins. The bins are a unique storage area due to their deep size. A user cannot easily see what is in the bins without opening the drawer. Produce items are also some of the quickest foods in the refrigerator to spoil, despite being temperature and humidity controlled to have the fruits and vegetables last longer. Allowing the consumer to have a list of items in their bins could ultimately lead to a more informed consumer and less food spoilage. A single camera could identify items by making predictions when the bins are open, but the camera would only be able to “see” the top layer of produce. If one could combine the data from the open bins with information from the user as they placed and removed items, it is hypothesized that a comprehensive produce bin inventory could be created. This thesis addresses the challenges presented by getting a full inventory of all items within the produce bins by observing if the hand can provide useful information. The thesis proposes that all items must go in or out of the refrigerator by the main door, and by using a single camera to observe the hand-object interactions, a more complete inventory list can be created. The work conducted for this hand analysis study consists of three main parts. The first was to create a model that could identify hands within the refrigerator. The model needed to be robust enough to detect different hand sizes, colors, orientations, and partially-occluded hands. The accuracy of the model was determined by comparing ground truth detections for 185 new images to the model versus the detections made by the model. The model was 93% accurate. The second was to track the hand and determine if it was moving in or out of the refrigerator. The tracker needed to record the coordinates of the hands to provide useful information on consumer behavior and to determine where items are placed. The accuracy of the tracker was determined by visual inspection. The final part was to analyze the detected hand to determine if it is holding a type of produce or empty, and track if the produce is added or removed from the refrigerator. As an initial proof-of-concept, a two types of produce, an apple and an orange, will be used as a testing ground. The accuracy of the hand analysis (e.g., hand with apple or orange vs. hand empty) was determined by comparing its output to a 301-frame video with ground truth labels. The hand analysis system was 87% accurate classifying an empty hand, 85% accurate on a hand holding an apple, and 74% accurate on a hand holding an orange. The system was 93% accurate at detecting what was added or removed from the refrigerator, and 100% accurate determining where within the refrigerator the item was added or removed.

Download

Share

COinS