Carolina Galleguillos and Serge Belongie

1
GroZi a Grocery Shopping Assistant for the Blind
Carolina Galleguillos and Serge
Belongie Department of Computer Science and
Engineering, UCSD cgallegu,sjb_at_cs.ucsd.edu
Abstract
Object Recognition
Use of the System

• Two types of recognition (mn, mltltn)
• Detection (of objects).
• Verification (objects detected are in that
  list).
• Algorithms
• Color histogram matching.
• SIFT features matching.
• Haar like features in Adaboost framework.
• Shared features.

Grocery shopping is a common activity that people
all over the world perform on a regular basis.
Unfortunately, grocery stores and supermarkets
are still largely inaccessible to people with
visual impairments, as they are generally viewed
as "high cost" customers. We propose to develop
a computer vision based grocery shopping
assistant based on a handheld device with haptic
feedback that can detect different products
inside of a store, thereby increasing the
autonomy of blind (or low vision) people to
perform grocery shopping. Our solution makes use
of new computer vision techniques for the task of
visual recognition of specific products inside of
a store as specified in advance on a shopping
list. These techniques can avail of complementary
resources such as RFID, barcode scanning, and
sighted guides. We also present a challenging
new dataset of images consisting of different
categories of grocery products that can be use
for object recognition studies.
Create a Shopping List

• Online Website
• Website stores data and images of different
  products.
• Feedback from users.
• Provides walking path.
• Prepare shopping list
• Download information into Mozi Box.

CbCr Histograms
Get to the Grocery Store
SIFT keypoints

• Separate project.
• Mozi Box with GPS.
• Visual waypoints.
• Traffic/Street sign reading.
• Use in addition to cane and asking sighted
  bystanders.

Haar features
CbCr chrominance plane. L2 distance between
histograms is used for matching.
Navigate the Store
Motivations
Integral histogram applied for better performance
Porikli, 2005. Processing time 25 frames per
second.

• Finding aisle
• (OCR, RFID, ask).
• Avoiding obstacles (cane).
• Finding products (sweep of aisle, spot product,
  barcode check).
• Checking out (coupon and cash).

• Increase independence of people with low vision
  (specially blind) to perform grocery shopping in
  a supermarket or store.
• There are 1.3 million legally blind people in
  the U.S.
• Help to plan shopping list, walking path to the
  store and grocery shopping.
• Advance research on object recognition for
  mobile robotics with constrained computing
  resources.

Text detection is also part of the GroZi project.
Obtaining Data
Adaboost and Haar features are used to detect
text on images.

• Training data was obtained from two major
  sources
• Sunshine Store _at_ UCSD (video capture)
• - 30 minutes of capture
• - Divided into 29 .avi files
• - Boxes containing product
• manually cropped
• Web (online images)
• - Froogle, Shopwiki, Amazon.
• - Groceries, Yahoo images
• - General specialized (UPC code)
• queries.
• Test data will be obtained from
• Collected videos/images from MoZi box (in situ).

MoZi Box
Future Directions
General purpose low-cost mobile system geared for
computer vision applications.

• Obtain a final version of the data set that can
  be viewed online, which is continuously updated,
  with more products and user interaction.
• Test detection and recognition of products
  using SIFT on top of Haar like features.
• Apply color Haar like features (opponent
  channels) for a better detection.
• Bar code scanning to perform active learning in
  order to learn relations between barcodes and
  product images.

• Finite memory Compact Flash (CF) cards ranging
  from 256 MB to 4 GB.
• Processor speed in the neighborhood of
  60-400MHz.

• Frame rate enough snapshots to cover the shelf
  with some overlap (as in panoramic stitching) .
• Color Calibration Macbeth color chart to
  calibrate the color space.

Dataset
Acknowledgments

• Special thanks to Stephan Steinbach from Calit2,
  Michele Merler and Tom Duerig from UCSD computer
  science department.

The GroZi data set corresponds to 128 different
grocery products, that have been obtained from
online images and collected videos.
Web image
captured image