SAVI Andrew Ebaugh Saurav Chatterjee Shopping assistant for the visually impaired

Basic concept of SAVI What does it do?  SAVI allows blind people to locate and identify items via RFID technology. What is it good for?  Blind people can shop by themselves.  Facilitates in identifying items once they get home.  It can substitute for Braille signs and tags.

Currently, blind individuals cannot navigate a grocery store independently. Have to depend on others to help with shopping (Braille stickers on products) Little choice on prices and brand of items being purchased Have to depend on others when checking out at the register Organizing their items once they return home is a problem Why is SAVI needed?

Product Demo We will display three active pieces: Personal Server, ear piece, & scanning glove Simple demo idea: Identify common grocery items with our system without using sight. Plan to set up a display of several tagged items on shelves Generate a grocery list from among the several possible items Locate the correct items by handling the objects and scanning the tags  As our audio board also transmits over FM, we’ll have a small stereo set up to broadcast what the device user will be hearing

Product Demo, cont. Our original scenario was pretty close to this  it was for an actual user in a grocery store  It showed a user receiving audio feedback at the register, which we will not demo  It also showed the user putting away groceries at home, which we will not demo

Implementation – Overall diagram Processing Input Output

Implementation, cont. RFID scanner – mote running TinyOS transmits RFID tag scan to a mote in a CF card on personal server CF card mote – accessible as a serial port on personal server, /dev/tts/3 Serial forwarder service – dumps whatever is received on the serial port to listeners on a network port (4444 TCP) ODBS listener service – Connects to network port, and registers as a listening client, accepting forwarded packets.

Implementation, cont. ODBS – Multi-threaded: port listening thread, database query thread, output thread Product information is stored in SQL database Upon receiving a RFID scan, the service queries the product database Additionally queries for brand, price, sale information Each piece of information returns an associated audio clip; from all of these, a presentation “playlist” is generated

Implementation, cont. Database is provided by sqlite and accessed by sqlite JDBC-connector Contains 7 tables: product, brand, measure, location, sale, alphanum Product table pseudo-SQL : CREATE TABLE product (rfid, upc, brand_id, name, price, price_unit_id, price_quantity, sale_type, sale_quantity, sale_price, quantity, quantity_unit_id, location_id, section_id, clip) Ex: INSERT INTO product VALUES ('E00781BCC1957A75', ' ', 9, 'Oreo Cookies', '3.69', NULL, NULL, 2, NULL, NULL, '1.13', 0, 9, 18, 'product-11');

Implementation, cont. Presentation clip – Once the clip has been generated, ODBS makes a call to Runtime.exec() to execute an mp3 audio player Execution call takes the form of: madplay /mp3/brand-04.mp3 /mp3/product-02.mp3 /mp3/alphanum-39.mp3 MP3 player talks to audio board as /dev/dsp

Implementation, cont. Slappy board – Audio is provided by board that plugs into personal server Board drivers are modular and with dependent modules (soundcore, ac97_codec) provide standard linux audio interface to be accessed by applications In addition to standard stereo output and line input, provides FM transmission on configureable frequency. All this together gives a range of options, from enhanced ODBS access of audio board, to a multitude of output devices

Implementation, cont. What is done?  All the pieces described so far are working together  System works end-to-end, from tag scan to audio What isn’t?  Mainly finishing things up and work on aesthetics  Glove and personal server aren’t currently something one would want to wear  Last minute hardware glitches

Evaluation How well does it work?  Once the kinks are worked out, the system works pretty well. The key metrics: Distance glove needs to be from tag Length of time glove needs to be reading tag Delay between successful scan and start of audio (i.e. processing time) Length and relevance of audio clip

Evaluation, cont. – metrics  Distance glove needs to be from tag Depends on how charged the batteries are, and how sensitive the antenna is For our glove reader with a Mylar antenna in palm and full batteries, the distance is 3-4 inches.  Length of time glove needs to be reading tag We have toyed with different reading frequencies to tune this metric Currently check for a tag 4 times per second. At this speed, the reader can detect a tag when moved by it at an acceptable speed (i.e. as if you were feeling around the object)

Evaluation, cont. – metrics  Delay between successful scan and start of audio This was our main concern, that doing the processing and database queries would take time Actually, this time is minimal From successful scan, audio will start in under a second The longest portion of this is starting up the mp3 player, suggesting a possible point for improvement  Length and relevance of audio clip Probably needs the most work The overall clip is built on subclips. Currently output is: brand, name, price, any sale information It would be best if this was configurable, because playing this information can take some time

Future Work Lots to be done, lots of possibilities Database contains a lot of information not currently used: UPC code, location, quantity There should really be an audio output component always loaded, rather than making an external call The product database could be a central store database accessed over wi-fi or similar Could integrate a shopping list or store navigation aid Would like to have a scanner at the checkout register that relates product name, price, subtotal as rung up The glove we have is definitely a prototype: should have a lightweight, chargeable battery, and several buttons Possible text-to-speech for audio generation, and linking with existing UPC-based product databases