The Panlingual Mobile Camera CSE 490f Jonathan Pool, Neb Tadesse, Tim Wong, Luke Woods Tim Wong, Luke Woods

Overview ► Overall problem & solution ► Contextual inquiry description & results ► ESM description & results ► Representative tasks ► Task Scenarios ► Summary ► Q & A

The Overall Problem ► Have you ever been in a situation where you can’t understand some text? ► Language barriers ► Illiteracy ► Vision disability

A Solution ► A camera phone that:  Allows the customer to photograph text  Efficiently receives a fast explanation of the text

Contextual Inquiry Description ► Three subjects in different situations:  (1) subject wearing blurring goggles to hinder vision  (2) subject amidst many non-English signs + labels  (3) subject with low English fluency ► In various contexts: locations + activities

Contextual Inquiry Description ► Goals:  (1) subjects’ purpose  (2) kinds of text subjects wanted to understand  (3) urgency of understanding  (4) alternative sources of understanding  (5) means of coping without understanding  (6) subjects’ text-selection strategies

Contextual Inquiry Results

ESM Description ► Goals:  Determine how well users can take photographs ► Clear or blurry? ► Size vs distance?  Reveal how users might indicate the specific text to be understood  Determine what sort of environments people are encountering throughout the day

ESM Description ► Design:  Prompt approx every hour between 8am and 9:30pm ► Asked to take one or more photographs of text ► Encouraged to take diverse photos  Subjects were also allowed to take any additional photographs at any time  Subjects could record any audio-visual comments

ESM Results ► Not everyone takes good photos

ESM Results ► Environment doesn’t always allow for good photos

ESM Results  Some photos contained both legible and illegible text  Some users pointed towards text ► Cannot rely on the user to indicate area

Representative Tasks ► (Easy) Customer looks at a road sign in a foreign language and tries to infer whether it defines a speed limit. ► (Moderate) The customer wanders around a foreign bookstore looking for the philosophy section. ► (Hard) The customer approaches an apartment building and wants to know whether a person is listed in the building's directory, which is written in a script that the customer doesn't know.

Task Scenarios ► (1) A traveler in Germany rents a car. He comes across this sign and wants to know if he can park here legally. He wants to take a photo and annotate it, describing the info he wants to understand.

Task Scenarios ► (2) A person travels to China, and wants to go to a Chinese restaurant. However, the person does not understand the menu in Chinese. She takes a photo. She wants to circle several random items of her choosing for translation.

Task Scenarios ► (3) A person understands spoken Japanese, however he cannot read it. He wishes to hear an audio clip of this road sign. He takes a photo and requests to receive an audio translation.

Task Scenario ► UI Design ideas:

Summary ► A device that can allow you to translate and/or understand text on the fly ► Supports multiple delivery mediums:  Audio  Text ► Must be fast, efficient, and easy to use.

Q & A ► Questions????