Text-Free User Interfaces Presenter: Anuj Tewari Course: CS260 Instructor: John Canny Based on work by Indrani Medhi

Outline: Text-Free User Interfaces Introduction Initial ethnography and iterative rapid prototyping Initial evaluation Optimal audio-visual representation Methodology Discussion

Outline: Text-Free User Interfaces Introduction Initial ethnography and iterative rapid prototyping Initial evaluation Optimal audio-visual representation Methodology Discussion

Illiteracy 1-2 billion illiterate population in the world. 98% live in developing countries. India’s rate of literacy estimated at ~60%. Many labeled “literate” are semi-literate with low functional literacy. Photo: Indrani Medhi Two residents of a slum in Jayanagar, Bangalore

Target Users Women from Bangalore slums Informal sector domestic workers Income range: INR (US$20-60) per month Illiterate or semi-literate Most have never seen a PC (those who have seen have never touched) Photo: Indrani Medhi initial explorations before text-free UI work

NGO Partner Stree Jagruti Samiti Head: Geeta Menon Small NGO Works with several Bangalore slums Focus on children’s rights, women’s empowerment, fair wages for domestic laborers Geeta Menon leading a Saturday Stree Jagruti Samiti meeting Photo: Indrani Medhi

How can an application UI be converted into one that is usable by non-literate users? The Problem

Outline: Text-Free User Interfaces Introduction Initial ethnography and iterative rapid prototyping Initial evaluation Optimal audio-visual representation Methodology Discussion

Initial Ethnography TVs, radios, CD players common; some mobile phones; no PCs Established slums – some with 30-year history in city; concrete housing Subjects most concerned about… Jobs Healthcare Education Nimmi interviewing a subject Photo: Indrani Medhi

Text-Free UI? Design principles: Pen or touch interface Liberal use of imagery ○ No text ○ Semi-abstracted cartoons Voice annotation Aggressive use of mouse- over functionality Consistent help icon Naukri.com for domestic labourers?

Nouns vs. Verbs Goal is to achieve exact association between image and word. Without action cues drawings tend to be interpreted as nouns. With action cues, drawings tend to be interpreted as verbs. Phenomenon well-understood by cartoonists and diagram designers. “Kitchen sink” or “washing dishes”? Text-Free UI Details “Pot” or “cooking”?

Cultural Differences Goal is to achieve as global a representation as possible. Abstracted representations are very dependent on culture. Minor misrepresentations can result in major confusion. An urban family of three? Text-Free UI Details

Religious Differences Goal is still to achieve as global a representation as possible. Religious differences can be the strongest of cultural differences. There is no “neutral” culture or religion. Increasingly general representations for time indication Text-Free UI Details From when to when?

Quirks of Non-Literacy Various degrees of literacy Total illiteracy Numbers, but not alphabets Alphabets, but not words Words, but not sentences Semi-literate, but not fluent Many who are non-literate can read numbers Accustomed to currency Able to do basic arithmetic Not necessarily 100% accurate Illiteracy doesn’t necessarily mean inability to read numbers. Text-Free UI Details

Text-Free UI? Original designRevised design

Text-Free UI Design principles: –Pen or touch interface –Liberal use of imagery No text (but numbers OK) Semi-abstracted cartoons –Aggressive use of mouse- over functionality –Voice feedback –Consistent help icon Screenshot of text-free job search

Other Domains Principles transfer to other applications: Text-free healthcare information Text-free maps

Outline: Text-Free User Interfaces Introduction Initial ethnography and iterative rapid prototyping Initial evaluation Optimal audio-visual representation Methodology Discussion

Questions 1. Can non-literate users use traditional text- based user interfaces at all? 2. Do the proposed design principles for text- free user interfaces allow non-literate users to use computers, and to what extent? 3. Which of the design principles make the most difference for a text-free UI? Initial Evaluation

Experimental Set-Up Three versions: Text-based Text-free without help icon Text-free with help icon Subjects: Illiterate or semi-literate ○ (no literate subjects) No PC experience Subject grouping: Five-women group x 2 Individual x 4 Total time up to one hour per session Task: For a friend who is unemployed, find the best- paying job in her neighborhood. Measured: Task completion (yes/no) Time required (second) Number of prompts required (n) All comments recorded; some trials video-recorded. Initial Evaluation

Quantitative Results Text-based UIs were completely unusable. Obvious, but first formal such test in literature With text-free UI, 30% were able to complete task. (However, not enough participants for statistical significance.) Tabulation of initial results Initial Evaluation

Qualitative Results Strong preference for text-free principles, in particular… Voice annotation Help icon Collaborative use more successful Subjects eager to engage, once comfortable with set up Various barriers to technology: fear, lack of awareness, lack of cognitive model Initial Evaluation Indrani and a subject during initial evaluation

Questions 1. Can non-literate users use traditional text- based user interfaces at all? 2. Do the proposed design principles for text- free user interfaces allow non-literate users to use computers, and to what extent? 3. Which of the design principles make the most difference for a text-free UI? Initial Evaluation NO Yes, but only partially; more work required Hypothesis: - Voice feedback - Imagery - Help

Outline: Text-Free User Interfaces Introduction Initial ethnography and iterative rapid prototyping Initial evaluation Optimal audio-visual representation Methodology Discussion

Text-Free UI Design principles: –Pen or touch interface –Liberal use of imagery No text (but numbers OK) Semi-abstracted cartoons –Voice feedback –Aggressive use of mouse- over functionality –Consistent help icon From initial round of research… Screenshot of text-free job search

Goal of Study Determine what types of icon representations are best for UIs for non-literate users. Audio: ○ With voice annotation ○ Without voice annotation Visual: ○ Photorealistic video ○ Hand-drawn animation ○ Photograph ○ Hand-drawn image ○ Text

Domain of Testing Desired characteristics of domain: Admits to visual representations, Consists of information which subjects are not exposed to on a daily basis, Has universally recognized meaning, Offers a range of cognitive complexity, Offers a range of visual complexity, and Can be applied to a practical purpose. Healthcare / medical imagery 13 medical symptoms: multimodal in nature, involving elements of color, temperature, and some requiring motion spanning

Drawing PhotographAnimationVideo Ensure consistency across representations… Creation of Representations

Tested Representations Drawing PhotographAnimationVideo Text Each with and without voice annotation (total 10 representations), for 13 medical symptoms

Experimental Set-Up Representations: One of 10 types Evenly distributed among subjects All 13 symptoms per subject Subjects: Illiterate or semi-literate ○ (no literate subjects) No PC experience Size: Individual; n = for each representation Total time up to one hour per session Threefold task:  Speak phrase that first comes to mind when shown a representation.  Explain symptom representation is intended to convey  Enact or depict the intended symptom (as explained by tester), for analysis Measured: Accuracy (correct, close, wrong) Response time (seconds) All comments recorded

Voice annotation critical, with dominant effect 30% quicker response time with audio Among cases with audio, hand- drawn representations (both animation and static imagery) most accurately recognized But, effect not dramatic Among cases without audio, animation most accurately identified 23% more accurate than static drawing Quantitative Results Number of accurate responses (out of 20) for various representations

Qualitative Results Test / text anxiety of subjects Subjects uncomfortable with test-like environment of studies Short-term category conditioning Priming for “aches” over other types of symptoms when they appeared in sequence Richer information is not necessarily better understood overall Photorealism contains clutter; associated with actual instance Unrelated Context-laying activity changes value of static versus dynamic imagery Socio-economic conditions correlated with cognition Those with less formal education had difficulty processing visuals and audio at the same time; confused by multi-modal interface

Outline: Text-Free User Interfaces Introduction Initial ethnography and iterative rapid prototyping Initial evaluation Optimal audio-visual representation Methodology Discussion

Initial Ethnography Open-ended interviews Various combinations: focus groups and individuals Topic: everything and anything Follow leads around hardships Structured interviews Focus on themes from open-ended interviews Prepare questions Be willing to deviate Goal: Information, intuition, rapport An open-ended interview in a subject’s home Methodology Time and attention critical! Photo: Indrani Medhi

Iterative Prototyping Build initial prototype, based on intuition from interviews. Iterate prototype and test. Maintain informality Work with small groups ○ 2-4 people Prompt whenever necessary Record every comment ○ Video-record, if possible (but, high procedural overhead) Listen to every comment Modify prototype Refine until steady state. Testing a prototype Methodology Photo: Indrani Medhi

Participative Design Modify standard methods for non-literate partners. Encourage storytelling. Use aids that can be physically manipulated. Be prepared for long, counseling-like sessions. Methodology Eliciting descriptions of medical symptoms with physical design aids Photo: Indrani Medhi Subject enacting medical symptoms

Formal Evaluation Base on techniques from courses, textbooks Plan according to standard practices Avoid direct counterfactuals Place task in a story Use friend or neighbor as protagonist, but not subject Chavan’s “Bollywood method” Be flexible on… Location (avoid labs) Formal instructions Etc. Methodology Video: Archana Prasad

Text-Free UI History Initial principles Optimal AV representa- tion Full-context video Text-Free UI for mobiles Psychology of non-literate users Text-Free healthcare information Online site (babajob.com) Practical Systems UI Research Other Research CompletedOngoingFuture Key: UI authoring tool Semi- automated cartooning

Discussion questions  Is it possible to replicate the same study on a mobile? How would the process/guidelines for design be different?  What might be some other areas where text-free UI could have potential? How?  What are some of the best tricks to eliminate designer biases from the design process?