Raja Kushalnagar Poorna Kushalnagar Fadi Haddad SingleScreenFocus Raja Kushalnagar Poorna Kushalnagar Fadi Haddad

Classroom access is environment dependent For deaf visual learners: Direct Instruction: Teacher to student (sign/lip-read) Lecture info is visual Teacher and student jointly use active visual Mediated Instruction: Teacher to Speech-to-Text Display to student Deaf students do not have equal access Auditory info becomes simultaneous visual info Teacher and student may differ on active visual Visuals often far apart (dispersion) Classroom access is environment dependent Deaf students are primarily visual learners. In an ideal classroom, there is direct instruction from the teacher to the student. All information is visual including using sign language, and visual aids such as blackboards or whiteboards, overhead presentations, written words, physical objects. Instructors teaching directly to deaf students will use appropriate cues to direct deaf students to the appropriate area of focus. But in mediated instruction, deaf students will get visual access to audio through interpreters/captioners. The deaf students do not have equal access to the information because they have to manage multiple visual areas of focus simultaneously. I’d like to mention that NTID’s secondary goal is to support deaf people learning nationwide as well. As part of our department’s research and development, we have been investigating differences in student learning between deaf and hearing especially in STEM (Science, Technology, Engineering and Math) areas. We know that STEM courses tend to have more visual information and therefore more split attention challenges for the deaf. In response to our findings, we are developing technology to help manage this challenge, including TSD. Speech-to-Text Display: Read and Scan. Repeat.

Accessibility: Visual Dispersion Viewing speech-to-text on laptop Information loss due to: Separation between Text Presenter information Split focus All else constant, use of speech-to-text use is easier when visual support for the student is maximized. This video example illustrates the challenges in managing multiple visuals that are widely dispersed. It is is from my graduate school days. Audio description.

Prior Work ClassInFocus: Cavender, et al. ASSETS 2009 Multiple Views: Kushalnagar, et al. ASSETS 2010 Captions versus transcripts: Kushalnagar, et al. W4A 2013, Lasecki, et al. 2014 Cavender found that DHH liked consolidating visuals on a single screen. Kushalnagar found that DHH liked consolidation, and sizing visuals according to their importance. Kushalnagar, and Lasecki found that there are tradeoffs between displaying the speech-to-text in a separate page, versus displaying it within the video.

Consolidating Classroom Visuals Investigate the efficacy of the following: Project the Speech-to-Text information next to the teacher. Display speech-to-text transcript as captions/subtitles (few lines) Idea: If captions are easy to read because they’re in the video, why not flip this around and display the speech-to-text with the classroom visuals?

Abstract Contains three components in one screen Organized visual attention Goals to Deaf and hard-of-hearing students in mainstream schools Integrate into school’s multimodal technology Tablets iPad Android Windows

Introduction Issues Deaf and hard-of-hearing has several accommodations Sign Language Interpreters Real-time Captions Note-takers Multiple visual reception in classroom without any source of auditory reception Student has more chance to miss any information Teach and interpreters’ movements relies on student’s position in the classroom

Introduction Solutions Integrate all accessibilities into tablet or monitor for visual attention Less chance to miss any information with all accommodations on tablet or monitor More awareness on what is happening in the classroom without an auditory reception No limitation to staff’s movements

Background Related to project ‘ClassInFocus’ Similar User Interface models Goal for deaf and hard of hearing students in mainstream classes ClassInFocus does not use body tracking system Restricts teacher and interpreter’s movement SingleScreenFocus tracks Teacher and interpreter moves without any restriction Very few related work

How does it work? Private router network is connected to the PC Two tracking camera is connected to router The computer retrieves camera feed from the router and then displays two cameras to the main monitor Tablets connected to the computer

Equipment Settings Opening Idea Prototype Settings Kinetic Camera with Body Tracking Issues: Small amount of time Prototype Settings Prototype project: Two swivels Two iPod Tablet that controls the PowerPoint PC connected to Large TV

Evaluation Involved into two environment Surveys Thirteen Participants Regular Classroom without SingleScreenFocus Regular Classroom with SingleScreenFocus Surveys With or without SingleScreenFocus? Do they recommend SingleScreenFocus to others? Thirteen Participants

Questions Q1: What is your rating of the [Regular View/SingleScreenFocus] view lecture? Q2: Did the [Regular View/SingleScreenFocus] view help during the lecture? Q3: Is [Regular View/SingleScreenFocus] view easy to use? Q4: I am confident in using [Regular View/SingleScreenFocus] view in class. Q5: I would recommend [Regular View/SingleScreenFocus] view for use by other deaf or hard of hearing students. Q1: SV 4.5 (0.5) RV 3.7 (0.9) Q2: SV 4.7 (0.5) RV 4.0 (0.8) Q3: SV 4.9 (0.3) RV 3.8 (0.8) Q4: SV 4.6 (0.5) RV 3.9 (0.6) Q5: SV 4.8 (0.4) RV 4.2 (0.7)

Responses Is [Regular View/SingleScreenFocus] view easy to use?

Responses Advantages: I did not have to move my head much Easy to see everything Everything on one screen Disadvantages: Lighting, dark sometimes, bright sometimes Interpreter moving too fast Teacher feed not necessary

Results The participants highly recommended SingleScreenFocus They generally prefer SingleScreenFocus in class over no SingleScreenFocus Many positive feedback comments with small minor adjustment changes Those could be solved with the future developed software that’ll match to their needs

Future Work Implement multiple software and hardware into one system User Preferences Camera feed on left or right side of PowerPoint Options to choose along with PowerPoint Teacher Feed Interpreter Feed Captioning Teacher and student account with access to the PowerPoint Teacher permits student to enter control system Once the teacher completed presenting one slide and presses next, the slide will not proceed until the interpreter has completed. Universal System Android Apple Windows