1. Raja Kushalnagar Poorna Kushalnagar Fadi Haddad
SingleScreenFocus
Raja Kushalnagar
Poorna Kushalnagar
Fadi Haddad

2. Classroom access is environment dependent
For deaf visual learners:
Direct Instruction: Teacher to student (sign/lip-read)
Lecture info is visual
Teacher and student jointly use active visual
Mediated Instruction: Teacher to Speech-to-Text Display to student
Deaf students do not have equal access
Auditory info becomes simultaneous visual info
Teacher and student may differ on active visual
Visuals often far apart (dispersion)
Classroom access is environment dependent
Deaf students are primarily visual learners. In an ideal classroom, there is direct instruction from the teacher to the student. All information is visual including using sign language, and visual aids such as blackboards or whiteboards, overhead presentations, written words, physical objects. Instructors teaching directly to deaf students will use appropriate cues to direct deaf students to the appropriate area of focus.
But in mediated instruction, deaf students will get visual access to audio through interpreters/captioners. The deaf students do not have equal access to the information because they have to manage multiple visual areas of focus simultaneously.
I’d like to mention that NTID’s secondary goal is to support deaf people learning nationwide as well. As part of our department’s research and development, we have been investigating differences in student learning between deaf and hearing especially in STEM (Science, Technology, Engineering and Math) areas. We know that STEM courses tend to have more visual information and therefore more split attention challenges for the deaf. In response to our findings, we are developing technology to help manage this challenge, including TSD.
Speech-to-Text Display: Read and Scan. Repeat.

3. Accessibility: Visual Dispersion
Viewing speech-to-text on laptop
Information loss due to:
Separation between
Text
Presenter information
Split focus
All else constant, use of speech-to-text use is easier when visual support for the student is maximized.
This video example illustrates the challenges in managing multiple visuals that are widely dispersed. It is is from my graduate school days.
Audio description.

4. Prior Work
ClassInFocus: Cavender, et al. ASSETS 2009 Multiple Views: Kushalnagar, et al. ASSETS 2010 Captions versus transcripts: Kushalnagar, et al. W4A 2013, Lasecki, et al. 2014
Cavender found that DHH liked consolidating visuals on a single screen.
Kushalnagar found that DHH liked consolidation, and sizing visuals according to their importance.
Kushalnagar, and Lasecki found that there are tradeoffs between displaying the speech-to-text in a separate page, versus displaying it within the video.

5. Consolidating Classroom Visuals
Investigate the efficacy of the following:
Project the Speech-to-Text information next to the teacher.
Display speech-to-text transcript as captions/subtitles (few lines)
Idea: If captions are easy to read because they’re in the video, why not flip this around and display the speech-to-text with the classroom visuals?

6. Abstract Contains three components in one screen
Organized visual attention
Goals to Deaf and hard-of-hearing students in mainstream schools
Integrate into school’s multimodal technology
Tablets
iPad
Android
Windows

7. Introduction
Issues
Deaf and hard-of-hearing has several accommodations
Sign Language Interpreters
Real-time Captions
Note-takers
Multiple visual reception in classroom without any source of auditory reception
Student has more chance to miss any information
Teach and interpreters’ movements relies on student’s position in the classroom

8. Introduction Solutions
Integrate all accessibilities into tablet or monitor for visual attention
Less chance to miss any information with all accommodations on tablet or monitor
More awareness on what is happening in the classroom without an auditory reception
No limitation to staff’s movements

9. Background Related to project ‘ClassInFocus’
Similar User Interface models
Goal for deaf and hard of hearing students in mainstream classes
ClassInFocus does not use body tracking system
Restricts teacher and interpreter’s movement
SingleScreenFocus tracks
Teacher and interpreter moves without any restriction
Very few related work

10. How does it work? Private router network is connected to the PC
Two tracking camera is connected to router
The computer retrieves camera feed from the router and then displays two cameras to the main monitor
Tablets connected to the computer

11. Equipment Settings Opening Idea Prototype Settings
Kinetic Camera with Body Tracking
Issues: Small amount of time
Prototype Settings
Prototype project:
Two swivels
Two iPod
Tablet that controls the PowerPoint
PC connected to Large TV

12. Evaluation Involved into two environment Surveys Thirteen Participants
Regular Classroom without SingleScreenFocus
Regular Classroom with SingleScreenFocus
Surveys
With or without SingleScreenFocus?
Do they recommend SingleScreenFocus to others?
Thirteen Participants

13. Questions
Q1: What is your rating of the [Regular View/SingleScreenFocus] view lecture?
Q2: Did the [Regular View/SingleScreenFocus] view help during the lecture?
Q3: Is [Regular View/SingleScreenFocus] view easy to use?
Q4: I am confident in using [Regular View/SingleScreenFocus] view in class.
Q5: I would recommend [Regular View/SingleScreenFocus] view for use by other deaf or hard of hearing students.
Q1: SV 4.5 (0.5) RV 3.7 (0.9)
Q2: SV 4.7 (0.5) RV 4.0 (0.8)
Q3: SV 4.9 (0.3) RV 3.8 (0.8)
Q4: SV 4.6 (0.5) RV 3.9 (0.6)
Q5: SV 4.8 (0.4) RV 4.2 (0.7)

14. Responses
Is [Regular View/SingleScreenFocus] view easy to use?

15. Responses Advantages: I did not have to move my head much
Easy to see everything
Everything on one screen
Disadvantages:
Lighting, dark sometimes, bright sometimes
Interpreter moving too fast
Teacher feed not necessary

16. Results The participants highly recommended SingleScreenFocus
They generally prefer SingleScreenFocus in class over no SingleScreenFocus
Many positive feedback comments with small minor adjustment changes
Those could be solved with the future developed software that’ll match to their needs

17. Future Work Implement multiple software and hardware into one system
User Preferences
Camera feed on left or right side of PowerPoint
Options to choose along with PowerPoint
Teacher Feed
Interpreter Feed
Captioning
Teacher and student account with access to the PowerPoint
Teacher permits student to enter control system
Once the teacher completed presenting one slide and presses next, the slide will not proceed until the interpreter has completed.
Universal System
Android
Apple
Windows