Feedback and Presentation in the Microsoft Distributed Classroom Project Richard Anderson University of Washington and Microsoft Research

 t Richard Anderson UW & MSR2 Overview Presentation system –Develop system for lecture delivery that increases instructor flexibility Feedback system –Develop system that supports increased in class communication Design experiment –Experiment with system in actual classroom

 t Richard Anderson UW & MSR3 Educational Technology …in the winter of … I attended a mathematical school kept in Boston…on entering his room, we were struck at the appearance of an ample Blackboard suspended on the wall, with lumps of chalk on a ledge below, and cloths hanging at either side. I had never heard of such a thing before. [Mr. May, 1866]

 t Richard Anderson UW & MSR4 Classroom Technology Black board White board Overhead projector Transparencies –Persuasion, PowerPoint, LaTeX PC with data projector

 t Richard Anderson UW & MSR5 Research goal Develop presentation system for PC/Data Projector –Increase flexibility in lecture delivery –Support greater communication Design for instructional use at the university level Evaluate based on pedagogical impact

 t Richard Anderson UW & MSR6 Scenarios Distributed classroom –Multi site video conferenced classes –Data projectors in each room –Possible desktop access Lecture hall –Public projected display –Wireless enabled class –Possible student devices Laptop, tablet, handheld

 t Richard Anderson UW & MSR7 Teaching styles Wide range of teaching styles and opinions Instructors will use technology in different ways Instructors will use technology in unexpected ways We are not comparing between different technologies –PC delivery already established –Some situations require PC delivery

 t Richard Anderson UW & MSR8 DISC Project Internet based video conferenced classes Multicast technology to support multiple sites Deploy system on a PC Simple startup (the big red button)

 t Richard Anderson UW & MSR9 Preliminary Study UW Professional Master’s Program –Video conferenced class –Netmeeting for PowerPoint distribution –Smartboard ™ Electronic whiteboard Instructor interviews Student interviews Classroom observations Review of video archive (5 years of data)

 t Richard Anderson UW & MSR10 Study conclusions Need to support handwriting and slides Current electronic whiteboards not adequate Important to integrate whiteboard and slides –Every observed use of whiteboard was in the context of a slide Use pen based device as presenter console

 t Richard Anderson UW & MSR11 Electronic whiteboard comments "The whiteboard would be better if: both it and the slides could be seen at the same time the handwriting was captured more accurately; if I write in anything other than slowly-drawn block letters, it looks like a heap of pick-up sticks." "The problem with the whiteboard is that, unless you draw very carefully, half of it won't come out, and similarly when you erase. This is enough of a hassle that I wound up not using it much."

 t Richard Anderson UW & MSR12 PowerPoint comments "I could do some of what I wanted with PowerPoint animations, but these took a huge amount of time to prepare. Previously I had used slide overlays (transparencies) a lot, frequently presenting a slide that was partially filled in (some background and a question) and then using class discussion to fill it in." "PowerPoint is also a pain for the same reason it's a pain in non-distance classes - the slides impose a rigid structure on the lecture, and make it more difficult to adjust to the interactions that occur during it. "

 t Richard Anderson UW & MSR13 Presentation system Key idea – integration of ink and slides Lecture from tablet pc –Mouse input adequate for gesturing, but not for writing Write on PowerPoint Synchronized display of slides Real time transmission of ink

 t Richard Anderson UW & MSR14 Implementation Export slides as images, broadcast images –Advantages: easy and it worked. Viewer does not need to have PowerPoint. Easier support of a range of devices. –Disadvantages Loss of fidelity Modifying intermediate form Loss of animations, transitions

 t Richard Anderson UW & MSR15 Technical Issue: Slide distribution Current approach –Broadcast slide deck using multicast –Interpacket delay introduced to avoid network saturation –Rebroadcast on request Issues –Dealing with clients joining sessions –Large images (instructor’s baby pictures) Recognized as a challenge for distributed presentations

 t Richard Anderson UW & MSR16 Whiteboard features Scrollable overlays (virtual mylar) –Simulate overhead projector Minimize or hide slide for whiteboard use Ink and transparent ink Region highlight Remote pointer

 t Richard Anderson UW & MSR17 Classroom use Fujitsu tablet UW Distance master's class (Professional Master's Program, PMP) –Database –UW Lecturer, MSR classroom –3 hour evening lecture for 10 weeks Several sessions in introductory computer programming

 t Richard Anderson UW & MSR18 Observation results Used for both highlighting and examples Lecturers used "dense PowerPoint" –Presentations not designed for ink annotation Novelty wore off quickly –Instructors soon became comfortable Used from podium Some difficulty writing on Fujitsu tablet User interface issues to be resolved

 t Richard Anderson UW & MSR19 Unexpected use Alon Halevy: "Could I get volunteers to come up and write their sample queries on the tablet" Martin Dickey. TV Talk show style. Took the tablet to the audience, and had them write on it. Wireless tablet. Used the device to allow new interaction. Technology to enhance classroom collaboration.

 t Richard Anderson UW & MSR20 Current experiment Use presentation tool in Introductory Computer Programming, Summer 2002 Evaluate tool in sustained, large lecture use Deeper questions –Does the presentation tool enable a better educational experience? –Does the presentation tool alter the pedagogy?

 t Richard Anderson UW & MSR21 Research questions Gestures for lecturing from tablet –Specific technology very important –Limited experience with tablet pc –Specific gestures for presentation –Constraints imposed by “performance environment” Opportunities for manipulation of slide content

 t Richard Anderson UW & MSR22 Feedback System Motivation and challenge –Increase level of interaction in large lecture classes –Give instructor information on level of understanding of lecture Real time feedback –Students annotation of materials on personal devices –Real time summarization for instructor

 t Richard Anderson UW & MSR23 Prior work Telep (and others) Active class (UCSD) Class talk (UMass) Classroom 2000 University of Maryland Results mixed

 t Richard Anderson UW & MSR24 Challenges to Interaction in Large Classes Student apprehension –Concerns about interrupting class for “unimportant” questions Comment verbalization –Difficulty in expressing comments Feedback lag –Questions not formed until topic has been changed Single speaker paradigm

 t Richard Anderson UW & MSR25 Design Principles Non-verbal communication Anonymity Shared context for feedback Rapid, automatic synthesis of feedback Simple interfaces Value to all users

 t Richard Anderson UW & MSR26 Important questions Will instructors be willing to use system? Will instructors be able to take advantage of feedback during lecture? Will students be willing to use system? Will students benefits to students outweigh the costs of using the system?

 t Richard Anderson UW & MSR27 Feedback System Viewers select annotations from context menus Annotations associated with slide geometry Real time summary of annotations provided to viewer Potential to deploy on a wide range of devices –Take advantage of existing student devices

 t Richard Anderson UW & MSR30 Feedback categories Low impact to make comments Allow presenter to solicit particular feedback –State of mind (I’m confused, I understand) –Suggested action (Explain, Elaborate, Example, Define) –Pacing (Slower, Faster, Just Right) Presenter may set categories in a preprocessing phase

 t Richard Anderson UW & MSR31

 t Richard Anderson UW & MSR32 Summarization of feedback Presenter needs summarization if there is significant feedback Summarization done on "semantic regions" –Identify slide geometry - rectangles –Summarize comments in rectangles Alternate summarization methods –Intensity or Histogram Comment menus also linked to regions

 t Richard Anderson UW & MSR33 CSE 100 Experiment Design experiment Done in actual class Understand use and effect in real context Engineering a learning environment by introducing an intervention Study environment to guide design Conduct pen and paper feedback Use laptops in class for feedback

 t Richard Anderson UW & MSR34 Study CSE 100: Fluency in information technology Large, diverse introductory class –150 students, large lecture format Observations –Very limited student participation –Most students surveyed reported speaking in class less than once per week –Instructor reported few questions, so she could not gauge student understanding

 t Richard Anderson UW & MSR35 Pencil and paper study Group of students given paper copies of the slides Students asked to annotate slides from predefined categories –Slow Down, Explain, Question, Cool topic Simulate student experience Evaluate type of feedback delivered

 t Richard Anderson UW & MSR36 Classroom study Limitations: –One class period –Seven laptops (8 of 100+ students participated in study) –Novelty of system and participation in a study likely influenced how students used system –Instructor had a TA mediate feedback (intended use of the system is for instructor to get feedback directly)

 t Richard Anderson UW & MSR37 Study data Logging of system actions Classroom observations Student questionnaires Instructor interviews Draw qualitative conclusions from multiple sources of data

 t Richard Anderson UW & MSR38 System usage Students using laptops gave between 1 and 9 annotations Students using paper gave between 7 and 15 annotations Category type LaptopPaper Slow Down 27 Question221 Explain1219 Cool topic1613

 t Richard Anderson UW & MSR39 Instructor response Change in presentation style Altering pace of lecture in response to slow down comments Providing additional explanation in response to question or explain Examples of terms annotated –Tracking number – forward reference, clarified later –Batch processing – old term which students weren’t familiar with

 t Richard Anderson UW & MSR40 Reactions Positive instructor interaction –“[The students] can actually say something right away, and if the teacher has the wherewithal or time, they can actually make small improvement or small changes that will treat those anxieties or concerns doing what they’re doing that the students like”

 t Richard Anderson UW & MSR41 Student reactions Students completing survey said that the instructor responded to their comments “Further explanations were given when asked to” “The computer guy [the mediator] told [the instructor] that there was a question and she did respond to it”

 t Richard Anderson UW & MSR42 Effect on classroom Student participation on day of experiment was consistent with previous classes –6 student initiated interactions on day of experiment, 11 teacher initiated interactions –Average of 6 student initiated interactions in earlier classes, 6 teacher initiated interactions New style of interaction –Teacher response to feedback creating discussion

 t Richard Anderson UW & MSR43 Conclusions Positive evidence from design experiment –Students used system –Instructor was able to take advantage of information –Linking feedback to content was important –Selected categories appropriate –Positive response from instructor and students –Level of interaction did not drop

 t Richard Anderson UW & MSR44 Future work System development Larger field tests –Introductory computing, summer 2002 –Direct instructor use of system –Sustained use of feedback system Feedback infrastructure as platform for “structured interaction”

 t Richard Anderson UW & MSR45 Acknowledgments UW –Tammy Vandergrift, Steve Wolfman, Ken Yasuhara MSR DISC Project –portal.learningwebservices.com/disc