1. User Centered Design in the Course Management Project http://bugs.sakaiproject.org/confluence/display/CM/Home Marc Brierley, Stanford University Daphne Ogle, University of California, Berkeley

2. Agenda Course management project overview Framing the work in a user-centered design (UCD) approach User research –interviews and observations Making sense of the research –user, goal and workflow modeling The rest of the UCD process –requirements, framework & design How does this process change things?

3. Overview of Course Management Project Why? –SIS integration –Confusing and overlapping functionality What? –New APIs –Framework enhancements –New and enhanced tools When? –Winter ‘07 for production

6. Stage 1 - User Research Interviews and Observations Across several institutions and user groups Support panel: “Data” from the users’ perspective

7. User Research Goals Observing real users in real context What we wanted to find out –Problems / pain points –Domain knowledge –Broader context / not just product-focused –Basic understanding of current tasks –Clear understanding of goals

8. Interview & Observation Nuts & Bolts Interview Prep Running the interview Observing quietly and unobtrusively Processing the findings (right away!)

9. Interview Prep Recruiting –Who are users? –What if you don’t know m/any users? Bridge people Scheduling –People, time, location Coming up with questions –Categories & general questions Confirmation / Reminder

10. Interview Questions Flow 1. Introductions 2. Why we're here: we've been asked to design a next generation tool for the management of courses within a Learning Management System 3. What we'll ask: your day, your frustrations Overview Questions 1. Tell us about your responsibilities and typical day 2. What would you say is the main goal in the task(s) you have? 3. What’s your favorite aspect of your job? 4. What don’t you like about your job? What’s tasks would you rather not have to do? Activity questions General 1. What kind of administrative tasks do you deal with around teaching your course? 2. Can you talk generally about how students sign up for and switch classes? What’s the beginning of the term like? How far into the semester can students switch courses? 3. Is your class ever listed as a cross listing? What does that mean? 4. What does the term "section" mean to you? * How do you handle sections? * What are your section types? * How often do students switch sections?

11. Interview Questions CMS-related (if there is an existing product) 1. How do you handle the creation of a site for your course? 2. How do you handle content, if any? 3. With regards to course syllabus information, does your cms support uploading of a local file as well as entering text right into the syllabus page? Which option do you use? 4. How are students added and dropped from the site? Other general question if they haven't come up: Opportunity * What things waste your time? Priorities/Goals * What makes a good work day? A bad one? * What is the most important thing you do? * What does your boss think is important? Support * What kind of training do you receive?

12. Interview Questions Connections * Who else do you work with to get things done? * Who else provides information to you and to whom do you provide information? Final Questions/Wrap-up 1. Are there any less frequent tasks you do that I might not have observed today? 2. Follow-up on interesting points Observation * Environment Observations * Observations/General

13. Running the interview It takes 2 –Interview leader & primary note taker –Balance between natural conversation and moving things forward Context is important! Set expectations –Introductions –Not questioning users skills, abilities, work –Ask permission to follow-up via email –Ask for suggestions of others to talk to Be respectful of time –You’ll likely want to talk to them again

14. Observing quietly and unobtrusively Observe REAL work Notice things outside of the system Environment is important Focus on detail - Analyze for goals Harder than it seems!

15. Document the findings (right away!) Collaborative analysis of what you learned while it’s fresh Organized note taking pays off here –Composition book –Indexes Don’t guess Don’t have to wait until the last interview or observation –Start creating models early and iterate as you learn more

16. Stage 2 - Model the Users, WHY? Helps key stakeholders understand what was learned in research Guides the design and development of the software Prioritizes requirements

17. Types of User Models Mental Models Personas Context Scenarios

18. Mental Models Across the various users, what are the mental constructs that shape how people think about the situation or problem? We found (no huge surprise) that perceptions of how courses are structured made a big impact.

19. Mental Models: Course Structures

22. Personas What are the users’ values and goals (not just tasks)? Looking for patterns of usage and behavior What are their human characteristics?

23. Sara Windsor Faculty, MIT Goals: Build on course materials from term to term Not to have to ask for help To spend as little time as possible doing administrative work; she’ll delegate to her TA’s To get tenure (get credit for tenure for everything she does) Use technology to help create an engaging and interactive environment for her on-line students where she can track their progress To be respected by students, colleagues and dean of school Level of Expertise: Teaching for 10 years, uses software apps like Word, excel, ppt, email, on-line research Sara taught at the University of Pennsylvania before joining MIT 2 years ago. She is married with 2 kids and stays extremely busy teaching an undergrad psychology course for the 2nd year and a new on-line course in organizational management. She is fairly comfortable with computers. She’s excited to be using technology to help with her teaching but she sees as a means to an end in order to stay in touch with students and allow them access to course resources when they need it. In fact, the LMS will replace her face-to-face communication with students for her on-line course. She knows that even students she sees in class twice a week would like to her to be more responsive and looks to the course site to help with that. She used Blackboard at PA and notices that she has expectations based on that experience…some good, some bad. Carol Hudson CMS part-time support/student Carol is has been back in school for year working toward her masters degree Education. She’s doing support for Yale’s LMS 12 hours a week to bring in some income while she’s in school. Her student loans barely cover tuition and rent. Carol likes to help people and gets frustrated when she doesn’t feel like she has the answer for customers…or even worse is confused herself. She uses the LMS for a couple of her own classes since she began working in the support group she’s gained empathy for the work faculty have to put into managing the course sites. Goals: Make it through graduate school with good grades Earn money to pay for living expense while in school Give the right answers Teach people Level of Expertise: Experience as a student on LMS. Fairly comfortable with technology and the web.

24. Whiteboard Exercises Summarize notes Persona scales

25. Summarize Notes

26. Persona Scales Brainstorm types of characteristics to make scales Put each user on continuum or in a bucket Look for the groupings

29. Writing a persona State the user’s goals –Life: Retire by 45 –Experience: Not feel stupid –End: Set up my course site Include behavior patterns, not job description Add some biographic info to make the persona seem real, but not too much

30. Sara Windsor Faculty, MIT Goals: Build on course materials from term to term Not to have to ask for help To spend as little time as possible doing administrative work; she’ll delegate to her TA’s To get tenure (get credit for tenure for everything she does) Use technology to help create an engaging and interactive environment for her on-line students where she can track their progress To be respected by students, colleagues and dean of school Level of Expertise: Teaching for 10 years, uses software apps like Word, excel, ppt, email, on-line research Sara taught at the University of Pennsylvania before joining MIT 2 years ago. She is married with 2 kids and stays extremely busy teaching an undergrad psychology course for the 2nd year and a new on-line course in organizational management. She is fairly comfortable with computers. She’s excited to be using technology to help with her teaching but she sees as a means to an end in order to stay in touch with students and allow them access to course resources when they need it. In fact, the LMS will replace her face-to-face communication with students for her on-line course. She knows that even students she sees in class twice a week would like to her to be more responsive and looks to the course site to help with that. She used Blackboard at PA and notices that she has expectations based on that experience…some good, some bad. Carol Hudson CMS part-time support/student Carol is has been back in school for year working toward her masters degree Education. She’s doing support for Yale’s LMS 12 hours a week to bring in some income while she’s in school. Her student loans barely cover tuition and rent. Carol likes to help people and gets frustrated when she doesn’t feel like she has the answer for customers…or even worse is confused herself. She uses the LMS for a couple of her own classes since she began working in the support group she’s gained empathy for the work faculty have to put into managing the course sites. Goals: Make it through graduate school with good grades Earn money to pay for living expense while in school Give the right answers Teach people Level of Expertise: Experience as a student on LMS. Fairly comfortable with technology and the web.

31. The Rest of the UCD Process Stage 3 - Requirements Stage 4 - Conceptual Framework Stage 5 - Design

32. Stage 3 - Requirements Constructing context scenarios, based on persona and organizational goals Creating an initial list of each persona's major data/information needs Creating an initial list of functional needs (these are actions that personas' will need to take in relation to that data) Noting persona capabilities that will affect the design (for example, physical or training limitations) * *

33. Context Scenario: Create a New Course Site Goals: Distribute course content Start communicating with students Organize materials for first time Two weeks before the term, Sara is teaching Abnormal Psychology for the first time and wants to get her “course site” going. Gathers a rough set of materials (books, articles, slides, etc.) and starts grouping things together Writes syllabus draft based on these materials Confirms with four previously recruited TAs that they will be working with her

34. Three days later, she sits down at her computer in her home office and logs into her institution’s LMS to set up her course site She sees the “create site” function and chooses it She sees a list of the courses she’s teaching this term including Psychology 101 and Abnormal Psychology (including its discussion sections) She sees Psychology 101 already has a course site and remembers she did that last month She creates the course site for Abnormal Psychology She sees that the site was created She goes to the site to post her syllabus and readings for the first week She tells her students to read the materials before the first lecture As Sara logs out of the LMS she hopes she will remember where she left off when she puts What ifs: She doesn’t see the courses she is teaching when she chooses “create site”?

35. Stage 4 - Conceptual Framework A number of key-path scenario descriptions A coherent conceptual framework that describes the product, service, or system that satisfies your persona's goals and the organization's objectives Descriptions of key features and major interface elements and how they work together in the framework Primary navigation characteristics and information architecture. * *

36. Stage 5 - Design Full set of mockups and behavior specifications based on framework and scenarios

37. How do we expect the outcomes of this process to impact the design?

38. Creating a Course Site - Current

47. Creating a Course Site - New

52. When do you do all this stuff? Before development starts!

53. Other related activities Domain/market research (CIO/Deans/marketing) Development/Design Iteration Usability Testing Graphic Design/Branding

54. Resources Course Management Confluence Site, http://bugs.sakaiproject.org/confluence/ display/CM/Home http://bugs.sakaiproject.org/confluence/ display/CM/Home Cooper, http://www.cooper.com/http://www.cooper.com/ –http://www.cooper.com/newsletters/2001_07/perfecting_your_personas.htmhttp://www.cooper.com/newsletters/2001_07/perfecting_your_personas.htm Menlo Innovations, http://www.menloinstitute.com/ http://www.menloinstitute.com/

55. Thanks! See you at the Course Management BOF, place and time? Questions?