1. Tests & Quizzes (aka SAMigo) Keli Amann, User Experience Specialist Christine Doherty, User Support Specialist Stanford University June 17, 2010

2. Feature Overview 2.7 Highlights 2.8 Roadmap Assessment in 3akai Q&A Agenda 211th Sakai Conference - June 15-17, 2010

3. What are SAMigo’s Features: 311th Sakai Conference - June 15-17, 2010 Assessment Authoring Assessment Publishing Assessment-Taking Management Assessment Grading Question Pools Management Assessment-Type Management

4. Online test Self-study questions Homework, problem-sets Paper, code, or project submission Language drills Quick knowledge probes Surveys Others??? Pedagogical Flexibility 411th Sakai Conference - June 15-17, 2010

5. Screen overview 511th Sakai Conference - June 15-17, 2010 Assessments Pending Assessments Published Assignments Active Inactive New Assessments Create assessment OR Import assessment Edit Assessment Assessment Settings Publish Preview View student submission

6. 2.7 Highlights 11th Sakai Conference - June 15-17, 20106

7. Main Page 711th Sakai Conference - June 15-17, 2010 Assessments Pending Assessments Published Assignments Active Inactive New Assessments Create assessment OR Import assessment

8. Create Assessment options instead of 2 Create buttons Assessment builder = standard Edit Assessment page (default) Markup text = Quick Create feature Placement of Assessment Type menu under Title field to emphasize it’s optional Label changes with helper hints Prompt for entering title in text field Explanation of which files types can be imported Main Page 811th Sakai Conference - June 15-17, 2010

9. 9 2.6 2.7 Main Page: Create

10. 11th Sakai Conference - June 15-17, 201010 2.6 2.7 Main Page: Create

11. 11th Sakai Conference - June 15-17, 201011 2.6 2.7 Main Page: Create

12. Main Page, Pending 1211th Sakai Conference - June 15-17, 2010 Assessments Pending Assessments Published Assignments Active Inactive New Assessments Create assessment OR Import assessment

13. Actions drop-down menus replace links under assessment titles Addition of Edit, Preview, Publish actions Copy changed to Duplicate Main Page, Pending 1311th Sakai Conference - June 15-17, 2010

14. Main Page: Pending 11th Sakai Conference - June 15-17, 201014 2.6 2.7

15. Main Page: Published 1511th Sakai Conference - June 15-17, 2010 Assessments Pending Assessments Published Assignments Active Inactive New Assessments Create assessment OR Import assessment

16. Preview action will show changes made to published version of assessment In Progress column shows how many students have started (but not submitted) assessments Submitted column shows number of students with completed submissions; hyperlinked number leads to Total Scores page (same as Scores action) Release To column displays selected groups by hovering over information icon Main Page: Published 1611th Sakai Conference - June 15-17, 2010

17. Main Page: Published 11th Sakai Conference - June 15-17, 201017 2.6 2.7 *OOTB, Edit (not pictured) only appears if students haven’t started

18. Main Page: Columns 11th Sakai Conference - June 15-17, 201018 2.6 2.7

19. Edit Assessment Page 1911th Sakai Conference - June 15-17, 2010 Assessments Pending Assessments Published Assignments Active Inactive New Assessments Create assessment OR Import assessment Edit Assessment Assessment Settings Publish Preview

20. Auto-scored Multiple Choice (single and multiple correct) Survey (multiple choice, different rating scales available) Fill in the Blank Numeric Response Matching True/False Manually scored Short Answer/Essay Audio Recording File Upload About Question Types 2011th Sakai Conference - June 15-17, 2010

21. Reorder of action links; addition of Publish action Add Part action moved down to level of Add Question menu Edit Assessment Page 2111th Sakai Conference - June 15-17, 2010

22. 22 2.6 2.7 Edit Assessment Page: Add Part

23. 11th Sakai Conference - June 15-17, 201023 2.6 2.7 Edit Assessment Page: Preview, Publish

24. About Settings Configures delivery of assessments (how they appear to students) Options include: display/organization late handling number of submissions feedback/grading options high security and time limit for high stakes testing Most settings are editable after publishing, but it is still important for instructors to review carefully 2411th Sakai Conference - June 15-17, 2010

25. Improvements to Preview Fill in the Blank and Numeric Response text fields and Matching selection menus made functional Timer (inaccurate in Preview mode) replaced with graphic/explanation Submit for Grading and Exit confirmation pages included 11th Sakai Conference - June 15-17, 201025

26. Preview screen 11th Sakai Conference - June 15-17, 201026

27. Partial Credit for Multiple Choice Questions Allows instructor to enter percentages of credit to give student for selections other than correct answer Correct answer is set to 100% automatically; can not be edited Reset links allow instructor to go back to default grading logic or reset percentage values to 0 Validation requires value to be between 0 and 100 (inclusive); no negative numbers allowed Applies to Single Correct Answer only Can not be used in conjunction with Negative Marking Feature needs to be turned on in Sakai Properties; not OOTB Contributed by Rice University 11th Sakai Conference - June 15-17, 201027

28. 11th Sakai Conference - June 15-17, 201028

29. 11th Sakai Conference - June 15-17, 201029

30. 11th Sakai Conference - June 15-17, 201030

31. 11th Sakai Conference - June 15-17, 201031

32. 11th Sakai Conference - June 15-17, 201032

33. Print Assessment (with or without feedback) Print action appears in Actions menu on main authoring page and as link in Edit Assessment page Available for both pending and published assessments Options include: Showing short answer key Showing feedback (question-level and selection-level) Showing intro and part titles Changing font size of document Printing as HTML or downloading a PDF Feature needs to be turned on in Sakai Properties; not OOTB Contributed by Universidad Politecnica de Valencia 11th Sakai Conference - June 15-17, 201033

34. 11th Sakai Conference - June 15-17, 201034

35. 11th Sakai Conference - June 15-17, 201035

36. 11th Sakai Conference - June 15-17, 201036

37. 11th Sakai Conference - June 15-17, 201037

38. 11th Sakai Conference - June 15-17, 201038

39. 11th Sakai Conference - June 15-17, 201039

40. Email Notification Option to send site members (or group members) email notification upon publishing or republishing assessment Publishing without notification is set as the default to prevent accidental spamming Important settings are pre-populated in notification: Assessment title Available date, due date Group released to (entire site, section(s)/group(s) or anonymous users) Assessment URL Time limit (if applicable) Number of submissions allowed Grading and feedback settings Option to include a personal message 11th Sakai Conference - June 15-17, 201040

41. 11th Sakai Conference - June 15-17, 201041

42. 11th Sakai Conference - June 15-17, 201042

43. 11th Sakai Conference - June 15-17, 201043

44. 11th Sakai Conference - June 15-17, 201044

45. Editable Parts in Published Assessments In 2.6, only questions and settings were editable in published assessments In 2.7, Parts are editable in the following ways: Title Information (longer text field) Attachments (add or remove) Operations that are more likely to create discrepancies in scoring are not editable: Random Draw options Question Ordering (when not using Random Draw) 11th Sakai Conference - June 15-17, 201045

46. 11th Sakai Conference - June 15-17, 201046

47. 11th Sakai Conference - June 15-17, 201047

48. File Upload in Question Feedback Option to upload files as feedback for each question; multiple files per question allowed Available on Questions page or student’s full submission page (under Scores) Update button must be clicked in scoring page after file uploaded with attachment helper Useful for returning commented papers to students Note: File size now displayed next to all file uploads (student submissions and files uploaded by instructor) 11th Sakai Conference - June 15-17, 201048

49. 11th Sakai Conference - June 15-17, 201049

50. 11th Sakai Conference - June 15-17, 201050

51. 11th Sakai Conference - June 15-17, 201051

52. 11th Sakai Conference - June 15-17, 201052

53. Improvements for No Submission Records Applies to Total Scores page when student identities can be seen by graders Comments: Disabled for students with no submissions ‘What’s This’ link to clarify purpose of Comments field Adjustment Scores: If adjustment score is entered for student with no submission, the score is now sent to the Gradebook and to the Export file Useful for handling offline submissions (e.g., students experiencing technical difficulties) 11th Sakai Conference - June 15-17, 201053

54. 11th Sakai Conference - June 15-17, 201054

55. Improvements in Delivery Mode Changes in button labels “Save and Continue” is now Next “Save for Later” is now Exit Changes in button configuration Navigation buttons to left, Save/Exit/Submit buttons to right Submit for Grading is rightmost button on the last question page Auto-Save and Save (on demand) Improved button layout in audio recording applet 11th Sakai Conference - June 15-17, 201055

56. 11th Sakai Conference - June 15-17, 201056 2.6 Button Configuration in Delivery Mode

57. 11th Sakai Conference - June 15-17, 201057 2.7 Button Configuration in Delivery Mode

58. Auto-Save and Save (on demand) Auto-Save: Saves student work without interruption (no screen refresh) Runs at configurable interval (set in Sakai Properties) Resets session timeout for inactivity Useful for long assessments on one page or high stakes testing Save (on demand): Allows students to save anytime while remaining on same page Useful for answers that require significant time (e.g., essay questions) Contributed by Rutgers University 11th Sakai Conference - June 15-17, 201058

59. Improved Button Layout in Audio Recording Applet Record and Stop are two separate buttons now, replacing toggle Reduces chances of students erasing recording by accident 11th Sakai Conference - June 15-17, 201059

60. Students View All Submissions/Scores In 2.6, only the submission with the recorded score (highest or last) displays in the student’s Submitted Assignments list In 2.7, the student’s Submitted Assignments list will show all submissions/scores by default Students can collapse the Submitted Assignments list to show only recorded scores Useful for self-study or practice exam scenarios; students can see feedback for multiple submissions Contributed by Universidad Politecnica de Valencia 11th Sakai Conference - June 15-17, 201060

61. 11th Sakai Conference - June 15-17, 201061

62. 11th Sakai Conference - June 15-17, 201062

63. Performance Improvements Performance has been improved in the following areas: Generation of Statistics for high number of submissions (~500) Loading of Questions page (under Scores) for File Upload and Audio Recording questions Downloading of student submission files (from File Upload questions) 11th Sakai Conference - June 15-17, 201063

64. 2.8 Roadmap (independent) 64 Respondus integration Record Average Score for multiple submissions* Allow students to use Rich-Text Editor for Short Answer/Essay question type** Remove 4000 character limit for student response in Short Answer/Essay question type** Allow instructors to enter zero score for unsubmitted assessments Allow instructors to show students scores only, without showing any other feedback (e.g., questions/responses) More to be discussed * Based on contributions from Rice University ** Based on contributions from Longsight

65. Assessment in Sakai 3.0? 65 User Research and Concept Development: Fall 09-Summer 2010 No such thing as Tests & quizzes; functionality distributed All learning activities initiated from (syllabus schedule, to do, repository) in same basic way Activities stored in repository for sharing or reuse: find in “drawer” or repository Work ongoing for instructor evaluation student views of syllabus, to do, grades http://confluence.sakaiproject.org/display/UX/ Sakai+3.0+capabilities+for+learning+activities

66. Questions? 11th Sakai Conference - June 15-17, 201066

67. Contacts Questions and feedback: samigo-dev@lists.stanford.edu Lydia Li, Stanford University Project Manager lydial@stanford.edu Homepage: http://confluence.sakaiproject.org/confluence/display/SAM 6711th Sakai Conference - June 15-17, 2010

68. Appendix Review of capabilities of Sakai 2.x 6811th Sakai Conference - June 15-17, 2010

70. Organize & Publish Assessments 7011th Sakai Conference - June 15-17, 2010 Pending Assessments Starting page for assessment authoring Can be created using markup text (aka Quick Create) Can be exported/imported in QTI XML format Can create duplicates easily When ready, published via Settings Published Assessments for student assessment-taking When students respond, Scores action appears Can edit published assessments and most settings Can remove published assessments

71. Auto-scored Multiple Choice (single and multiple correct) Survey (multiple choice, different rating scales available) Fill in the Blank Numeric Response Matching True/False Manually scored Short Answer/Essay Audio Recording File Upload Question Types 7111th Sakai Conference - June 15-17, 2010

72. At all levels of assessment authoring: Assessment/Part Headers Questions Selections Feedback Rich Text & Attachments 7211th Sakai Conference - June 15-17, 2010

73. Quick Create Efficie ncy created by reducing number of clicks, pages loading and saving Useful for transferring assessments created in other applications or LMS (e.g., MS Word, WebCT), no need to paste into different fields Supports Multiple Choice, Fill in the Blank, Short Answer/Essay, and True/False question types Rich text, attachments, and additional questions can be added later, through assessment authoring page 73 11th Sakai Conference - June 15-17, 2010

74. Import/Export Supports QTI v 1.2 Export types: IMS QTI-compliant XML or IMS Content Packaging (for use with attachments) Useful for: Copying assessment to different site Archiving assessments offline 7411th Sakai Conference - June 15-17, 2010

75. Settings Configures delivery of assessments (how they appear to students) Options include: display/organization late handling number of submissions feedback/grading options high security and time limit for high stakes testing Most settings are editable after publishing, but it is still important for instructors to review carefully 7511th Sakai Conference - June 15-17, 2010

76. Publishing Release assessment to students Copy made from pending assessment – pending assessment can be modified without affecting published version Can set Available Date in future -- useful for instructors who want to do bulk of setup early Can retract any time, making assessment unavailable to all students Can remove published assessment Can edit published assessment 7611th Sakai Conference - June 15-17, 2010

77. Taking an Assessment (Students) Take an Assessment section List determined by available, due, and retract dates, plus number of allowed submissions Can submit when done or save a work in progress Submitted Assessments section Provides submission confirmation with timestamp Depending on settings, students can review scores, feedback, grader’s comments, statistics 7711th Sakai Conference - June 15-17, 2010

78. Grading Ability to grade all questions for one student or grade all responses to one question Manual score adjustments can be made to individual questions or entire assessment Comments can be made for individual questions or entire assessment Ability to send scores to Gradebook Ability to export scores/responses to Excel Ability to calculate statistics and histograms 7811th Sakai Conference - June 15-17, 2010

79. Question Pools Organize questions into multiple subpools Move/copy/remove multiple questions at a time Author questions from within Pool Manager Import questions from IMS QTI-compliant XML file Randomly draw from pools for an assessment Question pools belong to an individual user, available across all user’s sites Ability to share question pools with other site members 7911th Sakai Conference - June 15-17, 2010

80. Assessment Types Create assessment “types” for different disciplines, different uses Hide complexity by selecting which settings are visible to the instructor. Pre-set defaults by making setting choices for the type (e.g., Self-study always has immediate feedback) 80 11th Sakai Conference - June 15-17, 2010

81. Assessment Creation Workflow July 200910th Sakai Conference - Boston, MA, U.S.A.

82. 11th Sakai Conference - June 15-17, 201082