1. Technology Enhanced Learning Test and Quizzes in Sulis Facilitator: Angelica Risquez

2. Index  Benefits of T&Q  T&Q key features  T&Q homepage  Question Pools  New assessment: Quick create  New assessment: Slow create  Changing and previewing assessments  Publishing assessments  Working with scores and answers  Retracting a published assessment  Exporting/importing assessments  Assessment types

3. Benefits of online formative T&Q  Learners develop understanding progressively and have opportunities for feedback and reinforcement  Allows for scalable formative testing and feedback provision that would not be possible without automatisation  Available anytime, anywhere  You can delegate on students to create questions  If well used, may reduce teachers’ workload and increase students’ responsibility for their own learning  It may increase participation in face-to-face sessions Useful case: http://www.elearning.ac.uk/effprac/html/casestudy5.htmhttp://www.elearning.ac.uk/effprac/html/casestudy5.htm

4. Test & Quizzes in Sulis: Key Features  Automatic grading: True/false, matching, multiple choice, fill- in-the-blank, numeric response, survey (grades automatically transferred to the Gradebook tool)  No automatic grading: short answer/essay and file upload  Question Pools to re-use questions and share them  Import assessments from other sites you own and import previously created tests/quizzes/surveys (QTI -compliant files)  Create questions with online forms (standard approach) or single text box (quick create approach)  Customisation: location for taking a test, IP addresses, availability of test, time limits, number of submissions allowed, and late submissions  Attachments (by uploading local files, specifying URLs, and selecting files from Resources) to any question.  Templates (or “assessment types”) to speed customisation

5. T&Q Homepage (instructor view) 1. Core Assessments (unpublished) and Published ass’s. 2.You can return to this screen with the reset arrow ! 3.New assessment options 1.Quick Create 2.Create (recommended) 3.Import (xml file)

6. Question Pools: create  In the T&Q tool, click the Question Pools button. In the bar near the top of the screen, click Add New Pool  You can also import an assessment as a question pool, after exporting an assessment from another site: In the T&Q home page (instructor view) click Question Pools. In the bar above the list of pools, click Import and follow instructions to import your assessment  You can also create subpools, move and delete pools  To add questions to a pool or subpool, click the “Add” link (right hand side of the screen, opposite to “Questions”), and follow steps  To Copy a question from a Question Pool to an assessment, you use the pulldown list on the Questions screen (see slide 6) to select the type of question to add. Choose the option “ Copy from Question Pool ”

7. New Assessment: “Quick” Create Only for multiple choice (single or multiple answer), true/false, survey and fill-in-the-blank questions. 1.Name the assessment and click “Quick Create” (don’t choose assessment type for now). Sulis will display the Questions screen 2.Click the links in for each type of question to read specific instruction and examples 3.Review each question on the Validate Assessment/Question Pool screen 4.Click “create assessment”: Sulis displays the T&Q home page and the new assessment listed under Core Assessment (you can create a Question Pool instead) When you finish with the process, your new quiz will appear under “Pending assessments”. Note: the quiz is not yet available to students!

8. New Assessment: “Slow” Create To be used for other type of questions, to create parts (needed for random assessments), and to reuse questions from Question Pools  Name the assessment and click “Create” (don’t choose assessment type for now). From menu “Add question”, add questions one by one or bring from Question Pool When you finish with the process, your new quiz will appear under “Pending assessments”. Note: the quiz is not yet available to students!

9. Create questions  You can either create new questions in your assessment or add questions from a Question Pool.  You may add question to any unpublished assessment.  Under the appropriate part, select the type of question you want to add with the “ Add Question ” pull down menu  Answer point value : If questions have two correct answers, the points for each correct question accumulate (e.g. 10+10=20). For ungraded questions enter 0 (zero). Incorrect answers are worth negative points.  Using the Assign to Part pulldown list, select the part (that is, the section) of the assessment to which you want to assign this question  To assign a question to a question pool, select the name of the desired pool from the Assign to question pool pulldown list.  To provide question-level feedback enter text in the boxes under “ Correct/Incorrect Answer Feedback.” (voluntary)  You can require participants to provide a reasons for choosing an answer under Required Rationale New Assessment: “Slow” Create (2)

10. Create questions (continued) Multiple choice questions  To indicate the correct response, click the Correct Answer radio button for that response (multiple correct answers allowed)  To display the possible answers in random order, und Randomize Answers Survey choice questions  You do not have the option of randomizing answers or requiring a rationale. Short answer/essay questions  Answer point value: enter the point value you want to assign to this question. For an ungraded question, enter 0 (zero). (MANUALLY GRADED) Fill the blank questions  Place curly braces { } around words requiring a blank For example: {gravity}.  If more than one response is correct for a given blank, or to allow alternate spellings, use commas to separate the alternate responses. For example: Darth Vader actually was Luke Skywalker's {father, dad, poppa, pater, pa, daddy}.  For questions with more than one blank, each blank must contain a unique response (mutually exclusive). For example: The colors of the Irish flag are {orange, white, green}, {orange, white, blue}, and {red, white, blue}. To answer correctly, the participant must respond with some combination of orange, white, and green. New Assessment: “Slow” Create (3)

11. Create questions (continued) Numeric Questions  Place curly braces { } around numbers that require a blank (for example: 3*3= {9} y 2+2= {4} requires a blank before the digits 9 and 4).  If a range of values is acceptable, insert a pipe between the highest and lowest acceptable values (for example: 5|8 accepts answers of 5, 6, 7, and 8). Matching Questions  In the Choice box, enter a choice. In the Match box, enter the correct match for the choice you specified.  Click the Save Pairing button. Sulis saves the pair you have created and displays it under the Add/Edit Pairing and Optional Feedback header File Upload Questions  In the Answer Point Value box, enter the point value you want to assign to this question. For an ungraded question, enter 0 (zero). (MANUALLY GRADED) You may change questions in any unpublished assessment. Display the Questions screen for the assessment, find the question you want to change and click the Edit option for it. New Assessment: “Slow” Create (4)

12. To create a random quiz from a question pool, you need to edit the settings of a part of the test. The part should be empty of questions, if it is now, you need to create a new part (click “Add Part”). You can use the Default part as long as no questions have been added yet: click “Edit” for Default Part and under Type, choose “Random draw from question pool” (choose pool). New Assessment: “Slow” Create (5)

13. Changing and Previewing Assessments  You may change the settings, parts, and questions for an assessment at any time BEFORE it is published (unpublished assessments are listed under Core Assessments on the T&Q home page)  After an assessment has been published, you may change only the following attributes:  Delivery, Due, and Retract dates  Gradebook options  Feedback Delivery options  Feedback Components options  To preview an assessment before publishing it, display the Questions screen for the assessment and click the Preview Assessment link  By default, only students or site members with the appropriate permissions may submit an assessment. Instructors and site organizers may preview an assessment but may not submit an assessment.

14. Publishing Assessments When you publish an assessment, you make it available to your site's participants. After the assessment has been published, its contents CANNOT be changed. 1.In the Core Assessments section, click the Settings link under the title of the assessment you want to publish. 2.Work through the sections by opening them and choosing the appropriate options for this assessment.  To open all sections, click the Open link under the title of the screen.  To close all sections, click the Close link under the title of the screen.  To open a section, click the triangle icon ( ► ) for it.  To close a section, click its triangle icon ( ▼ ) again.

15. 3.Make sure all settings are configured as you want them. Be sure to click Save Settings and Publish. If you click Save Settings, the quiz will NOT be published. 4.Sulis displays a confirmation screen. Click Publish. Publishing Assessments (2)

16. Working with Scores and Answers In the Published Assessments section, find the assessment you want and click the Scores link under its name. Sulis displays the Total Scores screen You entered this info Sulis generated this total  To adjust a student's total score and add an overall comment, enter an adjustment to the score in the Adj box. To add a comment, enter it in the Comment box. After entering changes for all appropriate participants, click Update. Sulis calculates and displays the final scores.

17. Working with Scores and Answers (2)  To see the submission status of an assessment for all participants, click the Submission Status button  If the Date entry is blank for a student, s/he has not submitted the assessment.  You may contact any student who is expected to submit an assessment by sending him or her an email (Click the Email link under the student’s name)  To allow one or more students to retake an assessment, click the Allow retake? link under the student’s name

18. Working with Scores and Answers (3)  To see individual questions and all responses to it to quickly compare all participants' answers to any question, click the Questions option  The Question section displays the question, the number of points allotted to it, all possible answers, and the correct answer(s)  The Responses section lists all participants by user name, date the assessment was submitted, score on the question, student's response, and a box for comments. You may view responses, view the highest scoring submission for each student or all submissions, and view each response. You can also download all files uploaded by students in response to file upload questions  You can also adjust a student's score and add a comment for one or more individual questions

19. Working with Scores and Answers (4)  To see statistics for the assessment as a whole and for each question and its responses, click the Statistics option  The Total Score section shows the total number of submissions and the total score possible on the assessment, along with statistical information such as the mean, median, mode, and range of the scores for the assessment as a whole.  Below, Sulis displays statistics for each question and its possible answers (with a check mark for the correct answer ), a histogram showing the number of students who selected each response, the total number of responses to the question and the percentage of participants who gave the correct response.

20.  You can also see an item analysis for each question Working with Scores and Answers (5)  … and also download all questions and scores into Excel

21. Retracting a Published Assessment To remove a test from students’ view, you need to retract it.  After you have retracted a published assessment, you can change the settings, parts, and questions in it, as needed, and then republish the assessment.  If you change a retracted assessment and republish it, the newly published assessment will have a different URL 1.Find the assessment in the Published Assessments section 2.Click the title of the assessment and click Edit to retract

22. Exporting Assessments  You may want to export an assessment to upload in another course or share with a colleague 1.In the Pending Assessments section and click the Export link under the title of the quiz, then select QTI v 1.2 and click Export 2.Your browser displays a window with downloading instructions (click Download ; after the browser displays the XML file, click File / Save Page As, enter a name for the new file, and save it in your pc)

23. Importing Assessments You can share quizzes among instructors with the export/import features 1.Lecturer A follows instructions to export quiz (previous slide), and sends the file to Lecturer B 2.Lecturer B goes to T&Q home page (instructor view). In the New Assessment section, clicks the Import Assessment button and follows instructions. Note: you can use this to reuse quizzes from another course, but using Question Pools is a preferred solution

24. Assessment Types Assessment Types are collections of settings (no questions) that are useful if you are going to have publish several of the same type of test  Predefined types ( choose existing assessment type list). You can edit them if you want. Please check their settings carefully before choosing one! (e.g. grading options?)  You may create your own: 1.Click Assessment Types button 2.Type a name for new assessment type, click create. 3.Choose appropriate options in each section (click ► to open each section). Be careful with “can edit default value”

25. Questions? Angélica Risquez Technology Enhanced Learning Advisor Centre for Teaching and Learning Millstream Courtyard University of Limerick Limerick, Ireland [T] +353-61-202 580 [F] +353-61-338 044 [E] angelica.risquez@ul.ie [W] www.ul.ie/ctl