1. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Assessments Through The Learning Process October 2013 Eric Shepherd. Questionmark, CEO eric@questionmark.com www.questionmark.com eric@questionmark.com www.questionmark.com

2. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Thank you Slide 2

3. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  You have colored cards on your chairs/tables  Get one of each color, lettered A, B, C and D  Let’s practice holding up cards  During this presentation we’ll be using questions to show you the power of questions  Please answer quickly and don’t be embarrassed – we are here to learn! Using the cards Slide 3 A B C D

4. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  Learners generally prefer response cards to simple hand-raising  Response Cards Shown to Improve  classroom engagement  amount they learn  amount they retain -- Dr. Will Thalheimer, citing multiple examples of research on the use of response cards Research Suggests… Slide 4 A B C D

5. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  100 chests of tea  34 chests of tea  342 chests of tea  342 tea bags How much tea was dumped into Boston Harbor at the Boston Tea Party? Slide 5 B C A D

6. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  342 chests of tea  Over 46 tons of tea  Worth (at the time) over £10,000  £678,827 using the retail price index  £866,694 using the GDP deflator  £7,422,564 using average earnings  £10,025,165 using per capita GDP  £25,350,172 using the GDP Citation: Lawrence H. Officer, "Five Ways to Compute the Relative Value of a UK Pound Amount, 1830 - 2005" MeasuringWorth.Com, 2006. Answer: Slide 6 C

7. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Learning Without Questions Slide 7 Content

8. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Learning Benefits of Questions Slide 8 Content Questions

9. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  …expose poor presentation skills  …stimulate thinking and keep you awake  …frustrate our color blind friends  All of the above We are using answer cards to… Slide 9 B C A D

10. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  Provide Assessment Management Systems  To create, deliver and report on assessments  For exams, tests, quizzes, and surveys  More than 1,500 customers have purchased Questionmark  Corporate, Government, and Education markets  More than 14,000 authoring systems installed  Last month users created 40,000+ questions in Questionmark  Offices in US, UK, Canada, Germany, Belgium  Available in numerous languages Questionmark Slide 10

11. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Worldwide Practical Experience Slide 11

12. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  Alberta Health Services  Aluminerie Alouette Inc.  Apache Corporation  ATB Financial  British Columbia Hydro  Canada Border Services Agency - CBSA  CMLTO - College of Medical Laboratory Technologists of Ontario  Department of National Defence  Everest Colleges Canada Inc  ExxonMobil  Flakeboard Company Limited  Fortis BC Inc  Grant MacEwan University  International Competency Assessment Board  Invesco Canada Ltd.  Justice Institute of British Columbia  Loyalist College  Manitoba Hydro  Memorial University of Newfoundland  Moneris Solutions  National Association of Pharmacy Regulatory Authorities  Northern Ontario School of Medicine  Omnivex Corporation  Plains Midstream  Safety Codes Council  Scotiabank  Telus : Telus Supporting Units  University of Calgary : Faculty of Veterinary Medicine - University of Calgary  University of Manitoba - Faculty of Medicine  University of Ottawa  University of Ottawa - Faculty of Medicine  University of Saskatchewan Sample List of Canadian Customers Slide 12

13. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Who Else Uses Questionmark? Slide 13 Financial Services Pharma / Health High Tech / Manufacturing TelcomEnergy / Utilities Public SectorEducationTransportation Prof. Services Retail

14. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  …brain dump; tell me everything  …I don’t know; say something interesting  …hurry up; I am only here for the networking  …why so many questions? What would you like from this session? Slide 14 B C A D

15. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Things that Drive What-To-Do Slide 15 Increase Revenue = € £ $ ¥ Reduce Costs = € £ $ ¥ Reduce Time to Market

16. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Practical Limitations for Planning Slide 16 Money = € £ $ ¥ How Long Do We Have? People or Talent Resources Laws Health & Safety

17. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  Increase revenue, time-to-market, laws  Increase revenue, reduce costs, safety  Increase revenue, reduce costs, time-to-market  Money, time-to-market, laws and talent What are the business drivers that drive the use of learning and assessments? Slide 17 B C A D

18. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  Money, health and safety  Money, laws, health and safety  Money, laws, resources, and safety  Money, laws, time-to-market, health and safety and, access to talent and resources What are the practical limitations of using assessments? Slide 18 B C A D

19. Strategy Tactics & Plans Job Roles Resources Tasks Knowledge, Skills, Abilities Tactics & Plans Job Roles Resources Tasks Knowledge, Skills, Abilities Job Roles Resources Tasks Knowledge, Skills, Abilities Tactics & Plans Job Roles Resources Tasks Knowledge, Skills, Abilities Job Roles Resources Tasks Knowledge, Skills, Abilities Slide 19

20. Strategy Tactics & Plans Job Roles Resources Tasks Knowledge, Skills, Abilities Tactics & Plans Job Roles Resources Tasks Knowledge, Skills, Abilities Job Roles Resources Tasks Knowledge, Skills, Abilities Tactics & Plans Job Roles Resources Tasks Knowledge, Skills, Abilities Job Roles Resources Tasks Knowledge, Skills, Abilities Slide 20 Knowledge, Skills and Abilities Required How To Do It What Will They Do What Will They UseWho Will Do It What To Do

21. Slide 21 Knowledge, Skills and Abilities Required How To Do It What Will They Do What Will They UseWho Will Do It What To Do Competencies

22. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Mental Check to Confirm a Competency 22 “Given What Do What” Time Tools Materials Circumstances Data Given What Do using Competencies Do What Very Well Well Poorly Perform

23. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Knowledge (a cognitive Skill) Memory is a cognitive skills that provides ability to recall knowledge: “Knowledge How” “Knowledge To” Above Knowledge Skills Cognitive skills above recalling knowledge (Comprehension, Application, Analysis, Synthesis and Evaluation) Observable Skills - Motor Skills i.e. physical movement/coordination - Inter-personal skills Attitudes and Behaviors Motivation Ability to achieve goals Team fit Ability to Do/Perform “Competency” Competency is the “Ability To Do” (or “Ability To Perform”) and requires…. 23 Cognitive Skills

24. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  Results Oriented  Ability to meet and exceed goals, make difficult decisions, act on opportunities that create value for the company.  Demonstrate ability to exceed stated job responsibilities, take informed risks, make principled decisions, and faces high-stakes issues head on.  Create a work environment where others feel it is safe to take risks.  Customer Results  Consistently meets Customer Needs. Provides solutions for external and internal customers that meet and/or exceeds customer expectations. Identifies opportunities that will benefit the customer and works in partnership with the customer to drive the business forward. Example - Attitudes and Behaviors Slide 24

25. Slide 25 Competencies Required Knowledge, Skills and Abilities Required

26. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  How do we know the Knowledge, Skills and Abilities required?  Job Task Analysis  Survey people who do the job  MoSCoW analysis  Knowledge, Skills and Abilities that we:  M ust have  S hould have  C ould have  W ould be nice to have Slide 26 Knowledge, Skills and Abilities Required Assessment

27. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Slide 27 Skill Survey, Quiz, or Test, What We Need Knowledge, Skills and Abilities Required Knowledge Skills Abilities What We Have Knowledge Skills Abilities Assessment

28. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Slide 28 What We Need Knowledge, Skills and Abilities Required Knowledge Abilities What We Have Knowledge Skills Abilities - = Skills Topics To Train Knowledge Skills Abilities

29. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  Helps us survey skills  Helps us know what actions to take  Know what knowledge we have  Determine qualified candidates Which is the best answer to the question… Why are skill surveys useful? Slide 29 B C A D

30. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Learning Assessment Thru Learning Process Evidence Accumulated Prepare For Learning In/As Learning Assessments Of-Learning Assessments Mentoring Instructor Collaborative & E-learning Work Place Slide 30

31. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Learning Assessment Thru Learning Process Evidence Accumulated Prepare For Learning In/As Learning Assessments Of Learning Assessments Job Task Analysis Course Evaluations Needs Analysis Mentoring Instructor Collaborative & E-learning Work Place Slow Forgetting Curve Slide 31

32. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Learning Pre-Learning Assessments Evidence Accumulated Mentoring Instructor Collaborative & E-learning Work Place Purpose: Create intrigue Route to course Inform instructors Establish benchmarks Purpose: Create intrigue Route to course Inform instructors Establish benchmarks Prepare For Learning In/As Learning Assessments Of Learning Assessments Slide 32

33. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Learning In-Learning Assessments Evidence Accumulated Mentoring Instructor Collaborative & E-learning Work Place Purpose: Encourages focus on learning Provide instant feedback to instructor Stimulate and strengthen memory Correct misconceptions Purpose: Encourages focus on learning Provide instant feedback to instructor Stimulate and strengthen memory Correct misconceptions Prepare For Learning In/As Learning Assessments Of Learning Assessments Slide 33

34. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. In a Perfect World Slide 34

35. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Slide 35 In the Actual World They don’t get it All!

36. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Learning Post-Learning Assessments Evidence Accumulated Mentoring Instructor Collaborative & E-learning Work Place Purpose: Was learning environment okay? Knowledge transfer occurred? Reduce Forgetting Curve Certifications Purpose: Was learning environment okay? Knowledge transfer occurred? Reduce Forgetting Curve Certifications Prepare For Learning In/As Learning Assessments Of Learning Assessments Slide 36

37. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Learning Curve Slide 37 Ouch! The Forgetting Curve Time Knowledge/Skills Learning Experience Questions Purpose: Content repetition Memory retrieval practice Strengthens memory recall Purpose: Content repetition Memory retrieval practice Strengthens memory recall

38. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. In the Actual World They Forget It! Slide 38

39. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Slow the Forgetting Curve Slide 39 Time Knowledge/Skills Learning Experience Questions Purpose: Memory retrieval practice Strengthens memory recall Reduce Forgetting Curve Purpose: Memory retrieval practice Strengthens memory recall Reduce Forgetting Curve

40. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Questions Thru the Learning Process Slide 40 Time Knowledge/Skills Questions Create Intrigue Strengthen Memory Reduce Forgetting

41. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  30 minutes of instruction and 30 minutes of questioning  30 minutes of instruction and 30 minutes of review and study time  Both will produce similar results Which of the following combination of instructional activities will increase learning and reduce forgetting? Question borrowed from the work of Dr. Will Thalheimer Slide 41 B C A

42. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Remember what is learned Retrieval Practice Develop misconceptions Feedback Forget what’s been learned Repetitions, Feedback, Retrieval Practice, Spacing Don’t absorb everything Repetitions Attention wanders Pre-questions; directs attention Why Don’t They Get It? Slide 42

43. Dr. Will Thalheimer research: Learning Benefits from Questions Learning Benefit MinMax Pre-questions (to focus attention)5%40% Repetitions Initial Subsequent 30% 15% 110% 40% Feedback15%50% Retrieval Practice30%100% Questions Spaced Over Time5%40% 100%380% Slide 43

44. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  30 minutes of instruction and 30 minutes of questioning  30 minutes of instruction and 30 minutes of review and study time  Both will produce similar results Which of the following combination of instructional activities will increase learning and reduce forgetting? Question borrowed from the work of Dr. Will Thalheimer Slide 44 B C A

45. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Tests and Exams That Measure Slide 45

46. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  Regulatory  Governments  Health and Safety  Examples  Driving Tests  Pilots Exams  Doctor Certification  Non-Regulatory, certification  Vendor Specific  Microsoft  Oracle  SAP  Vendor Neutral  ECDL/ICDL Tests and Exams That Measure Slide 46  Regulatory Compliance Demonstrating compliance to government  Health/Safety/Environmental  Financial Services

47. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Norm Referenced (pass/fail determined after test) Criterion Referenced (pass/fail score determined before test) Setting a Pass/Fail score? Slide 47 250 people Only 50 people can pass 250 people 1 st group of people 65 pass 2 nd group of people 45 pass 3 rd group of People 75 pass

48. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. How each type of assessment measures the participant… Norm Referenced Participant is compared to others in a group? Typically used for standardized college placement tests like the SAT, ‘competition’ tests (e.g. advancement test for limited number of openings) Pass/Fail can only be determined after the assessment Criterion Referenced Participant is compared to a benchmark? Typically used for certification tests, licensing exams, end of course exams, etc. Typically Pass/Fail score is determined before the assessment Slide 48

49. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  Norm referenced  Criterion referenced  Neither of A nor B  Both A and B Are Regulatory Compliance tests Norm or Criterion referenced? B C A D Slide 49

50. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. How Do I Know It’s a Good Test Slide 50

51. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Choices # of Respondent s % Selected Difficulty (also p-value) 0 = Too difficult 1 = Too easy Participant Mean Outcome Discrimin- ation Outcome Correlation A B C D Item Statistics - Choices Slide 51

52. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Choices # of Respondent s % Selected Difficulty (also p-value) 0 = Too difficult 1 = Too easy Participant Mean Outcome Discrimin- ation Outcome Correlation A 21 B 52 C 323 D 112 Item Statistics - # of respondents Slide 52

53. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Choices # of Respondent s % Selected Difficulty (also p-value) 0 = Too difficult 1 = Too easy Participant Mean Outcome Discrimin- ation Outcome Correlation A 214% B 5210% C 32364% D 11222% Item Statistics - % of respondents Slide 53

54. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Choices # of Respondent s % Selected Difficulty (also p-value) 0 = Too difficult 1 = Too easy Participant Mean Outcome Discrimin- ation Outcome Correlation A 214% B 5210% C 32364%.64 D 11222% Item Statistics - Difficulty (also P-value) Slide 54

55. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Choices # of Respondent s % Selected Difficulty (also p-value) 0 = Too difficult 1 = Too easy Participant Mean Outcome Discrimin- ation Outcome Correlation A 214%36% B 5210%47% C 32364%.6466% D 11222%52% Item Statistics - Participant Mean Slide 55

56. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Choices # of Respondent s % Selected Difficulty (also p-value) 0 = Too difficult 1 = Too easy Participant Mean Outcome Discrimin- ation Outcome Correlation A 214%36% B 5210%47% C 32364%.6466%0.49 D 11222%52% Item Statistics - Outcome Discrimination Slide 56

57. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Choices # of Respondent s % Selected Difficulty (also p-value) 0 = Too difficult 1 = Too easy Participant Mean Outcome Discrimin- ation Outcome Correlation A 214%36%-0.32 B 5210%47%-0.22 C 32364%.6466%0.490.24 D 11222%52%-0.15 Item Statistics - Outcome Correlation Slide 57

58. Cronbach’s Alpha  Internal consistency test reliability: A measure of how well the assessment questions relate to one another to measure the same topic  Crucial information for legal defensibility – is the test reliably measuring what is was designed to measure?  Speaks to the quality of assessment questions: Are they of high statistical quality? Slide 58

59. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  Skew  Mode  Kurtosis  Mean (arithmetic)  Median  Mode  Standard Deviation  Variance  Standard Error of measurement  Standard Error of the Mean Other Measures Slide 59

60. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  Informed Judgment Poll SMEs to determine criteria for passing (no. of correct answers, no. of attempts) Not recommended for high-stakes assessments  Borderline Group Method Candidates are interviewed prior to an exam and borderline cases are identified After the exam, the scores of borderline candidates can be used to determine the cut score  Contrasting Group Compare SME to others  Angoff Method Poll SMEs Consider marginal candidates and the probability of getting specific questions right (0-100%) Make a mathematical calculation to determine cut score Setting Cut Scores for Criterion Reference Tests Slide 60

61. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. SME ASME BSME CTotal Q1 75% 75.00% Q2 70%80% 76.67% Q3 65%75%70%70.00% Q4 60%85%90%78.33% Q5 80% 85%81.67% Q6 80% 80.00% Q7 75%80%75%76.67% Q8 65%90%65%73.33% Q….. 75%80%75%76.67% Q50 65%85%65%71.67% Totals 71%81%76% Angoff Method What is the % chance that a borderline candidate will get question right? Slide 61

62. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Emerging Trends and Practices

63. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Mobile Delivery of Assessments Observational assessments Mobile exam rooms Deliver to the device in your hand Slide 63

64. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  QR = Quick Response  Type of barcode  Allows more data than traditional barcode  Used in marketing, advertising, entertainment  Increasing use in e-learning/m-learning What are QR Codes? Slide 64 Can encode 4000+ characters of data!

65. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. QR Codes are read by Apps Slide 65 www.url.com/xxx iPhone Redlaser Neoreader ScanLife Android Google Goggles QR Droid Scanlife Windows Mobile QR Reader BeeTag Quickmark Blackberry ScanLife BeeTag QR Code Scanner

66. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. BooksPowerPointSignage Enables quick launching of quizzes and course evaluations Slide 66 Benefits Ease of access for increased response rates Demographic data easily captured Better data accuracy Easier analysis of survey results by demographic Benefits Ease of access for increased response rates Demographic data easily captured Better data accuracy Easier analysis of survey results by demographic

67. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Embed in a Wiki Slide 67 Embedded in a Wiki

68. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Embedding in Blogs, Portals, Wiki’s For LearnersFor Instructors  Focus the learners’ attention  Practice memory recall  Identify what was learned  Provide feedback to assist the learner to:  Correct misconceptions  Instill confidence  Point to relevant material  Easily use assessments throughout the learning process  Provide seamless user experience  Centralised management of assessment content and results Slide 68

69. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Questionmark Assessments Management System

70. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Assessment Management System Slide 70 Software-as-a- Service For OnPremise Deployment

71. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  Criterion-Referenced Test Development  Sharon A. Shrock & William C. Coscarelli  ISBN: 1-890289-09-4  Work-Learning Research white papers  Will Thalheimer  www.work-learning.com www.work-learning.com  Tests That Work  Odin Westgaard  ISBN 0-7879-4596-X  Performance-Based Certification: How to design a valid, defensible, cost-effective program  Judith Hale, Ph.D., CPT  ISBN: 0787946400 Recommended Books Slide 71

72. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Assessments through the Learning Process Security of Questionmark’s OnDemand Service Alignment, Impact and Measurement with the A-model Embedded Assessments: Building Knowledge Checks, Surveys and Other Assessments into Learning Materials How Questionmark Aids ISO 10667 Compliance Recommended White Papers Slide 72 https://www.questionmark.com/whitepapers

73. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Keep in Touch! Slide 73 blog.questionmark.com www.questionmark.com/newsletter twitter.com/questionmark linkedin.comlinkedin.com Look for Questionmark under “Groups” and join youtube.com/questionmarkvideo flickr.com/questionmarkpictures facebook.comfacebook.com Become a “fan” for news, updates and more

74. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Session Evaluation Slide 74 https://www.questionmark.com/go/709015

75. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. What questions do you have? Eric Shepherd. Questionmark, CEO eric@Questionmark.com www.questionmark.com