1. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003 CLEAR 2003 Annual Conference Pushing the Boundaries of Regulation Practice Makes Perfect: Or Does It? Moderator I. Leon Smith, President & CEO Professional Examination Service Presenters Terry Kowalski, Business Development Promissor Steven S. Nettles, V. P., Research & Development Applied Measurement Professionals, Inc. Paul H. King, Vice President for Programs Professional Examination Service William A. Hatherill, CEO Federation of State Boards of Physical Therapy

2. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003 Opinions about Practice Tests Please complete the four pre-presentation ratings NOW. THANK YOU!

3. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003 No Amount of Practice Will Help!

4. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003 Presentations will Address the Following Questions What are the different ways practice tests are offered? What are candidate reactions to practice tests? Is there a relationship between practice test results and the “real thing”?

5. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003 Presentations will Address the Following Questions (continued) Who takes practice tests and when? How are practice tests marketed and positioned? Are there ethical concerns when a sponsor of a credentialing program offers practice tests?

6. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003 PRACTICE TESTS IN STATE REGULATORY PROGRAMS Terry Kowalski Business Development - Promissor

7. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003 On-Line Practice Tests Markets State Regulatory Potential licensure candidates Pre-licensing schools Professional Associations

8. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003 Why offer practice tests? For test-takers…. An experience that emulates the actual exam. For test sponsors… An opportunity to provide a valued customer service An opportunity for additional revenue

9. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003 State Regulatory Market Background Insurance and real estate Offered P&P practice tests for 10 years Migrated tests to new CBT platform January 2003

10. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003 Practice Tests Same format, content, structure Same length as actual exam Same rigorous test development process Same delivery platform

11. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003 Practice Tests The candidate gets Item level feedback on correct response Score reports with total correct/incorrect by content area A guide to content areas that need further study A chance to familiarize themselves with test format

12. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003 Practice Tests Candidates are advised repeatedly not to use the practice test as a predictive indicator of performance on the actual licensing exam, but as a guide to content areas needing further study and a way to familiarize themselves with the format.

13. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003 Survey Results - Dec ‘02 Percentage of candidates agreeing Instructions clear - 96% Easy to navigate – 95% Item formatting easy to read – 92% Easy to mark for review – 93% Review screen helpful – 87% Feels better prepared for licensing exam after P.T. – 89% Easy to check for correct answer – 92% Would pay more to take P.T. in actual test center –24%

14. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003 Presentation Follow-up Please give me your business card to receive an e-mail of the presentation materials -OR- Presentation materials will be posted on CLEAR’s website THANK YOU!

15. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003 Candidate Perceptions of Web-delivered Self Assessment Examinations Steven S. Nettles, EdD VP, Research and Development Applied Measurement Professionals, Inc.

16. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003 Survey Plan A 27 item survey was placed on the Web for approximately 2 months this summer. Candidates from the last 12 months were encouraged to visit the site and complete the survey. 188 respondents completed the survey.

17. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003 Gender

18. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003 Racial Group

19. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003

20. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003

21. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003

22. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003

23. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003

24. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003

25. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003

26. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003

27. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003

28. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003

29. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003

30. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003

31. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003

32. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003

33. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003

34. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003

35. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003

36. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003

37. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003 Summary Respondents appear to have positive perceptions about the Web-based SAEs addressed in this survey. The failure of many respondents to provide test score information allowed no conclusive results about the relationship between real and SAE exam scores.

38. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003 Presentation Follow-up Please give me your business card to receive an e-mail of the presentation materials -OR- Presentation materials will be posted on CLEAR’s website

39. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003 Practice Makes Perfect – Or Does It? Paul H. King Vice President for Programs Professional Examination Service

40. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003 Stated Purposes of Practice Test Provide exposure to format and content of certification test Identify possible areas of weakness so that further study efforts can be more focused Assess readiness to take certification test Emphasizes that specific score on the practice test does not guarantee the same or similar score on the certification test

41. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003 Practice vs. Actual Test Similarities Constructed to identical test specifications Actual pre-calibrated test questions from certification test Timing simulates that of certification test Passing score equated to same standard as certification test Feedback on test performance – overall and by domain or major area

42. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003 Practice vs. Actual Test Differences No eligibility requirements Shorter test – potential for more measurement error, especially for domains Uncontrolled test administration conditions Raw vs. scaled total scores Links to relevant study material provided

43. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003 Potential Advantages Accessibility – 24/7; anywhere with internet access; ease of registration; allowed interruptions Immediate feedback – performance and what to do next Narrows the disconnect between training courses and test expectations Significantly less cost than certification test to assess readiness ($45 vs. $215 /$355)

44. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003 Practice vs. Certification Candidates December 2002May 2003 LevelPractice Minus Repeaters Cert. Cands. Practice Minus Repeaters Cert. Cands. Level 1 (874 - 89) 785 (11%) 7031(754 - 57) 697(12%) 5696 Level 2 (776 - 99) 677 (16%) 4229(681 - 43) 638 (19%) 3320

45. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003 Timing and Number of Practice Tests

46. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003 Percentage Passing Practice Test by Month & Exam Level

47. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003 Consistency of Performance Practice (P) vs. Certification (C) LEVELTaking Both Pass or Fail Both Fail P Pass C Pass P Fail C Level 1 1217993 (81%) 166 (14%) 58 (5%) Level 2 1139887 (78%) 166 (15%) 86 (7%)

48. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003 “The Good Book” Standard 8.1: “Any information about test content and purposes that is available to any test taker prior to testing should be available to all test takers. Important information should be available free of charge and in accessible formats.” Comment: “…More detailed information, such as practice materials, is sometimes offered for a fee. Such offerings should be made to all test takers.” - Standards for Educational and Psychological Testing, AERA, APA, NCME (1999).

49. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003 Ethical Considerations Fee to take practice test – charging for what should be free? Necessary information vs. added value? Enabling early decision on readiness hence saving money? Recouping cost of developing actual items? Perceptions of performance – luring good performers into complacency? Misleading candidates as to readiness? Realistic guide to readiness and focus of future study? Education vs. certification – process or product? Test preparation or test readiness?

50. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003 Presentation Follow-up Please pick up a handout from this presentation.

51. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003 Practice makes Perfect: Or does it? William A. Hatherill, CEO Federation of State Boards of Physical Therapy

52. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003 Decision Making Process Context of the organization Ethical Issues? Mission oriented? Moving forward

53. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003 Context of the Organization Charged to develop alternate sources of revenue 90% of Federation revenue is from NPTE NPTE volume has declined by 30+% in last 5 years (19,500 to 14,000) Compromised old form NPTE Test Phobia = Level the Playing Field Transition from didactic learning to applied clinical application skills

54. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003 Ethics A branch of philosophy, is considered a normative science, because it is concerned with norms of human conduct, as distinguished from the formal sciences, such as mathematics and logic, and the empirical sciences, such as chemistry and physics

55. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003 Ethics Benchmark Study of Other Similar Organizations National Association of Boards of Pharmacy Association of Social Work Boards PSAT

56. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003 Mission The purpose of the Federation is to protect the public by providing service and leadership to promote safe and competent physical therapy practice.

57. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003 Mission Does the Mission support the development of a review guide? Does it protect the public? Does it harm the public? Does it promote safe and competent physical therapy practice?

58. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003 Ethics & Mission Should a review guide or practice exam be offered by the sponsor of a licensure examination? Appearance of a Conflict of Interest Concerns from stakeholders Design the Test = Design Study Guide Unfair Marketing Advantage: Assess to Applicants Internal Conflicts Marketing Resource Allocation

59. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003 Moving Forward: Research Student Needs and Perspective Faculty Perspective Financial Considerations

60. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003 Student Needs and Perspectives Student Need 87% plan to purchase some prep material for NPTE Pricing Strategy 65% will spend between $50 - $150 Marketing Strategy 67% plan to acquire one specific vendor’s product

61. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003 Student Needs and Perspectives Desirable Product Features Overall Quality Sample tests for review Rationales for correct answers Multiple, different exams with one purchase Immediate performance feedback

62. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003 Student Needs and Perspectives Positive features of Federation product Ability to stop and restart Reputation of the provider Similarity to look and feel of NPTE Inclusion of rationales

63. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003 Faculty Perspectives Mixed responses as to faculty’s role to assist students with NPTE prep 95% felt unauthorized copying and distribution would be a problem if site licenses were offered Faculty Training in Item Writing skills

64. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003 Financial Considerations Initial investment for product development $100,000 Estimated annual marketing expenses $5 per prospect/year or $50,000 Estimated market penetration for new product per year 10% year 1; 15% year 2; 20% year 3 Necessary break-even point Approximately 16%

65. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003 Product Performance: First Year? Approximately 5% Market Share $25,000 in revenue $35,000 in marketing expense $33,000 in start-up cost ($100K allocated over 3 years) $43,000 Loss

66. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003 Needs and Perspectives: Next Round Students who purchased Students who did not purchase Faculty

67. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003 Lessons Learned Refocus Product: Provide one time frame for one set price Online Review Guide becomes Practice Exam and Assessment Tool (PEAT) Reposition Product: Supplement for other review guides Instant Gratification – Allow purchase by Credit Card online Item Writing

68. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003 Study Considerations not followed Second Practice Exam Marketing Consider offering students promotional discounts for ordering at the same time as registering for the NPTE. (90% favor this option) Consider developing a “student distributor” program whereby selected students might receive a small cash payment ($25) Consider initiating discussions with existing Test Prep companies to package the FSBPT Exam with the Test Prep companies review guide

69. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003 Unintended Consequences Copyright Violation Exam Prep Company, requested eliminating PEAT in turn for shutting down the “Discussion Forum”

70. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003 PEAT Today Year 3 2003 Approximate 14% Market Share Projected $120,000 In Annual Revenue $ 0 In Marketing Expense Preparing to launch PTA PEAT

71. Presented at CLEAR’s 23rd Annual Conference Toronto, Ontario September, 2003 Opinions about Practice Tests Please complete the four post-presentation ratings NOW. THANK YOU FOR YOUR PARTICIPATION!