1. 1 Planning for Successful Student Learning Outcomes Assessment for the Voluntary System of Accountability with © 2009 ACT, Inc. All Rights Reserved. Presentation for Fitchburg State January 13, 2009

2. 2 Webinar Agenda CAAP Overview CAAP and the VSA CAAP Usage Strategies Plan for Success CAAP Resources

3. 3 Collegiate Assessment of Academic Proficiency

4. 4 The standardized assessment program from ACT that enables postsecondary institutions to assess, evaluate, and enhance the outcomes of their general education programs.

5. 5 Selected as one of the options for measuring student learning outcomes for the Voluntary System of Accountability.

6. 6 CAAP Benefits Systematically measure your students achievement levels and growth in your general education program

7. 7 CAAP Benefits Institution and individual student level data to help you identify opportunities to enhance student learning

8. 8 CAAP Benefits Help satisfy the need for greater accountability in higher education

9. 9 CAAP Benefits Help meet accreditation self-study requirements

10. 10 CAAP Benefits Help evaluate individual student readiness for upper division coursework

11. 11 CAAP Benefits Not to evaluate faculty. Not as a sole evaluation in high stakes environments.

12. 12 CAAP Features Modular format (6 content areas) Flexible administration Comprehensive reports Extensive national user norms Value-added measurement through linkage to COMPASS TM and the ACT ® test Identification of action items with Content Analysis Reports

13. 13 CAAP Test Modules Writing Essay Writing Skills Mathematics Reading Critical Thinking Science

14. 14 CAAP Administration Convenient in-class administration 40 minutes per test Pencil and paper Proctored Can be embedded in required freshman or capstone course 2-week test window Nine local questions can be added (except for VSA testing)

15. 15 CAAP Standard Reporting Package Institutional Summary Report Mean scores for each demographic and subscores in math, reading, & writing skills Student Score Reports – 2 copies One for institution and one for student Certificate of Achievement For students scoring at or above the national mean Student Roster Report Listing of each student with their CAAP results

16. 16 CAAP User Norms to

17. 17 CAAP User Norms Norms for each test module (and subscore) Broken out by: Institution type Year of students tested Institution ownership Three-year rolling norms Custom groupings available

18. 18 CAAP and the VSA Voluntary System of Accountability A joint initiative developed in partnership between NASULGC and AASCU Designed to help institutions Demonstrate accountability and stewardship to the public Measure educational outcomes to identify effective educational practices Assemble information that is accessible, understandable, and comparable Currently over 60 NASULGC/AASCU institutions use CAAP for their outcomes assessment

19. 19 CAAP and the VSA VSA College Portrait Consumer Information Student Experiences and Perceptions Student Learning Outcomes VSA participants measure critical thinking, analytical reasoning, and written communication CAAP Measures critical thinking and written communication using two test modules CRITICAL THINKING WRITING ESSAY

20. 20 VSA College Portrait Pilot http://www.voluntarysystem.org http://www.collegeportraits.org http://www.voluntarysystem.org http://www.collegeportraits.org VSA Web Site College Portrait data elements College Portrait prototype VSA Background/development VSA Enrollment

21. 21 CAAP and the VSA Student Learning Outcomes Freshman to Senior Learning Gains Critical Thinking Writing Essay The increase in learning on the performance task is _____ what would be expected at an institution with students of similar academic abilities. well above, above, below or well below what would be expected

22. 22 College Portrait Learning Outcomes Reporting College Portrait Prototype

23. 23 CAAP Writing Essay Two independent prompts 20 minutes each Essays scored at ACT by two trained raters ACT scoring rubric 1-6 http://www.act.org/caap/tests/holistic.html

24. 24 CAAP Writing Essay Formulating an assertion about a given issue Supporting that assertion with evidence appropriate to the issue, position taken, and a given audience Organizing and connecting major ideas Expressing those ideas in clear, effective language

25. 25 Writing Essay Sample Your college administration is considering whether or not there should be a physical education requirement for undergraduates. The administration has asked students for their views on the issue and has announced that its final decision will be based on how such a requirement would affect the overall educational mission of the college. Write a letter to the administration arguing whether or not there should be a physical education requirement for undergraduates at your college.

26. 26 CAAP Critical Thinking 32 items/4 passages Sample questions on-line Clarifying, analyzing, evaluating, and extending and argument Subject matterSocial Sciences, Natural Sciences, Humanities Formats include: Case Studies Dialogues Debates Editorials Statistical Arguments Experimental Results Overlapping Position

27. 27 CAAP and the VSA Sample Size ACT requires a minimum 200 freshman and 200 seniors tested on each module Test Dates Freshmen must be tested early in the fall semester Seniors must be tested within 6 months of graduation Both groups tested during the same academic year (September-May ) ACT (or SAT) Score Required Institutions must provide data file with CAAP tested students ACT or SAT scores – see CAAP VSA Guidelines for more details

28. 28 CAAP and the VSA Random selection Institution determines what constitutes a random representative sample Value-added VSA Model Cross-sectional Longitudinal Model –Consider the minimum number of records needed at the time of post-testing and then calculate back to establish the number for pre-test, and test 3-4 times that number

29. 29 CAAP Scoring Return materials to ACT for scoring and reporting Critical Thinking ACT proprietary scanning Writing Essay Two ACT-trained scorers Three-week turn-around time for standard reporting package

30. 30 CAAP VSA Report Provided upon completion of all freshmen and senior testing – late summer Provided upon receipt of your institution data file (with student matched ACT or SAT and CAAP scores) Report will include: Average scores for both freshmen and seniors on both test modules Student learning gains on each test module Average ACT (or SAT) for students tested

31. 31

32. 32 CAAP Administration for VSA Administer at least once every 3 years Report within 4 years of becoming VSA participant Updated at least every 3 years

33. 33 Additional CAAP Test Modules CAAP VSA Materials Order Form provides option for additional modules Administer to a different group of students than those tested on the VSA modules Option for local questions for only those who are not testing for VSA

34. 34 Any questions so far?

35. 35 Usage Strategies

36. 36 CAAP Usage Strategies Outcomes Measurement Value-Added Measurement Student Readiness Evaluation

37. 37 Outcomes Measurement

38. 38 Outcomes Measurement Establish benchmarks – (Scale scores, percentiles) Identify relative strengths and weaknesses – (Subscores, Content Analysis Report) Evaluate issues Propose solutions - set goals Implement changes Track results

39. 39 TEST MODULEC.A.R. DOMAINS Writing Skills -Usage/Mechanics -Rhetorical Skills Punctuation Grammar Sentence Structure Strategy Organization Style Mathematics -Basic Algebra -College Algebra Prealgebra Elementary Algebra Intermediate Algebra Coordinate Geometry College Algebra Trigonometry Reading - Arts/Literature - Social/Nat. Sciences Reasoning Skills Referring Skills Science Analysis Generalization Understanding Critical Thinking Analysis of Elements of Arguments Evaluation of Arguments Extension of Arguments Content Analysis Report

40. 40

41. 41 Content Analysis Report (Math Sample)

42. 42

43. 43 Value-Added Measurement

44. 44 Value-Added Measurement

45. 45 Value-Added Measurement Longitudinal (CAAP to CAAP) Longitudinal (Linkage) ACT to CAAP COMPASS to CAAP VSA Model – Cross-Sectional – Freshman to Senior Learning Gains

46. 46 Value-Added Measurement Longitudinal (CAAP to CAAP) Test cohort with CAAP upon program entry Test same cohort with CAAP upon completion of program Change in scores is a direct measure of value-added performance gain Evaluate results in terms of expectations Investigate opportunities for improvement

47. 47 Value-Added Measurement Longitudinal (ACT or COMPASS to CAAP) Use cohorts ACT or COMPASS scores as a benchmark Test cohort with CAAP upon completion of general education coursework Linkage report evaluates cohorts results in terms of expected gain Evaluate results in terms of expectations Investigate opportunities for improvement

48. 48 Value-Added Measurement Cross-sectional - VSA Model Test a cohort of incoming freshmen at the beginning of the fall term Test a similar group of seniors within 6 months of graduation Differences between the two group mean scores may indicate performance gain Challenge of this strategy is matching student characteristics across groups

49. 49 Student Evaluation

50. 50 Student Evaluation Determine student skill expectations Establish appropriate cut scores Propose appropriate remediation

51. 51 PLAN FOR SUCCESS

52. 52 Plan for Success #1 Confirm the fundamentals Review your institutional mission statement. Evaluate the objectives of your institutions general education program. Evaluate your institutional general education core curriculum.

53. 53 Plan for Success #2 Specify usage of results Who will review and analyze the data? How will the results be used to influence program evaluation and planning? Will results be tracked on a longitudinal basis?

54. 54 Plan for Success #3 Define population Specify characteristics of the student body that is the focus of the study. Specify characteristics of subgroups.

55. 55 Plan for Success #4 Sampling Test entire population or sample? How will sample be selected? How will subgroup sample(s) be selected? Is sample random? Is sample representative of student population?

56. 56 Plan for Success # 5 Faculty motivation Specify plans to communicate with faculty. Specify plans to enlist faculty support. Specify plans to share results with faculty.

57. 57 Plan for Success #6 Student motivation Specify plans to communicate with students. Specify plans to motivate students. Specify plans to share results with students.

58. 58 Plan for Success Submit CAAP-VSA Participation Agreement Confirm test modules, if additional* Confirm optional local questions* Confirm reporting requirements Confirm test period (two-week window) Confirm test venues Confirm students to be tested Confirm proctors Schedule training for proctors Order test materials at least one month prior to test dates *For non-VSA test modules only #7 CAAP Test Administration

59. 59 Critical Thinking - $13.50/student Writing Essay - $13.50/student –Includes Standard Reporting Package and CAAP VSA Report Additional CAAP Modules –Writing Essay - $13.50/student –Single Objective Test - $13.50/student 501+ - $12.70/student –Multiple Objective Tests - $ 13.50/student 501+ - $19.20/student CAAP Testing Costs

60. 60 CAAP RESOURCES

61. 61 CAAP Planning Workbook available online at http://www.act.org/caap/pdf/planguide.pdf http://www.act.org/caap/pdf/planguide.pdf Portions of Guide are available at http://www.act.org/caap/materials.html

62. 62 Portions of this can be found at http://www.act.org/caap/techhandbook http://www.act.org/caap/techhandbook

63. 63 Norms available on-line at http://www.act.org/caap/norms/ http://www.act.org/caap/norms/

64. 64 Available on-line at http://www.act.org/caap/pdf/userguide.pdf http://www.act.org/caap/pdf/userguide.pdf

65. 65 http://www.act.org/caap http://www.act.org/caap/vsa

66. 66 CAAP Overview CAAP and the VSA CAAP Usage Strategies Plan for Success CAAP Resources Recap

67. 67 ACT Contact Information ACT Postsecondary Assessment Services 500 ACT Drive P.O. Box 168 Iowa City, Iowa 52243-0168 Phone: 800-294-7027 Sandra Stewart Sandra.Stewart@act.org Direct # 319-337-1051 FAX: 319/337-1790 E-mail: outcomes@act.org