1. Copyright © 2004 Educational Testing Service Listening. Learning. Leading. David G. Payne, PhD Senior Executive Director, Client Relations Higher Education Division Patricia Connor Director of Higher Education Initiatives Educational Testing Service SUNY Conference Syracuse, NY April 27-28, 2005 SUNY General Education Learning Outcomes : Strengthened Campus-based Assessment

2. Copyright © 2004 Educational Testing Service Listening. Learning. Leading. 2 Overview  ETS  Who we are  What we bring to the table  Academic Profile and SUNY General Education Learning Outcomes Assessment  Customized SUNY General Education Learning Outcomes Assessment Modules  Key Features of the Customized SUNY Assessments

3. Copyright © 2004 Educational Testing Service Listening. Learning. Leading. 3 Educational Testing Service  Founded in 1947 by  American Council on Education  Carnegie Foundation for the Advancement of Teaching  College Entrance Examination Board.  Today, as the world’s premier educational measurement and research organization, ETS develops and annually administers more than 11 million tests.  As a Not for Profit organization, ETS has a specific Mission.

4. Copyright © 2004 Educational Testing Service Listening. Learning. Leading. 4 The ETS Mission  To help advance quality and equity in education by providing fair and valid assessments, research and related services.  Our products and services measure knowledge and skills, promote learning and performance, and support education and professional development for all people worldwide.

5. Copyright © 2004 Educational Testing Service Listening. Learning. Leading. 5 ETS’s Largest Exams and Annual Volumes  TOEFL – 780,000  GRE – 500,000  GMAT – 229,000  SAT I – 3.1 million  AP – 1.4 million  PRAXIS – 770,000

6. Copyright © 2004 Educational Testing Service Listening. Learning. Leading. 6 What else does ETS Provide?  Products and services to support Institutional and Curriculum Development for Higher Education  Academic Profile: General Education Test  Major Field Tests: Outcomes Assessments in 16 fields of study  ICT Literacy: Information and Communication Technology Assessment  Criterion: On-line Writing Evaluation  Free or Low-cost Test Preparation Materials for all Examinees  Mission-related services

7. Copyright © 2004 Educational Testing Service Listening. Learning. Leading. Academic Profile and SUNY Learning Outcomes

8. Copyright © 2004 Educational Testing Service Listening. Learning. Leading. 8 Academic Profile Test of general education that measures reading, writing, math and critical thinking in the context of the humanities, social sciences and natural sciences.

9. Copyright © 2004 Educational Testing Service Listening. Learning. Leading. 9 Academic Profile Skill Scores (Norm-Referenced Scaled Scores)  4 Skill Areas in Standard Form  Critical Thinking  Reading  Writing  Mathematics  Optional 40 minute Essay

10. Copyright © 2004 Educational Testing Service Listening. Learning. Leading. 10 Academic Profile: Standard Form  Covers all four skill areas in one test  Optional Essay  108 multiple choice questions  120 minutes  Generates total and content area scores for both individuals and groups  Provides demographic data  Allows for locally-written questions

11. Copyright © 2004 Educational Testing Service Listening. Learning. Leading. 11 Academic Profile and SUNY Learning Outcomes  Mathematics  5 specific skill areas  Basic Written Communication  3 specific skill areas  Critical Thinking  2 specific skill areas

12. Copyright © 2004 Educational Testing Service Listening. Learning. Leading. 12 Mapping Academic Profile to SUNY Learning Outcomes Academic Profile Standard Form 108 questions Reading Writing Mathematics Critical Thinking Academic Profile Optional Essay 40 minutes Range of Topics Online Delivery Holistic Scoring SUNY General Education Assessment Modules 40 minutes 36 Questions OR 40 minutes Essay Paper and Pencil or Online Delivery Flexible Score Reporting

13. Copyright © 2004 Educational Testing Service Listening. Learning. Leading. 13 Academic Profile and SUNY Learning Outcomes Mathematics 4 specific skill areas Basic Written Communication 3 specific skill areas Critical Thinking 2 specific skill areas

14. Copyright © 2004 Educational Testing Service Listening. Learning. Leading. 14 Academic Profile for SUNY General Education Learning Outcomes Assessment  Tests Available January 2006  Report Formats  Paper  Electronic  Normative data  National Norms  SUNY Campuses (depending on volumes)

15. Copyright © 2004 Educational Testing Service Listening. Learning. Leading. 15 Academic Profile for SUNY  Advantages  In use at many institutions nationwide  Nationally normed  Valid  Reliable  Paper and on-line versions  Web-based score reporting  Group and subgroup reports  Disadvantages  Not precisely mapped to SUNY Learning Outcomes

16. Copyright © 2004 Educational Testing Service Listening. Learning. Leading. Customized SUNY Learning Outcomes Assessment

17. Copyright © 2004 Educational Testing Service Listening. Learning. Leading. 17 Four Guiding Principles for Customized SUNY Assessments 1.Customized solution to match SUNY General Education Learning Outcomes  Subscores for as many skill/competencies as possible 2.Ease and efficiency of administration and reporting  Students  Faculty  Administrators 3.Simple, clear score reports that mirror the SUNY outcomes 4.ETS quality and experience applied to the SUNY needs

18. Copyright © 2004 Educational Testing Service Listening. Learning. Leading. 18 SUNY Mathematics Module: Skills Assessed  Interpret and draw inferences from mathematical models such as formulas, graphs, tables and schematics  Represent mathematical information symbolically, visually, numerically and verbally  Use arithmetical, algebraic, geometric and statistical methods to solve problems  Estimate and check mathematical results for reasonableness, determine alternatives and select optimal results  Recognize the limits of mathematical and statistical methods

19. Copyright © 2004 Educational Testing Service Listening. Learning. Leading. 19 Customized SUNY Mathematics Module: Key Features  Careful Mapping of Skills to Assessment  Format  36 Multiple Choice Items  Paper and Pencil Format  40 Minutes  Scores Reported  Individual Student Overall Scores  4 Subscores

20. Copyright © 2004 Educational Testing Service Listening. Learning. Leading. 20 Sample Mathematics Question Pat purchased 4 large picture frames for $20.00 each, 100 medium picture frames for $10.00 each, and 8 small picture frames for $5.00 each. What was the average (arithmetic mean) price per picture per frame? A.$7.50 B.10.00* C.11.67 D.15.00

21. Copyright © 2004 Educational Testing Service Listening. Learning. Leading. 21 According to the information in the table, what is the range of the cost per night for a room for 1 person? A.$25 B.$50 C.$62 D.$79* Sample Mathematics Question

22. Copyright © 2004 Educational Testing Service Listening. Learning. Leading. 22 Customized SUNY Basic Written Communication Module: Skills Assessed  Produce coherent texts within common college-level written forms  Demonstrate the ability to revise and improve such texts  Research a topic, develop an argument, and organize supporting details

23. Copyright © 2004 Educational Testing Service Listening. Learning. Leading. 23 Customized SUNY Basic Written Communication Module: Key Features  Careful Mapping of Skills to Assessment  Tasks and Format  Short essay (Produce Text)  Extended revising and editing of written work tasks  Paragraphs and Short Essays  Students will do corrections and make revisions  Paper and Pencil Format  45-50 Minutes  Scores Reported  Individual Student Overall Scores  2 Subscores  Text Production  Revisions and Corrections

24. Copyright © 2004 Educational Testing Service Listening. Learning. Leading. 24 Sample Short Essay Task Some people have suggested that everyone between the ages of 18 and 21 should be required to perform one year of community or government service, such as in the Peace Corps, Environmental Conservancy Corps, a hospital, the military, a rural or inner-city school, or other community outreach projects. Should government or community service be required of 18-21 year olds? Support your point of view with reasons from your own experience, observation, or reading.

25. Copyright © 2004 Educational Testing Service Listening. Learning. Leading. 25 Sample Revision Question “Being a female jockey, she was often interviewed.” Rewrite, beginning with She was often interviewed The next words will be A.“on account of she was” B.“because she was”* C.“by her being” D.“being as she was”

26. Copyright © 2004 Educational Testing Service Listening. Learning. Leading. 26 Customized SUNY Critical Thinking Module: Skills Assessed  Critical thinking and analytical writing skills, including the ability to  Identify, analyze, and evaluate arguments as they occur in their own or other’s work; and  Develop well-reasoned arguments

27. Copyright © 2004 Educational Testing Service Listening. Learning. Leading. 27 Customized SUNY Critical Thinking Module: Key Features  Careful Mapping of Skills to Assessment  Tasks and Format  2 Short Essays (20 - 25 minutes each)  “Develop an Argument”  “Evaluate an Argument”  Task Characteristics  Paper and pencil format  No specialized knowledge required

28. Copyright © 2004 Educational Testing Service Listening. Learning. Leading. 28 Sample Develop Argument Task Directions: Present your perspective on the issue below, using relevant reasons and/or examples to support your views. The media (books, film, music, television, for example) tend to create rather than reflect the values of society.

29. Copyright © 2004 Educational Testing Service Listening. Learning. Leading. 29 Sample Evaluate Argument Task Directions: Discuss how well reasoned you find this argument. Hospital statistics regarding people who go to the emergency room after roller-skating accidents indicate the need for more protective equipment. Within this group of people, 75 percent of those who had accidents in streets or parking lots were not wearing any protective clothing (helmets, knee pads, etc.) or any light-reflecting material (clip-on lights, glow-in-the-dark wrist pads, etc.). Clearly, these statistics indicate that by investing in high-quality protective gear and reflective equipment, roller skaters will greatly reduce their risk of being severely injured in an accident.

30. Copyright © 2004 Educational Testing Service Listening. Learning. Leading. 30 Customized SUNY Critical Thinking Module: Key Features (continued)  Scoring  Essays scored on holistic basis  Scoring can be either human or computer (to be determined based on input from SUNY)  Scores Reported  Individual overall score (based on both essays)  2 Subscores  Develop an Argument  Evaluate an Argument

31. Copyright © 2004 Educational Testing Service Listening. Learning. Leading. Score Reporting

32. Copyright © 2004 Educational Testing Service Listening. Learning. Leading. 32 Sample Individual Student Score Distribution

33. Copyright © 2004 Educational Testing Service Listening. Learning. Leading. 33 Sample Individual Student Score Distribution

34. Copyright © 2004 Educational Testing Service Listening. Learning. Leading. 34 Sample Mathematics Subscore Distribution

35. Copyright © 2004 Educational Testing Service Listening. Learning. Leading. 35 Customized SUNY Mathematics Module: Subscores Interpret and draw inferences from mathematical models such as formulas, graphs, tables and schematics Represent mathematical information symbolically, visually, numerically and verbally Use arithmetical, algebraic, geometric and statistical methods to solve problems Estimate and check mathematical results for reasonableness, determine alternatives and select optimal results xRecognize the limits of mathematical and statistical methods

36. Copyright © 2004 Educational Testing Service Listening. Learning. Leading. 36 Customized SUNY Basic Written Communication Module: Skills Assessed Produce coherent texts within common college-level written forms Demonstrate the ability to revise and improve such texts Research a topic, develop an argument, and organize supporting details

37. Copyright © 2004 Educational Testing Service Listening. Learning. Leading. 37 Customized SUNY Critical Thinking Module: Skills Assessed Assesses critical thinking and analytical writing skills, including the ability to Identify, analyze, and evaluate arguments as they occur in their own or other’s work Develop well-reasoned arguments

38. Copyright © 2004 Educational Testing Service Listening. Learning. Leading. Summary and Supplemental Information

39. Copyright © 2004 Educational Testing Service Listening. Learning. Leading. 39 Customized SUNY General Education Learning Outcomes Assessment  3 Customized Assessments  Mapped to SUNY Learning Outcomes  Each module is independent  Overall Scores and Subscores  Easily administered within a class period  Paper and Pencil Format  Demographic data options  Platform that can support future addition of new modules  Option for locally developed questions (to be determined with input from SUNY Central Administration and and individual campuses)

40. Copyright © 2004 Educational Testing Service Listening. Learning. Leading. 40 Customized SUNY General Education Learning Outcomes Assessment  SUNY Outcome Measures will add value beyond Board of Trustees Mandate  Accreditation (e.g., MSA)  Curriculum development  Benchmarking and Quality Improvement  Assist in the development of a culture of evidence- based decision making through campus-based assessment  Respond proactively to public demand for increased accountability in Higher Education

41. Copyright © 2004 Educational Testing Service Listening. Learning. Leading. 41 Customized SUNY General Education Learning Outcomes Assessment  Reporting Timetable: 10-20 work days  Report Formats  Paper  Electronic  Data reported directly to campus  Individually reliable scores AND subscores  Normative data  SUNY Campuses (dependent upon volume)  National Comparisons in the future  Demographic breakdowns available (e.g., Major, Year in School)

42. Copyright © 2004 Educational Testing Service Listening. Learning. Leading. 42 SUNY General Education Learning Outcomes Assessment Modules  ETS can also provide assistance with  Sampling plans  Data interpretation  Assisting campuses in linking results with curriculum  Support for motivating students in learning outcomes assessments  Development of additional assessments to meet individual campus needs  Test Development Workshops  Development of Standards Based Assessments

43. Copyright © 2004 Educational Testing Service Listening. Learning. Leading. 43 We welcome feedback! Questions/Comments from audience OR E-mail David Payne at: Dpayne@ets.org E-mail Trish Connor at: Pconnor@ets.org@ets.org