American Diploma Project Research Progress Report for Kentucky P-16 Council Meeting December 2, 2002

ADP States Indiana Kentucky Massachusetts Nevada Texas

ADP Partner Organizations Achieve, Inc. Education Trust Thomas B. Fordham Foundation National Alliance of Business

ADP Goal Make the high school diploma meaningful Secure demand for standards-based assessment data from postsecondary institutions and employers Help states ensure that they have set the bar for high school graduation in reading, writing and mathematics at the right level. Establish new benchmarks in reading, writing and mathematics that all states can use

ADP Partner State Commitments Partner state leaders* have agreed to rally behind current standards-based systems and to participate in a review of their current standards and assessments Business and postsecondary leaders in partner states have agreed to consider standards-based assessment data in admissions, placement and hiring decisions *Governors, chiefs, postsecondary executives, business leaders

ADP Project Commitments Conduct Research Provide Technical Assistance Develop New Benchmarks

Research –Legal Study: assisting partner states with new frontiers; legal implications of a K-16 system (Nixon-Peabody) –Defining Workplace Expectations: Defining the reading, writing and math skills needed for success in new economy occupations (ETS and NAB) –Defining Postsecondary Expectations: Defining the reading, writing and math skills needed for success in postsecondary institutions (Ed Trust and Achieve)

Legal Study Standards Reform, Federal Law and the American Diploma Project: A Framework for Making legally Sound Decisions Nixon-Peabody LLP Prospective Legal Analysis of: –Relevant federal laws –Ways that ADP implicates federal law Provides: –Framework for states for analyzing ADP efforts –Recommendations for minimizing risk

Relevant Federal Laws Due Process Discrimination –As applied to Education –As applied to Employment Practices Tests serve different purposes in different contexts –Looking back (K-12) –Looking forward (admissions & placement) –Implications for validity and opportunity to learn Deference often given to education judgments

Re: GI Forum v. Texas Education Agency Despite sobering differences in pass rates, the standards-based high school exit exam –provided an objective assessment of mastery of a discrete set of skills and knowledge linked to the state’s educational standards, –served to motivate students and –provided support for systemic accountability. TAAS met currently accepted standards for: –curricular validity –opportunity to learn the matter covered by the test –remediation efforts –multiple opportunities to pass

“ Central to the court’s reasoning on the due process and discrimination claims was its belief that the TAAS was ultimately educationally beneficial to all students, and that the state was using assessment data to address rather than exacerbate, disparities in educational opportunity and achievement.”

Recommendations Identify key employment and postsecondary expectations Conduct gap analysis and align standards for high school exit, college and employment Ensure that curriculum, instruction and interventions align to standards Assess student knowledge of standards and ensure predictive validity of assessments Phase in appropriate use of high school assessment data in admissions, placement and hiring decisions

Defining Workplace Expectations ETS, NAB & Project Staff Defined “good jobs” –Higher wages (top two tiers of pyramid, K/year) –Insurance and retirement benefits –Opportunities for additional education or training Examined labor market projections Tracked course-taking patterns Developed preliminary benchmarks

Distribution of Education & Jobs: 1973 v Percentage of employed who were high school dropouts/had not attended college 1973: 72% 2000: 41% Percentage of employed who had some college or a B.A. 1973: 21%2000: 48%

Employment Pyramid

Workplace Study Highlights Algebra II is the benchmark course for students aspiring to highly paid professional jobs or well-paid, white-collar jobs. Geometry is the benchmark course for students intending to work in well-paid, blue- collar jobs and low-paid/low-skilled jobs. Four years of English that is at least at grade level is a benchmark requirement in the vast majority of jobs.

Preliminary Workplace Benchmarks Based on course-taking patterns identified in study Two panels of curricular experts refined and revised benchmarks Benchmarks circulated by NAB among employers in occupations identified by ETS Consistency with postsecondary benchmarks

Flatter (leaner) management structures –are less dependent on manual labor; –require more skilled workers who can identify and solve problems, offer innovative ideas for improving productivity; –demand that workers can articulate clearly, think creatively, absorb data, read and write effectively; –expect that workers can draw on a stronger foundation of skills –prompt workers to cope with constant change Computers render jobs obsolete; raise stakes for workers who use them

Employers supported preliminary benchmarks Suggested students apply knowledge and skills in real-world situations w/o becoming “narrowly vocational” Encouraged interdisciplinary learning Reiterated importance of strong reading & writing skills for success in teams Strongly encouraged better teacher training Endorsed benchmarks (rigorous curricula) that will keep more students in the pipeline for postsecondary degrees and promotability

Employers’ suggested areas of emphasis: ELA Analytical thinking (v. “interpretive” thinking): develop, analyze, present a logical argument, in writing and orally; summarize & synthesize information; extract essential information; evaluate the validity of information Use correct English grammar and sentence structure Speak persuasively under time constraints Translate technical information to a non- technical audience Plan and complete extended research projects (research as part of problem-solving)

Employers’ suggested areas of emphasis: math Budgeting, accounting Probability and statistics Ability to express real-world problems in mathematical terms Ability to understand and apply mathematics in new and unfamiliar contexts Metric system Formulating arguments (e.g., proofs) Data interpretation

Defining Postsecondary Expectations Education Trust, Achieve, Project Staff Test Content Analysis (January ‘02) Examination of state standards Examination of alignment between standards & assessments Cross-content area faculty defined reading, writing and math competencies necessary for postsecondary success Cross-content area faculty discussed: “To what extent do current state standards & assessments reflect postsecondary expectations?”

Cross-State Postsecondary Expectations: Consistency across systems within states, especially in reading & writing –Some consistency across content areas Consistency across states Consistency with workplace expectations

Team findings (Reading): Postsecondary expectations are not fully addressed in the Kentucky Core Content Standards should include critical thinking skills specifically w/regard to reading Standards should emphasize analytic analyses (v. personal reflections) Quality and complexity of reading should be delineated Some standards too vague to understand actual student expectations

Team findings (Writing): Postsecondary expectations are not fully addressed in the Kentucky Core Content Clarify jargon Specify thesis and idea development Require analytic writing, development of arguments, compare and contrast essays Writing standards should be genre-specific

Team findings (Math): Math core content contains most of postsecondary expectations for non-math- dependent fields, but not for math-dependent Standards should include dimensional analysis, bases of number systems, developing proofs, translating between metric and other measurement systems, understanding formulas Organization (concepts v. skills v. relationships) is confusing Standards should clarify calculator policy

Team findings: Reading Assessment Greater emphasis needed on higher order comprehension skills and persuasive text analysis Content centrality: 80% alignment, but 20% could not be determined because standards are too broadly-worded Performance centrality: only 60 % alignment Level of cognitive demand too low (86% of items at levels 1 and 2.

Team findings: Writing Assessment Holistic scoring guide useful; would serve as strong foundation for genre-specific scoring guides Samples of proficient responses less than proficient for postsecondary admissions or placement Move on-demand writing to fall of 12 th grade Correct grammar & usage, analytic and persuasive writing should be emphasized

Team findings: Math Assessment Greater emphasis needed on algebraic ideas, concepts, skills and relationships (currently only three of 19 algebra objectives are assessed); also more on geometry Reduce percentage of assessment dedicated to probability and statistics Content centrality: only 43.3% alignment Performance Centrality: only 46.7% alignment Most important standards are left unassessed Level of cognitive demand too low (90% at levels 1 and 2

ADP State Policy Panels Review Research Findings Consider policy options for meeting ADP goals –Use of standards-based assessment data in high school graduation, college admissions/placement and hiring decisions Develop plan for meeting ADP goals and closing gaps between current standards/assessments and postsecondary and employer expectations over time

Kentucky Policy Issues Current high school assessments do not reflect most important content in the standards (as prioritized by postsecondary faculty and employers) Some important content is missing from standards and assessments Assessments do not generate individual scores or reports by domain & item-type Timing of assessments makes data difficult to use

Short-term steps for revising reading & writing standards & assessments Raise cognitive demand of items in bank Revise writing anchors to reflect postsecondary expectations for placement Revise rubric to be genre-specific Add analytical writing to writing portfolio Clarify calculator and formula sheet policies Revisit mathematics scoring guide Consider use of KEMPT to supplement KCCT Pilot alternative admissions, placement and hiring policies

Long-term considerations Matrix testing v. individual student (NCLB requirements) Streamline and revisit content of POS, expectations and core content Clarify quality and complexity of reading in standards Revisit timing of tests; consider adding end-of- course assessments Design predictive validity studies Revise admissions, placement and hiring policies Report scores on transcripts Default curriculum and interventions (OTL)

ADP Phase II: New Benchmarks Work with partner state policy panels to implement ADP goals Synthesize workplace and postsecondary expectations (to create draft benchmarks) Circulate draft benchmarks Supervise content area expert panels Establish National Advisory Panel Conduct campaign encouraging use of benchmarks by all states