The Five Step Program Improvement Process September 5, 2008 Mimi Lufkin Chief Executive Officer National Alliance for Partnerships in Equity Education Foundation

NSF Extension Services Grants Unified program of change Consulting services within a certain community of practice Adapt exemplary projects, research-based learning tools, pedagogical approaches, and service or support programs facilitate the interpretation of research knowledge into practice Inform NSF’s research agenda To integrate various findings about gender in science and engineering into a unified program of change To provide consulting services within a certain geographic region or within a community of practice, explaining in simple language the practical meaning and benefits of adopting programs, tools, or approaches that enhance the interest and persistence of female students in STEM studies through the undergraduate level, in those fields where they are underrepresented

STEM Equity Pipeline Goals Build the capacity of the formal education community Institutionalize the implemented strategies by connecting the outcomes to existing accountability systems Broaden the commitment to gender equity in STEM education The STEM Equity Pipeline has three goals: First, to build the capacity of the formal education community. The focus of the grant is on translating the research and knowledge of STEM and equity experts into practice at the local level. Through this grant, we are building on the unique strengths of participating states in drilling the expertise down to the classroom level. Let’s look at Missouri, for example… Missouri looks different from Illinois, which has a different structure for providing professional development to teachers. There, …. Second, to institutionalize the implemented strategies by connecting the outcomes to existing accountability systems. Our focus is on making the Perkins data and other state and/or local data sources real and powerful agents of change for local educational agencies. Perkins, as many of you know, is the federal funding source for career and technical education at the secondary and postsecondary levels. Third, to broaden the commitment to gender equity in STEM education. For example, we developed a brochure entitled, “Stereotypes Turn Girls off of Math and Science” and it encourages educators and parents to rethink their roles in encouraging girls’ participation in STEM classes. And, how do we define STEM?...

Model Our organizational model is an asset-based model of delivery: we ask each “group” to contribute their knowledge, skills, and experience in a way that best reflects their assets. For example, we have identified and recruited a National Advisory Board with strong skills in research and big-picture thinking. Their expertise – and what they contribute to the project – is a strong understanding of the research. We are asking the group of professionals who make up the Extension Services Group to share their expertise in facilitating and managing large-scale projects. We ask state teams to do what they do best – identify the unique strengths and possibilities that the state offers to “extend” the research into the classroom. And, in no small measure, we ask educators to do what they do best: put research into action to better serve the diverse students in their classrooms and school communities. Regardless of the group, we are continually using data to drive decision-making.

State Teams 5 states in year one California Missouri Illinois Oklahoma Wisconsin Last year, we had five brave states pilot this project: California, Missouri, Illinois, Oklahoma, and my home state of Wisconsin.

State Teams 2 states added in year 2 3 states added in year 3 13 states over the life of the grant Our selection of your state during this 2nd year represents a different standard for participation. During our work last year, we identified and set a bar for this year’s identified states that reflects a commitment to equity, groundwork laid by a network of state professionals committed to equity, and a willingness to leverage and tweak existing professional development channels. You were selected because your state demonstrated all these qualities – and I encourage you to be proud of your work and see this grant as an opportunity to extend your reach and expertise across the state.

Intellectual Specialization Accountability – Using Data to Drive Program Improvement The Five Step Process Professional Development - Implementing Effective Extension Services in the Formal Education Community

STEM Equity Pipeline Project Methods Professional Development Consulting and Technical Assistance Virtual Web-based Professional Learning Community Best Practices Handbook Professional development on implementation of the 5 Step Improvement Process and research-based gender equity in STEM strategies focused on school-identified needs Teacher training on creating equitable classrooms using research-based models for eliminating bias and stereotyping in instructional practices focused on gender, disability and culture Consulting and technical assistance from recognized experts in gender equity research and practice Access to a virtual web-based professional learning community Development of a Best Practices Handbook to be used by the Extension Services Group to train state teams to provide extension services (consulting, technical assistance and professional development) to their constituencies and as a dissemination tool for the National Advisory Board We are focusing our resources on a variety of professional development delivery methods with the clear goal of reaching educators where they are – at a conference, online, or in the throes of research and reflection.

Perkins Act Accountability Core Indicators on Nontraditional CTE Participation in CTE programs preparing students for nontraditional fields Completion of CTE programs preparing students for nontraditional fields

Nontraditional Fields Occupations or fields of work, including careers in computer science, technology, and other current and emerging high skill occupations, for which individuals from one gender comprise less than 25 percent of the individuals employed in each such occupation or field of work.

Document Performance Results Understand the problem completely before you seek solutions How do you analyze performance data? What questions should be addressed? What tools and methods can be used to present and analyze data? How should data quality problems be considered in analyzing data? See The Guide for detailed explanation

Perkins Accountability Measures Participation data 4S1 & 4P1 in Perkins III 6S1 & 5P1 in Perkins IV Participation Rate = # underrepresented students participating in NT CTE all students participating in NT CTE Participation data is collected in the accountability report and was called 4S1 or 4P1 during Perkins III and will be called 6S1 or 6P1 during Perkins IV. S stands for secondary, p for postsecondary. Completion data is collected in the accountability report and was called 4S2 or during Perkins III and will be called 6S2 or during Perkins IV. The equations are the formulas for the measures

Perkins Accountability Measures Completion data 4S2 & 4P2 in Perkins III 6S2 & 5P2 in Perkins IV Completion Rate= # underrepresented students completing NT CTE all students completing NT CTE Participation data is collected in the accountability report and was called 4S1 or 4P1 during Perkins III and will be called 6S1 or 6P1 during Perkins IV. S stands for secondary, p for postsecondary. Completion data is collected in the accountability report and was called 4S2 or during Perkins III and will be called 6S2 or during Perkins IV. The equations are the formulas for the measures

Data Collection Disaggregation required in Perkins IV Gender Male Female Special Population Underrepresented gender students in a nontraditional CTE program Single Parent Displaced Homemaker Limited English Proficiency Individuals with a Disability Economically Disadvantaged Race/Ethnicity American Indian or Alaskan Native Asian or Pacific Islander Black, non-Hispanic Hispanic White- non-Hispanic These are the data disaggregation elements required in Perkins IV. Perkins IV is silent on the definitions for some of these groups. The NAPE recommended definitions for these groups can be found in Section 1 of the Guide for Program Improvement for Perkins IV: Nontraditional CTE Program Participation and Completion. Special Populations Individuals with Disabilities: Perkins IV defines students with disabilities by referring to the Americans with Disabilities Act definition: An individual with a physical or mental impairment that substantially limits one or more of the major life activities of such individual.[1] Economically Disadvantaged: Not defined in Perkins IV. Check with your state to determine what criteria to use for identifying these students. Typically states use the same definition for economically disadvantaged students at the secondary level as is used for No Child Left Behind or that qualify students for the free and reduced lunch program in the state. At the postsecondary level the definition for economically disadvantaged students is usually related to PELL grant eligibility or other financial aid programs. Single Parent: Not defined in Perkins IV. Single parents are defined in other federal laws. For example: An individual who is unmarried or legally separated from a spouse; and has 1 or more minor children for whom the individual has custody or joint custody; or is pregnant.[2] Displaced Homemaker: An individual who has worked primarily without remuneration to care for a home and family, and for that reason has diminished marketable skills; has been dependent on the income of another family member but is no longer supported by that income; or is a parent whose youngest dependent child will become ineligible to receive assistance under part A of Title IV of the Social Security Act (42 U.S.C. 601 et seq.) not later than 2 years after the date on which the parent applies for assistance under such title; and is unemployed or underemployed and is experiencing difficulty in obtaining or upgrading employment. Limited English Proficient Student: A secondary school student, an adult, or an out-of-school youth, who has limited ability in speaking, reading, writing, or understanding the English language, and whose native language is a language other than English; or who lives in a family or community environment in which a language other than English is the dominant language. Migrant: Not defined in Perkins IV but is included due to the reference to the data collection elements in No Child Left Behind. The definition for migratory child in NCLB is: a child who is, or whose parent or spouse is, a migratory agricultural worker, including a migratory dairy worker, or a migratory fisher, and who, in the preceding 36 months, in order to obtain, or accompany such parent or spouse, in order to obtain, temporary or seasonal employment in agricultural or fishing work — (A) has moved from one school district to another; (B) in a State that is comprised of a single school district, has moved from one administrative area to another within such district; or (C) resides in a school district of more than 15,000 square miles, and migrates a distance of 20 miles or more to a temporary residence to engage in a fishing activity.[3] Student Pursuing Nontraditional Careers: The term “nontraditional fields” means occupations or fields of work, including careers in computer science, technology, and other current and emerging high skill occupations, for which individuals from one gender comprise less than 25 percent of the individuals employed in each such occupation or field of work. Tech Prep Student: A student participating in a Tech Prep program as described in Sec. 203 of the Perkins Act.[4] [1]Available at http://www.eeoc.gov/policy/ada.html. [2]For a definition for single parent see U.S. Code, Title 12, Chapter 13, Section 1700(d)(9) available at http://www.thecre.com/fedlaw/uscode/12/index-2.html [3] From NCLB available at http://www.ed.gov/policy/elsec/leg/esea02/pg8.html#sec1309 [4]Available at http://frwebgate.access.gpo.gov/cgi-bin/getdoc.cgi?dbname=109_cong_bills&docid=f:s250enr.txt.pdf.

Recommended Analyses Comparisons State performance level Best performer in state Selected peer benchmark Set your own benchmark Trends At least 2 yrs Prefer 3-5 yrs Site specific Statewide District School/College Programs There are many ways to check your progress. Comparisons can be made in many ways – not just on the states negotiated performance level or the state actual performance level for Perkins. Trend data is very important to determine progress and to know if your performance is going up or down. The nontraditional indicator can be fairly volatile especially if you are looking at program level data where the enrollment of one individual can mean a large percentage increase or the opposite! Disaggregating performance data by district, school/college and program can be very revealing in identifying where performance issues may reside. The more disaggregated the data is the more revealing it can be in helping pinpoint gaps in performance. Accurate and consistent identification of nontraditional program is also very important to ensure the quality and reliability of the data that is collected and reported is maintained. Comparing apples to apples is the only way to use data effectively to help you identify where to look for the root causes that are affecting performance.

Perkins Accountability Resources Peer Collaborative Resource Network www.edcountability.net National Alliance for Partnerships in Equity Growing Pains Nontraditional SOC/CIP Crosswalk for Males and Females www.napequity.org Your State Agency Website SOC- Standard Occupation Code CIP- Classification of Instructional Programs

Other Data Sources National Center for Educational Achievement – Just 4 the Kids National Center for Educational Statistics Southern Regional Education Board Data Library State Data Local Data NCLB – No Child Left Behind

Poll Have you used your Perkins data to identify gaps in performance for students pursuing nontraditional careers? I do this all the time I have tried with some success I have tried with no success I don’t have access to the data

Why Search for Root Causes? Keep from fixating on the “silver bullet” strategy Identify the conditions or factors that cause or permit a performance gap to occur Direct cause (i.e. instructional practice) Indirect cause (i.e. teacher training)

How to Identify Root Causes Search for most direct and highest impact causes Employ a systematic evidence-based process Formulate and test theories or hypotheses Draw on current research and evaluation Use multiple methods and data sources Likely to find multiple causes

Identify Potential Causes Analyzing Student Data Reviewing Research Literature Reviewing Program/Institutional Evaluations and Effectiveness Reviews Conducting Focus Groups Brainstorming Peer Benchmarking Interviews Surveys

Review Research Summary Chapter 6 Tables 16, 17, 19, and 20 Improving Performance on Perkins III Core Indicators: Summary of Research on Causes and Improvement Strategies Authors: Robert Sheets, Mimi Lufkin, and David Stevens for the National Centers for Career & Technical Education Available at www.napequity.org/nape_publications.htm

Choose Best Solutions Don’t be too quick to adopt best practices before getting the facts straight How do you identify possible strategies and model practices? How do you evaluate strategies and models? How do you compare and assess alternative solutions and make a decision?

Find and Evaluate Solutions Failure is expensive Build consensus among staff and stakeholders Get full support and commitment Select full range of choices – be creative Implement systematic analysis

Identify Potential Strategies and Models Review What Others Propose NSF- New Formulas for America’s Workforce Benchmark Peers and Leading Performers Programs and Practices That Work Best Practices in STEM Education EEES Best Practices Develop Your Own Solutions

Review Research Summary Chapter 6 Tables 18 and 21 Improving Performance on Perkins III Core Indicators: Summary of Research on Causes and Improvement Strategies Authors: Robert Sheets, Mimi Lufkin, and David Stevens for the National Centers for Career & Technical Education Available at www.napequity.org/nape_publications.htm

Poll How are you feeling about the Five-Step Program Improvement Process? Excited – This sounds like it could really help us be more effective! Curious – I want to know more! Frustrated – I need to track down our Perkins data Overwhelmed – I’ll let someone else handle this! Excited - I can’t wait to assess! Overwhelmed – Where do I begin?! Not very enthused, but I know I need to do it. Overworked - I’ll let someone else handle assessment.

Pilot Test and Evaluate Make sure your improvement strategy works somewhere before you attempt to apply it everywhere Choose a study design Select pilot sites Select outcome measures Identify Data Sources Train Pilot Site Staff Evaluate Initial Results

Assessment Resources STEM Equity Pipeline Webinars Building Effective Program Assessments Assessing Effectiveness Archived at www.stemequitypipeline.org Assessing Women and Men in Engineering – www.aweonline.org

5 Step Process Resources STEM Equity Pipeline Virtual Learning Community Professional Development Menu The Five Step Process www.stemequitypipeline.org/ProfessionalDevelopment/TheFiveStepProgramImprovementProcess.aspx

Questions? Mimi Lufkin Project Director, Chief Executive Officer National Alliance for Partnerships in Equity P.O. Box 369, Cochranville, PA 19330 610-593-8038 phone 610-593-7283 fax www.stemequitypipeline.org www.napequity.org