1. The Five Step Program Improvement Process
September 5, 2008
Mimi Lufkin
Chief Executive Officer
National Alliance for Partnerships in Equity Education Foundation

2. NSF Extension Services Grants
Unified program of change
Consulting services within a certain community of practice
Adapt exemplary projects, research-based learning tools, pedagogical approaches, and service or support programs
facilitate the interpretation of research knowledge into practice
Inform NSF’s research agenda
To integrate various findings about gender in science and engineering into a unified program of change
To provide consulting services within a certain geographic region or within a community of practice, explaining in simple language the practical meaning and benefits of adopting programs, tools, or approaches that enhance the interest and persistence of female students in STEM studies through the undergraduate level, in those fields where they are underrepresented

3. STEM Equity Pipeline Goals
Build the capacity of the formal education community
Institutionalize the implemented strategies by connecting the outcomes to existing accountability systems
Broaden the commitment to gender equity in STEM education
The STEM Equity Pipeline has three goals:
First, to build the capacity of the formal education community. The focus of the grant is on translating the research and knowledge of STEM and equity experts into practice at the local level. Through this grant, we are building on the unique strengths of participating states in drilling the expertise down to the classroom level. Let’s look at Missouri, for example… Missouri looks different from Illinois, which has a different structure for providing professional development to teachers. There, ….
Second, to institutionalize the implemented strategies by connecting the outcomes to existing accountability systems. Our focus is on making the Perkins data and other state and/or local data sources real and powerful agents of change for local educational agencies. Perkins, as many of you know, is the federal funding source for career and technical education at the secondary and postsecondary levels.
Third, to broaden the commitment to gender equity in STEM education. For example, we developed a brochure entitled, “Stereotypes Turn Girls off of Math and Science” and it encourages educators and parents to rethink their roles in encouraging girls’ participation in STEM classes.
And, how do we define STEM?...

4. Model
Our organizational model is an asset-based model of delivery: we ask each “group” to contribute their knowledge, skills, and experience in a way that best reflects their assets.
For example, we have identified and recruited a National Advisory Board with strong skills in research and big-picture thinking. Their expertise – and what they contribute to the project – is a strong understanding of the research.
We are asking the group of professionals who make up the Extension Services Group to share their expertise in facilitating and managing large-scale projects.
We ask state teams to do what they do best – identify the unique strengths and possibilities that the state offers to “extend” the research into the classroom.
And, in no small measure, we ask educators to do what they do best: put research into action to better serve the diverse students in their classrooms and school communities.
Regardless of the group, we are continually using data to drive decision-making.

5. State Teams 5 states in year one California Missouri Illinois Oklahoma
Wisconsin
Last year, we had five brave states pilot this project: California, Missouri, Illinois, Oklahoma, and my home state of Wisconsin.

6. State Teams 2 states added in year 2 3 states added in year 3
13 states over the life of the grant
Our selection of your state during this 2nd year represents a different standard for participation. During our work last year, we identified and set a bar for this year’s identified states that reflects a commitment to equity, groundwork laid by a network of state professionals committed to equity, and a willingness to leverage and tweak existing professional development channels.
You were selected because your state demonstrated all these qualities – and I encourage you to be proud of your work and see this grant as an opportunity to extend your reach and expertise across the state.

7. Intellectual Specialization
Accountability – Using Data to Drive Program Improvement
The Five Step Process
Professional Development - Implementing Effective Extension Services in the Formal Education Community

8. STEM Equity Pipeline Project Methods
Professional Development
Consulting and Technical Assistance
Virtual Web-based Professional Learning Community
Best Practices Handbook
Professional development on implementation of the 5 Step Improvement Process and research-based gender equity in STEM strategies focused on school-identified needs
Teacher training on creating equitable classrooms using research-based models for eliminating bias and stereotyping in instructional practices focused on gender, disability and culture
Consulting and technical assistance from recognized experts in gender equity research and practice
Access to a virtual web-based professional learning community
Development of a Best Practices Handbook to be used by the Extension Services Group to train state teams to provide extension services (consulting, technical assistance and professional development) to their constituencies and as a dissemination tool for the National Advisory Board
We are focusing our resources on a variety of professional development delivery methods with the clear goal of reaching educators where they are – at a conference, online, or in the throes of research and reflection.

10. Perkins Act Accountability
Core Indicators on Nontraditional CTE
Participation in CTE programs preparing students for nontraditional fields
Completion of CTE programs preparing students for nontraditional fields

11. Nontraditional Fields
Occupations or fields of work, including careers in computer science, technology, and other current and emerging high skill occupations, for which individuals from one gender comprise less than 25 percent of the individuals employed in each such occupation or field of work.

12. Document Performance Results
Understand the problem completely before you seek solutions
How do you analyze performance data?
What questions should be addressed?
What tools and methods can be used to present and analyze data?
How should data quality problems be considered in analyzing data?
See The Guide for detailed explanation

13. Perkins Accountability Measures
Participation data
4S1 & 4P1 in Perkins III
6S1 & 5P1 in Perkins IV
Participation Rate =
# underrepresented students participating in NT CTE
all students participating in NT CTE
Participation data is collected in the accountability report and was called 4S1 or 4P1 during Perkins III and will be called 6S1 or 6P1 during Perkins IV. S stands for secondary, p for postsecondary.
Completion data is collected in the accountability report and was called 4S2 or during Perkins III and will be called 6S2 or during Perkins IV.
The equations are the formulas for the measures

14. Perkins Accountability Measures
Completion data
4S2 & 4P2 in Perkins III
6S2 & 5P2 in Perkins IV
Completion Rate=
# underrepresented students completing NT CTE
all students completing NT CTE
Participation data is collected in the accountability report and was called 4S1 or 4P1 during Perkins III and will be called 6S1 or 6P1 during Perkins IV. S stands for secondary, p for postsecondary.
Completion data is collected in the accountability report and was called 4S2 or during Perkins III and will be called 6S2 or during Perkins IV.
The equations are the formulas for the measures

15. Data Collection Disaggregation required in Perkins IV
Gender
Male
Female
Special Population
Underrepresented gender students in a nontraditional CTE program
Single Parent
Displaced Homemaker
Limited English Proficiency
Individuals with a Disability
Economically Disadvantaged
Race/Ethnicity
American Indian or Alaskan Native
Asian or Pacific Islander
Black, non-Hispanic
Hispanic
White- non-Hispanic
These are the data disaggregation elements required in Perkins IV.
Perkins IV is silent on the definitions for some of these groups. The NAPE recommended definitions for these groups can be found in Section 1 of the Guide for Program Improvement for Perkins IV: Nontraditional CTE Program Participation and Completion.
Special Populations
Individuals with Disabilities: Perkins IV defines students with disabilities by referring to the Americans with Disabilities Act definition: An individual with a physical or mental impairment that substantially limits one or more of the major life activities of such individual.[1]
Economically Disadvantaged: Not defined in Perkins IV. Check with your state to determine what criteria to use for identifying these students. Typically states use the same definition for economically disadvantaged students at the secondary level as is used for No Child Left Behind or that qualify students for the free and reduced lunch program in the state. At the postsecondary level the definition for economically disadvantaged students is usually related to PELL grant eligibility or other financial aid programs.
Single Parent: Not defined in Perkins IV. Single parents are defined in other federal laws. For example: An individual who is unmarried or legally separated from a spouse; and has 1 or more minor children for whom the individual has custody or joint custody; or is pregnant.[2]
Displaced Homemaker: An individual who has worked primarily without remuneration to care for a home and family, and for that reason has diminished marketable skills; has been dependent on the income of another family member but is no longer supported by that income; or is a parent whose youngest dependent child will become ineligible to receive assistance under part A of Title IV of the Social Security Act (42 U.S.C. 601 et seq.) not later than 2 years after the date on which the parent applies for assistance under such title; and is unemployed or underemployed and is experiencing difficulty in obtaining or upgrading employment.
Limited English Proficient Student: A secondary school student, an adult, or an out-of-school youth, who has limited ability in speaking, reading, writing, or understanding the English language, and whose native language is a language other than English; or who lives in a family or community environment in which a language other than English is the dominant language.
Migrant: Not defined in Perkins IV but is included due to the reference to the data collection elements in No Child Left Behind. The definition for migratory child in NCLB is: a child who is, or whose parent or spouse is, a migratory agricultural worker, including a migratory dairy worker, or a migratory fisher, and who, in the preceding 36 months, in order to obtain, or accompany such parent or spouse, in order to obtain, temporary or seasonal employment in agricultural or fishing work —
(A) has moved from one school district to another;
(B) in a State that is comprised of a single school district, has moved from one administrative area to another within such district; or
(C) resides in a school district of more than 15,000 square miles, and migrates a distance of 20 miles or more to a temporary residence to engage in a fishing activity.[3]
Student Pursuing Nontraditional Careers: The term “nontraditional fields” means occupations or fields of work, including careers in computer science, technology, and other current and emerging high skill occupations, for which individuals from one gender comprise less than 25 percent of the individuals employed in each such occupation or field of work.
Tech Prep Student: A student participating in a Tech Prep program as described in Sec. 203 of the Perkins Act.[4]
[1]Available at
[2]For a definition for single parent see U.S. Code, Title 12, Chapter 13, Section 1700(d)(9) available at
[3] From NCLB available at
[4]Available at

16. Recommended Analyses Comparisons State performance level
Best performer in state
Selected peer benchmark
Set your own benchmark
Trends
At least 2 yrs
Prefer 3-5 yrs
Site specific
Statewide
District
School/College
Programs
There are many ways to check your progress. Comparisons can be made in many ways – not just on the states negotiated performance level or the state actual performance level for Perkins.
Trend data is very important to determine progress and to know if your performance is going up or down. The nontraditional indicator can be fairly volatile especially if you are looking at program level data where the enrollment of one individual can mean a large percentage increase or the opposite!
Disaggregating performance data by district, school/college and program can be very revealing in identifying where performance issues may reside. The more disaggregated the data is the more revealing it can be in helping pinpoint gaps in performance.
Accurate and consistent identification of nontraditional program is also very important to ensure the quality and reliability of the data that is collected and reported is maintained. Comparing apples to apples is the only way to use data effectively to help you identify where to look for the root causes that are affecting performance.

17. Perkins Accountability Resources
Peer Collaborative Resource Network
National Alliance for Partnerships in Equity
Growing Pains
Nontraditional SOC/CIP Crosswalk for Males and Females
Your State Agency Website
SOC- Standard Occupation Code
CIP- Classification of Instructional Programs

18. Other Data Sources
National Center for Educational Achievement – Just 4 the Kids
National Center for Educational Statistics
Southern Regional Education Board Data Library
State Data
Local Data
NCLB – No Child Left Behind

19. Poll
Have you used your Perkins data to identify gaps in performance for students pursuing nontraditional careers?
I do this all the time
I have tried with some success
I have tried with no success
I don’t have access to the data

21. Why Search for Root Causes?
Keep from fixating on the
“silver bullet” strategy
Identify the conditions or factors that cause or permit a performance gap to occur
Direct cause (i.e. instructional practice)
Indirect cause (i.e. teacher training)

22. How to Identify Root Causes
Search for most direct and highest impact causes
Employ a systematic evidence-based process
Formulate and test theories or hypotheses
Draw on current research and evaluation
Use multiple methods and data sources
Likely to find multiple causes

23. Identify Potential Causes
Analyzing Student Data
Reviewing Research Literature
Reviewing Program/Institutional Evaluations and Effectiveness Reviews
Conducting Focus Groups
Brainstorming
Peer Benchmarking
Interviews
Surveys

24. Review Research Summary
Chapter 6 Tables 16, 17, 19, and 20
Improving Performance on Perkins III Core Indicators: Summary of Research on Causes and Improvement Strategies
Authors: Robert Sheets, Mimi Lufkin, and David Stevens for the National Centers for Career & Technical Education
Available at

26. Choose Best Solutions
Don’t be too quick to adopt best practices before getting the facts straight
How do you identify possible strategies and model practices?
How do you evaluate strategies and models?
How do you compare and assess alternative solutions and make a decision?

27. Find and Evaluate Solutions
Failure is expensive
Build consensus among staff and stakeholders
Get full support and commitment
Select full range of choices – be creative
Implement systematic analysis

28. Identify Potential Strategies and Models
Review What Others Propose
NSF- New Formulas for America’s Workforce
Benchmark Peers and Leading Performers
Programs and Practices That Work
Best Practices in STEM Education
EEES Best Practices
Develop Your Own Solutions

29. Review Research Summary
Chapter 6 Tables 18 and 21
Improving Performance on Perkins III Core Indicators: Summary of Research on Causes and Improvement Strategies
Authors: Robert Sheets, Mimi Lufkin, and David Stevens for the National Centers for Career & Technical Education
Available at

30. Poll
How are you feeling about the Five-Step Program Improvement Process?
Excited – This sounds like it could really help us be more effective!
Curious – I want to know more!
Frustrated – I need to track down our Perkins data
Overwhelmed – I’ll let someone else handle this!
Excited - I can’t wait to assess!
Overwhelmed – Where do I begin?!
Not very enthused, but I know I need to do it.
Overworked - I’ll let someone else handle assessment.

32. Pilot Test and Evaluate
Make sure your improvement strategy works somewhere before you attempt to apply it everywhere
Choose a study design
Select pilot sites
Select outcome measures
Identify Data Sources
Train Pilot Site Staff
Evaluate Initial Results

33. Assessment Resources STEM Equity Pipeline Webinars
Building Effective Program Assessments
Assessing Effectiveness
Archived at
Assessing Women and Men in Engineering –

35. 5 Step Process Resources
STEM Equity Pipeline
Virtual Learning Community
Professional Development Menu
The Five Step Process

36. Questions? Mimi Lufkin Project Director, Chief Executive Officer
National Alliance for Partnerships in Equity
P.O. Box 369, Cochranville, PA
phone
fax