Dr. Albrecht Research Team EXAMPLE of EVALUATIO N RESEARCH SERVICE LEARNING

 Dr. Carol Albrecht was hired by the State of Texas Service Learning Center  18 public school districts in the state of Texas were involved in Service Learning Projects  We met with them to identify how the program impacted public school children, teachers, community partners and parents of students ABOUT THE PROJECT

THE PROCESS One Identified the Objectives, Conceptualized Variables and Developed Indicators. Two Selected the Sample. Three Constructed the Indicators Four Selected a Research Design. Five Collected and Analyzed the Data. Six Wrote the Report and Presented the Findings

STEP #1 One Identified the Objectives, Conceptualized Variables and Developed Indicators. Two Selected the Sample. Three Constructed the Indicators Four Selected a Research Design. Five Collected and Analyzed the Data. Six Wrote the Report and Presented the Findings

Does Service Learning… Have a positive impact on students? Have the support of the teachers? Have the support of the community, parents & partners? When, where and why is it effective? IDENTIFY THE OBJECTIVES

Development of Small Group Leadership Skills Increase in a Positive Affect Toward School Development of Problem Solving Skills Development of Post- High School Aspirations IDENTIFY THE OUTCOMES

Concept Leadership SkillsLiking for SchoolProblem Solving SkillsPost High School Aspirations Indicator Students’ Perceptions Measured by Index Questions on Pre and Post Test Students’ Perceptions as Measured by Index Questions on Pre and Post Test DEVELOP INDICATORS

STEP#2 One Identified the Objectives, Conceptualized Variables and Developed Indicators. Two Selected the Sample. Three Constructed the Indicators. Four Selected the Research Design. Five Collected and Analyzed the Data. Six Wrote the Report and Presented Findings.

3500 high school students2500 elementary students 200 of their teachers and 18 of the service learning coordinators 20 parents and 30 community partners SELECT A SAMPLE

IDENTIFY DIVERSE SCHOOLS

 Middle/High School  Raising money for various causes  Tutoring Elementary children  Landscaping Parks  Building playground equipment  Painting murals on boy/girl club walls SERVICE LEARNING PROJECTS

One Identified the Objectives, Conceptualized Variables and Developed Indicators. Two Selected the Sample. Three Constructed the Indicators Four Determined Research Design. Constructed Instrument. Five Collected and Analyzed the Data. Six Wrote the Report and Presented Data. STEP #3

USE A SURVEY TO DEVELOP INDICATORS Students report more confidence in leadership skills after participation in Service Learning Students report more confidence in problem solving skills after participation in Service Learning Students report greater affinity for teachers and school after participation in Service Learning Students report greater desire to pursue post secondary education after participation in Service Learning

RELATIONSHIP BETWEEN SUCCESS AND LEVELS OF SERVICE LEARNING PARTICIPATION Amount of participation is positively related to confidence in leadership skills Amount of participation is positively related to confidence in problem solving skills. Amount of participation if positively related to affinity for teachers and school Amount of participation is positively related to post secondary aspirations

 Development of Leadership Skills  Development of Problem Solving Skills  Increased Affect Toward School  Increased Academic Aspirations  Positive Affect Toward Program HOW THE STUDENTS FELT ABOUT PARTICIPATING IN THE PROGRAM

One Identified the Objectives, Conceptualized Variables and Developed Indicators. Two Selected the Sample. Three Constructed the Indicators. Four Selected the Research Design. Five Collected and Analyzed the Data. Six Wrote the Report and Presented Findings. STEP #4

 We decided to use multi-methods to increase the validity and reliability of our findings.  Used Experimental Design to evaluate the program and test our hypotheses  Pre-test survey  Intervention  Post-test survey  Used Focus Groups  In addition we used focus groups to gain additional information/insights and further test the validity/reliability of our survey. SELECT RESEARCH DESIGN

One Identified the Objectives, Conceptualized Variables and Developed Indicators. Two Selected the Sample. Three Constructed the Indicators. Four Selected the Research Design. Five Collected and Analyzed the Data. Six Wrote the Report and Presented the Findings. STEP #5

 Texas A&M students went all over Texas and administered the pre-test  Five months later they returned to administer the post-test  12 schools were selected for the focus groups  They were recorded by the students  They were later transcribed  A coding schema was developed to compare and contrast answers across schools ADMINISTERING THE TEST

One Identified the Objectives, Conceptualized Variables and Developed Indicators. Two Selected the Sample. Three Constructed the Indicators. Four Selected the Research Design. Five Collected and Analyzed the Data. Six Wrote the Report and Presented the Findings. STEP #6

WRITING THE REPORT  We wrote a final report and prepared a power point presentation for the teachers and administrators.  It was important to interpret the information accurately even though it was not what the Texas Center for Service Learning was hoping to hear.  In summary, we found very small and sometimes even differences comparing the pre- and post-tests. Most of the difference were NOT statistically significant.  These findings were consistent across two years  However, we had collected data that allowed us to do a more in-depth analysis, and so we proceeded.

 We now looked at the quantity and quality of the experience more specifically – and how that influenced outcomes.  To measure quantity of Service Learning, we measured how many total hours students had spent in service learning that year.  To measure quality, we measured sense of ownership  Did students help plan the Service Learning Project?  Did students make decisions about the Service Learning Project? EVALUATION

Mean Score on Outcome Measurements by Amount of Student Planned or Participated in Project Students Planned -Four Hours or More YesNo Attitudes Toward Attending College Attitudes Toward School Problem Solving Skills Leadership Skills 12.81*** 19.84*** 17.01*** 17.15*** 12.50*** 18.45*** 15.58*** 15.85*** Students Participated - Four Hours or More YesNo Attitudes Toward Attending College Attitudes Toward School Problem Solving Skills Leadership Skills 12.89*** 19.54*** 16.91*** 17.04*** 12.18*** 18.38*** 15.09*** 15.46*** ***p<.0001 * p<.05 **p< 0.01 ***p<

Mean Scores on Outcome Measurements by Sense of Ownership – Whether or Not Students Made Decisions Students Made Decisions YesNo Attitudes Toward College Attitudes Toward School Problem Solving Skills Leadership Skills 13.04*** 19.59*** 17.26***p 17.51*** 12.36*** 18.84*** 15.58*** 15.81*** * p<.05 **p< 0.01 ***p<

CHART 1. High School Students’ Perception of How Good They are at Speaking in Front of Groups by Whether or Not They Made Decisions about Service Learning Projects (p <0.0001)

CHART 2. High School Students’ Perception of How Good They are at Finding Resources by Whether or Not They Made Decisions about Service Learning Projects (p <0.0001)

Even though I had done research for many years, I learned and “re” learned some important things about evaluation research from this project. Click here to link to power point outlining the things we learned about: 1.The Impact that Timing can have on Results. 2.Asking the “Right” Questions and Selecting the “Right” Indicators. 3.The Impact of Social Desirability on Findings and how to Deal with it. 4.The Importance of using Multi-Methods when doing Evaluation Research.