A Recipe With Many Ingredients: Design and Implementation of Culturally Responsive Monitoring and Evaluation System for a Multisite Training Program Gloria Santa Anna, Sonia Lindop, Chris Spicer, Ricardo Gómez

MassTERI Massachusetts Training, Evaluation, and Research Initiative MassTERI contributes to the advancement of women and men in the school nutrition industry by providing innovative, relevant, and cost- effective training and professional development solutions .

Intermediate/long-term outcomes Professional Development MassTERI Component Short term outcomes Intermediate/long-term outcomes Research Diagnostic of current situation of SNP Increased awareness among policy makers A better prepared workforce that supports the nutritional needs of students Professional Development Increased skills and competencies of SNP MassTERI Dissemination Increased visibility of program Increased awareness of professional development for SNP Evaluation Improved effectiveness of curriculum, training, and materials Worth and value of the program

Locations

Professional Development Leadership: supervisory skills, language and cultural sensitivity, self- awareness, team and group dynamics, problem solving and conflict management. English as a Second Language: Reading, writing, listening and communication in the workplace for employees of all language backgrounds. Computer Skills: We teach and assess computer concepts and skills so that people can use computer technology in the workplace and in everyday life. Our courses include Word, Excel, e-mail and internet.

Participatory evaluation: the role of the Site Committee Members of the school nutrition community Discuss findings Plan training and evaluation activities Share and disseminate results

Kirkpatrick’s evaluation model Longer term; Has the training had an impact on the organization/business? Outcome evaluation Are the acquired skills being used in the everyday environment of the learner? Have participants advanced in skills, knowledge, or attitude? Evaluation of training What do they think about the training?

Evaluation of training model Pre-training Needs assessment Q method Self-assessment Start Baseline Intake survey Pre-test Middle Formative Survey Group interviews Finish Summative End-of-course test Survey Retrospective pretest

Retrospective Pre-test Post-test Training Pretest Retrospective pretest Post test How good are you at listening to the needs of your staff? 0 1 2 3 4 5 How good were you at listening to the needs of your staff? 0 1 2 3 4 5 How good are you at listening to the needs of your staff? 0 1 2 3 4 5

Q methodology

Levels of evaluation and tools 1: Reaction 2: Learning 3: Behavior 4: Results Survey X Questionnaires Interviews/Focus groups Tests Case studies Performance reviews Skills/behavior observation Action planning Action Learning Business metrics Retrospective pre-post questionnaire

What have we learned? Myths busted... “ The majority of these workers have second jobs.” 25% reported having a second job “Only a small group of ‘motivated employees’ are interested in training…most SNP view training as a waste of time and attended only when mandatory.” 73% of the participants, including both PT and FT employees, wanted to receive training “Older employees (>50 yo) are more resistant to change and not interested in training.” People of all ages interested in receiving training. No statistical difference between age groups.

Robust tools for lean evaluation budgets LimeSurvey (www.limesurvey.org) Q methodology (http://qmethod.org/about) R (http://www.r-project.org/) gnuplot (http://www.gnuplot.info/) Scribus (http://www.scribus.net/canvas/Scribus) LaTex (http://www.latex-project.org/) dotProject (http://www.dotproject.net/)

Bar chart with error bars in R (ggplot2)

R Graphs with ggplot2

R graphs (cont.)

Dot plot with impulses in gnuplot

Dot plot with confidence intervals in gnuplot

Sample gnuplot code #x y xlow xhigh location #56 1 49 60 Site-1 set yrange [6:0] set xrange [1:100] set key off set grid set ylabel "Location" set xlabel "Math score (mean with 95% CI)" set title "Math scores at five locations“ set arrow 1 from 52,0 to 52,6 nohead plot "errors2.txt" u 1:2:3:4:yticlabels(5) w xerrorbars lt 7

Thank You! To know more… Gloria Santa Anna, Project Director gasantaanna@admin.umass.edu Thank You!