1. A Recipe With Many Ingredients:
Design and Implementation of Culturally Responsive Monitoring and Evaluation System for a Multisite Training Program
Gloria Santa Anna, Sonia Lindop, Chris Spicer, Ricardo Gómez

2. MassTERI Massachusetts Training, Evaluation, and Research Initiative
MassTERI contributes to the advancement of women and men in the school nutrition industry by providing innovative, relevant, and cost- effective training and professional development solutions .

3. Intermediate/long-term outcomes Professional Development
MassTERI
Component
Short term outcomes
Intermediate/long-term outcomes
Research
Diagnostic of current situation of SNP
Increased awareness among policy makers
A better prepared workforce that supports the nutritional needs of students
Professional Development
Increased skills and competencies of SNP
MassTERI
Dissemination
Increased visibility of program
Increased awareness of professional development for SNP
Evaluation
Improved effectiveness of curriculum, training, and materials
Worth and value of the program

4. Locations

5. Professional Development
Leadership: supervisory skills, language and cultural sensitivity, self- awareness, team and group dynamics, problem solving and conflict management.
English as a Second Language: Reading, writing, listening and communication in the workplace for employees of all language backgrounds.
Computer Skills: We teach and assess computer concepts and skills so that people can use computer technology in the workplace and in everyday life. Our courses include Word, Excel, and internet.

6. Participatory evaluation: the role of the Site Committee
Members of the school nutrition community
Discuss findings
Plan training and evaluation activities
Share and disseminate results

7. Kirkpatrick’s evaluation model
Longer term; Has the training had an impact on
the organization/business?
Outcome evaluation
Are the acquired skills being used in
the everyday environment of the learner?
Have participants advanced in skills, knowledge,
or attitude?
Evaluation of training
What do they think about the training?

8. Evaluation of training model
Pre-training
Needs assessment
Q method
Self-assessment
Start
Baseline
Intake survey
Pre-test
Middle
Formative
Survey
Group interviews
Finish
Summative
End-of-course test
Survey
Retrospective pretest

9. Retrospective Pre-test Post-test
Training
Pretest
Retrospective pretest
Post test
How good are you at listening to the needs of your staff?
How good were you at listening to the needs of your staff?
How good are you at listening to the needs of your staff?

10. Q methodology

11. Levels of evaluation and tools
1: Reaction
2: Learning
3: Behavior
4: Results
Survey X
Questionnaires
Interviews/Focus groups
Tests
Case studies
Performance reviews
Skills/behavior observation
Action planning
Action Learning
Business metrics
Retrospective pre-post questionnaire

12. What have we learned? Myths busted...
“ The majority of these workers have second jobs.”
25% reported having a second job
“Only a small group of ‘motivated employees’ are interested in training…most SNP view training as a waste of time and attended only when mandatory.”
73% of the participants, including both PT and FT employees, wanted to receive training
“Older employees (>50 yo) are more resistant to change and not interested in training.”
People of all ages interested in receiving training. No statistical difference between age groups.

13. Robust tools for lean evaluation budgets
LimeSurvey (
Q methodology (
R (
gnuplot (
Scribus (
LaTex (
dotProject (

14. Bar chart with error bars in R (ggplot2)

15. R Graphs with ggplot2

16. R graphs (cont.)

17. Dot plot with impulses in gnuplot

18. Dot plot with confidence intervals in gnuplot

19. Sample gnuplot code #x y xlow xhigh location #56 1 49 60 Site-1
set yrange [6:0]
set xrange [1:100]
set key off
set grid
set ylabel "Location"
set xlabel "Math score (mean with 95% CI)"
set title "Math scores at five locations“
set arrow 1 from 52,0 to 52,6 nohead
plot "errors2.txt" u 1:2:3:4:yticlabels(5) w xerrorbars lt 7

20. Thank You! To know more… Gloria Santa Anna, Project Director
Thank You!