1. Undervaluing Evaluation and Evaluators: The Case of the Federal 21st Century Community Learning Centers Program in Texas
American Evaluation Association Conference, 2017
Washington, DC
Dr. Venita Holmes, Evaluator
Dr. Roger Durand, Durand Research and Marketing Assoc., LLC
Mel Waits, Waits Consulting Group

2. Undervaluing Evaluation and Evaluators: The Case of the Federal 21st Century Community Learning Centers Program in Texas1
(Texas ACE)
Focus of Presentation
Exclusion of evaluators from critical decisions related to evaluation design, measurement, data collection, dissemination of assessment results, and the involvement of stakeholders
Adverse consequences of evaluator exclusions for site-level decision-making and, especially, for the programs’ school-age stakeholders
Corrective recommendations to state and federal policy and program officials as well as to site-level decision-makers
1Texas ACE™  (Afterschool Centers on Education)

3. Guiding Research
Sturges (2015) - emphasizes clarifying evaluator roles, identifying intended users, expected subjects, and formal agreements are critical to evaluation in order to meet the information needs of diverse stakeholder groups.
Scott-Little, Hamann, and Jurs (2002) - meta-analyses found that after-school evaluations reflected “moderate compliance with The Program Evaluation Standards established by the Joint Committee on Standards for Educational Evaluation” (p. 387).
Iritis, Bickel, and Nelson (2005) - noted that appropriate data must be available to address evaluation questions and to draw evaluative conclusions, incorporating information from similar programs, based on sound standards or performance criteria, and/or cost-effectiveness analyses.

4. Key Evaluation Concepts
Texas ACE™
Key Evaluation Concepts
ACE’s Theory of Action - students in need, spending additional time (min. of 45 days in Cycle 9, and 30 days in Cycle 8) in well- structured and aligned after-school activities, taught by qualified personnel, focused on the four activity components will yield improvements in:
(1) academic performance
(2) attendance
(3) behavior
(4) promotion
(5) graduation rates
Critical Success Factors - essential for Texas ACE programs to meet Theory of Action objectives:
(1) Active participation and engagement of students and families in learning
(2) Increased sense of involvement of students and families in school
(3) Use of assessment data to revise/evaluate student services
(4) Implementation of strategies learned through training

5. Key Evaluation Resources
Texas ACE ™
Key Evaluation Resources
Texas ACE Blue Print - Independent Evaluation Guide:
Hiring independent evaluator
Evaluation questions
Measuring outcomes
Logic model
Executive summary/Final Report
ACE PRIME Assessment (eliminated):
In compliance with all state program requirements
On-track to implement high-quality programming
Develop training and technical assistance plan

6. Does the State Education Agency
Value the Texas 21st Century Evaluation/Evaluator?
Changing Role of Evaluators/Critical Questions
New requirement of only an “Executive Summary” rather than a full report
How does this influence the role of the evaluator?
How does this influence program management and accountability at the local and state levels?
Does this limit compliance to the “critical success factors,” which is to evaluate/revise student services based on evaluation findings?
No feedback from the State Education Agency on evaluation reports
What is the impact on local and national after-school program policy and funding?

7. Does the State Education Agency
Value the Texas 21st Century Evaluation/Evaluator?
Changing Role of Evaluators/Critical Questions
The State Education Agency may not be providing sufficient program support to improve evaluation (eliminated ACE PRIME Assessment)
How does this affect program outcomes and evaluation findings?
Minimal involvement of independent evaluators in establishing evaluation guidelines
Will the newly developed Evaluation Advisory Group initiated in August be empowered to establish more rigorous evaluation guidelines?

8. Does the State Education Agency
Value the Texas 21st Century Evaluation/Evaluator?
Changing Role of Evaluators/Critical Questions
Texas ACE Independent Evaluation Guide criteria for “regular” program student changed in Cycle 9 (30 days to 45 days), making it difficult for evaluators to compare findings across grant cycles
Were rigorous evaluations conducted to justify the change?
Will data be incorporated in the Tx21st data system to improve evaluation practices at the local level?
What is the impact of this for evaluators with fewer numbers of “regular” program participants on which to base evaluation findings and recommendations?

9. Does the State Education Agency
Value the Texas 21st Century Evaluation/Evaluator?
Changing Role of Evaluators/Critical Questions
Independent evaluator pay lowered over the past two years
What is the impact of evaluator pay on the number of evaluators willing to do the work?
What is the impact on the quality of the independent evaluation?
Late start for new grant cycle recipients
How does this affect selection of independent evaluators, timeliness of training, logic model development, and program activities, for example?

10. Does the State Education Agency
Value the Texas 21st Century Evaluation/Evaluator?
Changing Role of Evaluators/Critical Questions
The primary data source, the Tx21st data system, does not capture reliable and valid data.
Evaluators did not pose evaluation questions nor devise the metrics.
The Independent Evaluator Guide does not require evaluators to take into account the influence of students' background characteristics on improvements -- or even declines -- in outcomes.
Non-regular students (spent less than 30 or 45 days in the program) may not be an appropriate comparison group.

11. An Assessment of the Texas ACE ™ Evaluation Process based on
Education Evaluation Standards2
Utility Standards - intended to increase the extent to which program stakeholders find evaluation processes and products valuable in meeting their needs.
Feasibility Standards - intended to increase evaluation effectiveness and efficiency.
Propriety Standards - support what is proper, fair, legal, right and just in evaluations.
Accuracy Standards - intended to increase the dependability and truthfulness of evaluation representations, propositions, and findings, especially those that support interpretations and judgments about quality.
Evaluation Accountability Standards - encourage adequate documentation of evaluations and a metaevaluative perspective focused on improvement and accountability for evaluation processes and products.
2 Joint Committee on Standards for Educational Evaluation

12. An Assessment of the Texas ACE ™ Evaluation Process
based on
Education Evaluation Standards2
U1 Evaluator Credibility
U2 Attention to Stakeholders
U3 Negotiated Purposes
U4 Explicit Values
U5 Relevant Information
U6 Meaningful Processes & Products
U7 Timely & Appropriate Communicating & Reporting
U8 Concern for Consequences and Influence +
F1 Project Management
F2 Practical Procedures
F3 Contextual Viability
F4 Resource Use
P1 Responsive and Inclusive Orientation
P2 Formal Agreements
P3 Human Rights & Respect + - + + + + + - + - + - + + +
U = Utility - valuable
F = Feasibility - effectiveness and efficiency
P = Propriety - proper, fair, legal, right and just
2 Joint Committee on Standards for Educational Evaluation

13. An Assessment of the Texas ACE ™ Evaluation Process
based on
Education Evaluation Standards2
A5 Information Management
A6 Sound Designs and Analyses
A7 Explicit Evaluation Reasoning
A8 Communication and Reporting
E1 Evaluation Documentation
E2 Internal Metaevaluation
E3 External Metaevaluation -
P4 Clarity and Fairness
P5 Transparency and Disclosure
P6 Conflicts of Interests evaluation
P7 Fiscal Responsibility processes
A1 Justified Conclusions and Decisions
A2 Valid Information
A3 Reliable Information
A4 Explicit Program and Context Descriptions - - + - + + + + - - - - - -
P = Propriety - proper, fair, legal, right and just
A = Accuracy - dependability and truthfulness
E = Evaluation Accountability - adequate documentation
2 Joint Committee on Standards for Educational Evaluation

14. An Assessment of the Texas ACE ™ Evaluation Process based on
Education Evaluation Standards2
Assessment Results
U = Utility - valuable
F = Feasibility - effectiveness and efficiency
P = Propriety - proper, fair, legal, right and just
A = Accuracy - dependability and truthfulness
E = Evaluation Accountability - adequate documentation
2 Joint Committee on Standards for Educational Evaluation

15. Undervaluing Evaluation and Evaluators: The Case of the Federal 21st Century Community Learning Centers Program in Texas
Recommendations
Empowerment evaluation seeks to increase the likelihood that programs will achieve success by increasing the capacity of program stakeholders to plan, implement, and evaluate their own programs (Fetterman, 2012).
“Developmental evaluation has the purpose of helping develop an innovation, intervention, or program. In developmental evaluation, the evaluator typically becomes part of the program or innovation design team, fully participating in decisions and facilitating discussion about how to evaluate whatever happens" (Quinn, 2011, p. 20).
“Youth participatory evaluation involves young people in the process of evaluating the programs, organizations, agencies, and systems that have been designed to serve them.” (p. 4)  Flores also writes about involving young people as evaluators and developing knowledge.
More alignment with the 2017 Joint Committee on Standards for Educational Evaluation

16. References
Eddy, R.M. & Barry, T. (2009). The evaluator’s role in recommending program closure. American Journal of Evaluation, 30(3),
Iriti, J. E., Bickel, W.E., W, Nelson, C.A. (2005). Using recommendations in evaluations. American Journal of Evaluation, 26(4),
Fetterman, D. M. (2012). Empowerment Evaluation in the Digital Villages. Stanford, CA: Stanford University Press. 
Flores, K.S. (2008). Youth Participatory Evaluation: Strategies for Engaging Young People. San Francisco: John Wiley and Sons.
Joint Committee on Standards for Educational Evaluation. (2017). Program Evaluation Standards Statements. Retrieved from
Labin S., Duffy J., Meyers D., Wandersman A., Lesesne C. (2012). A research synthesis of the evaluation capacity building literature. American Journal of Evaluation, 33, 307–338.

17. References (cont’d)
Leos-Urbel, J. (2013). What works after school?, Youth and Society, 47,
Naftzger, N., Manzeske, D., Nistler, M., Swanlund, A., Rapaport, A., Shields, J., Sugar, S. (2012). Texas 21st Century Community Learning Centers: Final evaluation report. Naperville, IL: American Institutes for Research. Retrieved from /expandingminds/article/texas-afterschool-centers-education-ace-achieving-positive-results-and#sthash.buMttDUn.dpu
Newcomer, K. (2004). How we might strength evaluation capacity to manage evaluation contracts. American Journal of Evaluation, 25(2),
Patton, M.Q. (2011). Developmental Evaluation. New York: The Guilford Press, p
Scott-Little, C., Hamann, M.S., & Jurs, S.G. (2002). Evaluations of after-school programs: A meta-evaluation of methodologies and narrative synthesis of findings. American Journal of Evaluation, 23(4),

18. References (cont’d)
Simpkins, A.D. and others (2016). Designing culturally responsive sensitive after-school activities, Journal of Adolescent Research, 32,
Sturges, K. (2015). Complicity revisited. American Journal of Evaluation, 35,
Texas Education Agency. (2017). Texas ACE Blueprint. Independent Evaluation Guide, Retrieved from
Wholey, J. (2001). Managing for results. American Journal of Evaluation, 22(3),
Yarbrough, D.B., Shula, L.M., Hopson, R.K., & Caruthers, F.A. (2010). The Program Evaluation Standards: A guide for evaluators and evaluation users (3rd. Ed). Thousand Oaks, CA: Corwin Press.