1. Undervaluing Evaluation and Evaluators: The Case of the Federal 21st Century Community Learning Centers Program in Texas
American Evaluation Association Conference, 2017
Washington, DC
Dr. Venita Holmes, Houston Independent School District
Dr. Roger Durand, Durand Research and Marketing Assoc., LLC
Mel Waites, Waites Consulting Group

2. Undervaluing Evaluation and Evaluators: The Case of the Federal 21st Century Community Learning Centers Program in Texas
Exclusion of evaluators from critical decisions related to evaluation design, measurement, data collection, dissemination of assessment results, logic model construction, and the involvement of stakeholders
Adverse consequences of exclusions for site- level decision-making and, especially, for the programs’ school-age clients
Corrective recommendations to state and Federal policy and program officials as well as to site-level decision-makers
Focus of Presentation
Based on over 5 years of experience evaluating 3 large scale and 3 small programs in Texas

3. Undervaluing Evaluation and Evaluators:
The Case of the Federal 21st Century Community Learning Centers Program in Texas (Holmes, Durand, & Waites, 2017)
Guiding Research
Sturges (2015) emphasizes that clarifying evaluator roles, identifying intended users, expected subjects, formal agreements are critical to evaluation in order to meet the information needs of diverse stakeholder groups.
Scott-Little, Hamann, and Jurs (2002) meta-analyses found that after-school evaluations reflected “moderate compliance with The Program Evaluation Standards established by the Joint Committee on Standards for Educational Evaluation” (p. 387).
Iritis, Bickel, and Nelson (2005) noted that appropriate data must be available to address evaluation questions, to draw evaluative conclusions, with information from similar programs, sound standards or performance criteria, and/or cost- effectiveness information.

4. Texas ACE Key Evaluation
Undervaluing Evaluation and Evaluators:
The Case of the Federal 21st Century Community Learning Centers Program in Texas (Holmes, Durand, & Waites, 2017)
Texas ACE Key Evaluation
Terms
ACE’s Theory of Action - students in need, spending additional time (min. of 45 days in Cycle 9, and 30 days in Cycle 8) in well- structured and aligned after-school activities, taught by qualified personnel, focused on the four activity components will yield improvement in students’:
(1) academic performance
(2) attendance
(3) behavior
(4) promotion
(5) graduation rates
Critical Success Factors - essential for Texas ACE programs to meet Theory of Action objectives:
(1) students’ and families’ active participation and engagement in learning
(2) students’ and families’ increased sense of involvement in school
(3) use of assessment data to revise/evaluate student services
(4) implementation of strategies learned through training

5. Texas ACE Key Evaluation
Undervaluing Evaluation and Evaluators:
The Case of the Federal 21st Century Community Learning Centers Program in Texas (Holmes, Durand, & Waites, 2017)
Texas ACE Key Evaluation
Resources
Texas ACE Blue Print - Independent Evaluation Guide outlines:
Hiring independent evaluator
Evaluation questions
Analyzing and reporting program outcomes
Logic model
Executive summary
Final report
ACE PRIME Assessment (eliminated) ensured grantees were:
In compliance with all state program requirements
Were on-track to implement high- quality programming
Results were used to develop each grantee’s training and ongoing technical assistance plan for the year

6. Does the State Education Agency
Value the Texas 21st Century Evaluation/Evaluator?
Changing Role of Evaluators/Critical Questions
The new requirement of only an “Executive Summary” rather than a full report
How does this influence the role of the evaluator?
How does this influence program management and accountability at the local and state levels?
Does this limit compliance to the “critical success factor,” which is to evaluate/revise student services based on evaluation findings?
No feedback from the State Education Agency on evaluation reports
What is the impact of no feedback on local and national after-school program policy and funding?

7. Does the State Education Agency
Value the Texas 21st Century Evaluation/Evaluator?
Changing Role of Evaluators/Critical Questions
The PRIME Assessment minimized
Has the evaluator role been enhanced or reduced as a result of the elimination of the Prime Assessment (compliance with high-quality programs, training)?
The State Education Agency may not be providing sufficient program support to improve evaluation
How does this affect program outcomes and evaluation findings?
Minimal involvement of independent evaluators in establishing evaluation guidelines.
Will the newly developed evaluation advisory group initiated for the first time in August 2017 be empowered to establish more rigorous evaluation guidelines?

8. Does the State Education Agency
Value the Texas 21st Century Evaluation/Evaluator?
Changing Role of Evaluators/Critical Questions
Texas ACE Independent Evaluation Guide criteria for “regular” program student changed in Cycle 9 (30 days to 45 days), making it difficult for evaluators to compare findings across grant cycles.
Were rigorous evaluations conducted to justify the change?
The Texas ACE Independent Evaluation Guide cites a statewide evaluation (Naftzger, et al., 2012) that students who participate for 60 and 90 days have an increased chance of grade promotion (p. 9).
Will these data be incorporated in the Tx21st data system and the evaluation guide to improve evaluation practices at the local level?

9. Does the State Education Agency
Value the Texas 21st Century Evaluation/Evaluator?
Changing Role of Evaluators/Critical Questions
The requirement of a higher standard for a “regular” program participant – 30 to 45 days
What is the impact of this requirement on program reimbursement?
How practical is this change for at-risk and for high-school students?
What is the impact of this for evaluators with fewer numbers of “regular” program participants on which to base evaluation findings and recommendations?
How does this change impact cost analyses?
Undervaluing Evaluation and Evaluators:
The Case of the Federal 21st Century Community Learning Centers Program in Texas (Holmes, Durand, & Waites, 2017)

10. Does the State Education Agency
Value the Texas 21st Century Evaluation/Evaluator?
Changing Role of Evaluators/Critical Questions
Independent evaluator pay lowered over the past two years
What is the impact of evaluator pay on the number of evaluators willing to do the work?
What is the impact on the quality of the independent evaluation?
Late start for new grant cycle recipients
How does this affect selection of independent evaluators, timeliness of training, logic model development, program activities, for example?

11. Does the State Education Agency
Value the Texas 21st Century Evaluation/Evaluator?
Changing Role of Evaluators/Critical Questions
The primary data source, the Tx21st data system, does not capture reliable and valid data
Evaluators did not pose these questions nor devise the metrics
The Independent Evaluator Guide does not require evaluators to take into account the influence of students' background characteristics on improvements -- or even declines -- in outcomes.
Non-regular students are the comparison group.
Evaluation questions:
Are there a greater number of students experiencing improvement?
Are there a greater % of students experiencing improvement?
Are there greater amounts of improvements by students?
Metrics: (2 year comparisons - previous and current year)
Grade analysis - reading, math, science, social studies GPA change
School attendance analysis - school days absent
School discipline referral analysis - criminal and non-criminal referrals
Course completion analysis - course pass percentage
Achievement test scores (state test)

12. Alignment of Texas 21st Century Program with Joint Committee on Standards for Educational Evaluation1
Utility Standards - intended to increase the extent to which program stakeholders find evaluation processes and products valuable in meeting their needs.
Feasibility Standards - intended to increase evaluation effectiveness and efficiency.
Propriety Standards -support what is proper, fair, legal, right and just in evaluations.
Accuracy Standards - intended to increase the dependability and truthfulness of evaluation representations, propositions, and findings, especially those that support interpretations and judgments about quality.
Evaluation Accountability Standards - encourage adequate documentation of evaluations and a metaevaluative perspective focused on improvement and accountability for evaluation processes and products.
1Yarbrough, Shulha, Hopson, & Caruthers (2011)

13. Undervaluing Evaluation and Evaluators:
The Case of the Federal 21st Century Community Learning Centers Program in Texas (Holmes, Durand, & Waites, 2017)
Alignment of Texas 21st Century Program with Joint Committee on Standards for Educational Evaluation1
U1 Evaluator Credibility
U2 Attention to Stakeholders
U3 Negotiated Purposes
U4 Explicit Values
U5 Relevant Information
U6 Meaningful Processes & Products
U7 Timely & Appropriate Communicating & Reporting
U8 Concern for Consequences and Influence
+ +
- -
+ +
F1 Project Management
F2 Practical Procedures
F3 Contextual Viability
F4 Resource Use
P1 Responsive and Inclusive Orientation
P2 Formal Agreements
P3 Human Rights & Respect
+ +
+ +
+ +
- -
+ +
- -
+ +
- -
+ +
+ +
U = Utility - valuable
F = Feasibility - effectiveness and efficiency
P = Proprietary - proper, fair, legal, right and just
+ +
+ +
1Yarbrough, Shulha, Hopson, & Caruthers (2011)

14. Undervaluing Evaluation and Evaluators:
The Case of the Federal 21st Century Community Learning Centers Program in Texas (Holmes, Durand, & Waites, 2017)
Alignment of Texas 21st Century Program with 2017 Joint Committee on Standards for Educational Evaluation1 (cont’d)
A5 Information Management
A6 Sound Designs and Analyses
A7 Explicit Evaluation Reasoning
A8 Communication and Reporting
E1 Evaluation Documentation
E2 Internal Metaevaluation
E3 External Metaevaluation
- -
- -
P4 Clarity and Fairness
P5 Transparency and Disclosure
P6 Conflicts of Interests evaluation
P7 Fiscal Responsibility processes
A1 Justified Conclusions and Decisions
A2 Valid Information
A3 Reliable Information
A4 Explicit Program and Context Descriptions
- -
- -
+ +
+ +
+ +
+ +
+ +
- -
- -
- -
- -
- -
P = Proprietary - proper, fair, legal, right and just
A = Accuracy - dependability and truthfulness
E = Evaluation Accountability - adequate documentation
- -
1Yarbrough, Shulha, Hopson, & Caruthers (2011)

15. Summary of Challenges/Negative Outcomes
Alignment of Texas 21st Century Program with 2017 Joint Committee on Standards for Educational Evaluation1
Summary of Challenges/Negative Outcomes
Utility - 2 out of 5 (40%)
Feasibility - 0 out of 4 (0%)
Proprietary - 1 out of 6 (17%)
Accuracy - 6 out of 7 (86%)
Evaluation Accountability - 2 out of 3 (67%)
U = Utility - valuable
F = Feasibility - effectiveness and efficiency
P = Proprietary - proper, fair, legal, right and just
A = Accuracy - dependability and truthfulness
E = Evaluation Accountability - adequate documentation
1Yarbrough, Shulha, Hopson, & Caruthers (2011)

16. Undervaluing Evaluation and Evaluators: The Case of the Federal 21st Century Community Learning Centers Program in Texas
Recommendations
Empowerment evaluation is an approach that seeks to increase the likelihood that programs will achieve success by increasing the capacity of program stakeholders to plan, implement, and evaluate their own programs (Fetterman, 2012).
As Quinn (2011) defines it, “developmental evaluation has the purpose of helping develop an innovation, intervention, or program. In developmental evaluation, the evaluator typically becomes part of the program or innovation design team, fully participating in decisions and facilitating discussion about how to evaluate whatever happens (p. 20).
“Youth participatory evaluation involves young people in the process of evaluating the programs, organizations, agencies, and systems that have been designed to serve them.” (p. 4)  Flores also writes about involving young people as evaluators and developing knowledge.
More alignment with the 2017 Joint Committee on Standards for Educational Evaluation

17. References (cont’d)
Eddy, R.M. & Barry, T. (2009). The evaluator’s role in recommending program closure. American Journal of Evaluation, 30 (3),
Iriti, J. E., Bickel, W.E., W, Nelson, C.A. (2005). Using recommendations in evaluations. American Journal of Evaluation, 26 (4),
Fetterman, D. M. (2012). Empowerment Evaluation in the Digital Villages. Stanford, CA: Stanford University Press. 
Flores, K.S. (2008). Youth Participatory Evaluation: Strategies for Engaging Young People. San Francisco: John Wiley and Sons.
Joint Committee on Standards for Educational Evaluation. (2017). Program Evaluation Standards Statements. Retrieved from, standards-statements
Labin S., Duffy J., Meyers D., Wandersman A., Lesesne C. (2012). A research synthesis of the evaluation capacity building literature. American Journal of Evaluation, 33, 307–338
Leos-Urbel, J. (2013). What works after school?, Youth and Society, 47,

18. References (cont’d)
Naftzger, N., Manzeske, D., Nistler, M., Swanlund, A., Rapaport, A., Shields, J., Sugar, S. (2012). Texas 21st Century Community Learning Centers: Final evaluation report. Naperville, IL: American Institutes for Research. Retrieved from, /expandingminds/article/texas-afterschool-centers- education-ace-achieving-positive-results-and#sthash.buMttDUn.dpu
Newcomer, K. (2004). How we might strength evaluation capacity to manage evaluation contracts. American Journal of Evaluation, 25(2),
Patton, M.Q. (2011). Developmental Evaluation. New York: The Guilford Press, p
Scott-Little, C., Hamann, M.S., & Jurs, S.G. (2002). Evaluations of after-school programs: A meta-evaluation of methodologies and narrative synthesis of findings. American Journal of Evaluation, 23(4),
Simpkins, A.D. and others (2016). Designing culturally responsive sensitive after-school a ctivities, Journal of Adolescent Research, 32, 11-36
Sturges, K. (2015). Complicity revisited. American Journal of Evaluation, 35,
Texas Education Agency. (2017). Texas ACE Blueprint. Independent Evaluation Guide, Retrieved from, blueprint

19. References (cont’d)
Wholey, J. (2001). Managing for results. American Journal of Evaluation, 22(3),
Yarbrough, D.B., Shula, L.M., Hopson, R.K., & Caruthers, F.A. (2010). The Program Evaluation Standards: A guide for evaluators and evaluation users (3rd. Ed). Thousand Oaks, CA: Corwin Press.