The Roles Evaluators Play in Providing TA to SPDG Projects 1 Cheryl Leever Huffman C L Huffman & Associates 3316 Eton Avenue Oklahoma City, OK A Conversation

Disclaimer This is a conversation about supporting the staff of the SPDGs we evaluate with TA regarding evaluation as well as other aspects of the project. During our time together, my goal is for us to have the opportunity to explore various roles evaluators may play as part of a SPDG team, the different types of TA evaluators are commonly asked to provide, and factors affecting how they provide TA. 2

POLL I A Little Background On SPDG Evaluators

POLL II Some Background on the SPDG Staff Evaluators Work With

Evaluators Have A Lot On Their Plates Just With Evaluation! 5 Program Evaluation is “…the systematic collection of information about the activities, characteristics, and results of programs to make judgments about the program, improve or further develop program effectiveness, inform decisions about future program development, and/or increase understanding.” – Patton, 2008 Source: Patton, M. Q. (2008). Utilization-focused evaluation (4th ed.). Thousand Oaks, CA: Sage Publications, Inc.

What? So what? Now what? 6 Actually, Here’s My Favorite Definition of Program Evaluation:

No One Person Can Handle Everything Encompassed In That “Formal” Definition…… Whether acknowledged or not, most programs rely on a team of project people to conduct the evaluation. Every person on the “Evaluation Team” takes responsibility for a particular aspect of the evaluation process. 7

In General: Project Director - Overall Responsibility ◦ Hiring evaluator, timeline, liaison, staff training, collecting data, maintaining records, writing parts of reports, sharing reports Staff - Support ◦ Collecting data, maintaining records, participating in developing and conducting assessments, contributing to reports Evaluator - Focus, Analysis and Reporting ◦ Refining evaluation plan, assisting in selecting and/or developing assessments, analyzing data, reporting (periodically, informally and formally), writing major parts of reports 8 Typical Roles in the Evaluation of a Program:

POLL III Which tasks of this “traditional” evaluator role are you asked to provide for your SPDG? 9

Some Non-Traditional Roles Evaluators May Be Asked to Perform for their SPDG (and Some Possible Reasons Why) __General Grants Management Consultant ◦ May never have directed a project – or managed a large budget - before ◦ May not have anyone else to ask ◦ May feel uncomfortable asking someone else __Professional Organizer ◦ May never have had to deal with so much “stuff” before 10

Some Non-traditional Roles (Continued) __Historian/Archivist ◦ May have come into the project in mid cycle ◦ May need a document that they cannot find ◦ May be asked to do something that the evaluator knows didn’t work before __Counselor/Therapist ◦ May need someone to just listen when they get frustrated 11

POLL IV Which, if any, of these non-traditional roles have you been asked to perform? 12

POLL V Which, if any, of these non-traditional roles are you uncomfortable filling for your SPDG? 13

Some Reasons Evaluators Might Feel Uncomfortable with Non- Traditional Roles Don’t have experience in the area a project requests assistance Don’t have time to devote to delivering assistance in an area outside of evaluation Don’t want to get involved in anything that might remotely involve agency “politics” Others????? 14

POLL VI How do you feel about delivering TA in a non-traditional role? 15

A Few Strategies for Handling Requests for TA You Do Not Want to Address Tactfully explain to staff why you do not feel it is within your skill set to assist them. Have a list of resources or colleagues to refer staff to that could address their request. Point out when/why the project director must deal with their supervisor or contact their OSEP program officer to ask a question or resolve an issue 16

So, How Do We Deliver Effective TA- Whether in a Traditional or a Non-Traditional Role? First- Let’s revisit the Traditional Role: Evaluator – Focus*, Analysis and Reporting ◦ Refining evaluation plan, assisting in selecting and/or developing assessments, analyzing data, reporting (periodically, informally and formally), writing major parts of reports *We may, or may not, have talked about the word “Focus” in the introductory line. No matter what kind of TA we provide, we can always look to link it to a focus on Evaluation. 17

Some General Factors Affecting TA Delivery ◦ Proximity of Evaluator to Project’s Main Location – Not usually adjustable! ◦ Evaluator Being Kept Informed of What’s Going on with the Project – Sometimes the evaluator has to be a bit of a nag to be kept in the loop! ◦ Methods of Communication Available – The more choices the better!  Face-to-face/Skype/ /Phone 18

More Factors Affecting TA Delivery ◦ “Tone”- How you say it makes a big difference in how’s it’s heard!  Have you considered…..?  Perhaps you might try….?  Some projects I’ve worked with have used….. ◦ Others???? 19

Is There A Role That Evaluators Play That Transcends Traditional And Non-traditional? The Kellogg Foundation thinks so. In their discussion of the role of the evaluator, they conclude: "Experience tells us, however, that the most important overall characteristics to look for in an evaluator are the ability to remain flexible and to problem solve.” 20 The W.K Kellogg Foundation Evaluation Handbook. W.K. Kellogg Foundation, Battle Creek, MI, 2004.