1. Users Perspectives of the Factors that Influence Whether Recommendations get Implemented Shevaun Nadin & Bernadette Campbell Paper presented at the American Evaluation Association’s Annual Conference: Evaluation 2011

2. Introduction The Dreaded “Dusty Shelf”!! Not a new issue. Thankfully, many frameworks and guidelines exist to guide practice. But, what has not been explored extensively are implementers views. What do those responsible for implementation think about the issue?

3. Research Questions Given the variety of variables in the evaluation literature:  What do implementers think are the most important for getting recommendations implemented?  Are there different points of view?  Are there areas of consensus?

4. Methodology Q Methodology Useful for revealing and articulating different social perspectives, attitudes, etc. and noting similarities among them. Task is the “Q Sort”. I.e., Participants rank order a set of statements that represent the variety of ideas about a topic. Factor analysis then reveals different points of view, and consensus about the topic.

5. Procedure Undertook systematic review of evaluation use & recommendations literature. Resulted in 30 representative statements about factors that might influence recommendation uptake. Diverse sample of participants (n=15) rank ordered the statements according to their experiences.

6. Least Neutral Most Important to Important Uptake to Uptake -4-3-201234

7. Results Are there different points of view?: Analysis revealed 4 factors. Thus, four distinct points of view about what importantly influences recommendation uptake. Data analysis is in a preliminary stage. Factors need to be defined.

8. Results Are there areas of agreement among the views? 4 statements of consensus

9. Areas of Consensus StatementFactor (Ranking) ABCD Involving stakeholders in the evaluation process.2133 Whether program personnel care about the evaluation, and are responsible for /committed to it’s use. 3242 Effective & ongoing communication between you and the evaluator throughout the evaluation process. -201 The quality, attractiveness, and format of the evaluation report. -3-4 -3

10. Conclusions & Implications At least 4 distinct points of view exist. And there are areas of consensus among them. Theoretical contributions: An empirical classification study. Classifying different types of users/implementers. Practical contributions: Empirical results to guide practice.  E.g.: What “type” of user am I dealing with?; When resources limited, focus on involvement over a pretty report; etc.

11. Thank You!! Contact Us: Shevaun Nadin snadin@connect.carleton.ca & Bernadette Campbell bernadette_campbell@carleton.ca Department of Psychology Carleton University Ottawa, Canada