Evaluation Structure

2| Evaluation – A Multi-layered Approach All AHEC awardees are conducting pre and post tests as well as focus groups with an external facilitator. A National Library of Medicine (NLM) professional evaluator is guiding these efforts. These evaluation tools were approved by NLM’s Institutional Review Board prior to use. ​ Pre and Post Tests ​ Focus Groups ​ Product Content Analysis Some AHEC awardees have capstone products like posters and videos planned as a component of the curriculum. An additional content analysis of these items will be conducted with the help of score sheets and evaluation rubrics.

3| Evaluation – Pre-Tests ​ Pre- and post-tests were designed to evaluate whether concrete knowledge gains occurred for student participants. Because each AHEC’s focus was slightly different, each AHEC’s pre-test was modified slightly to include only relevant lesson material for that group. Consultation and feedback sessions with the NLM evaluator and each group were required to develop their individualized tests. Three of the five groups pilot tested their instruments in order to judge comprehension, appropriateness, and time needed for the test. Feedback on the pre-test experience varied, but not by much. ​ Pre and Post Tests ​ While most groups encountered no difficulties administering the test, one student group struggled and took up to 60 minutes to complete it. Pilot testing (and revising) the instrument with all student groups/regions could possibly have relieved some of this difficulty, but in this case it could not be avoided due to time constraints. There were also contextually distracting circumstances surrounding the pre-test for this group. ​ Several groups expressed appreciation of the editing process for the test, acknowledging the extent to which it provided a more clear, culturally relevant, and user friendly survey. Most AHECs create evaluation instruments in- house with only the benefit of whatever experience staff may have and not that of a collective AHEC effort or professional evaluators.

4| Evaluation – Focus Groups ​ Six focus groups (two for Colorado) will gather information from students that is best described in narrative form like ideas, self- concepts, beliefs and leadership constructs. ​ All groups were given the opportunity to provide feedback on focus group questions that the NLM evaluator drafted. Thoughtful and insightful edits thus produced a more concise and relevant set of questions that are ready for the first focus group in quarter three. Each AHEC will be using the same set of primary questions and will probe student groups with follow-up questions as appropriate. ​ Each external facilitator (identified by the AHEC) was provided a focus group orientation by the NLM professional evaluator. Preparing for the focus group facilitation has been a new experience for several AHEC staffers. ​ Focus Groups

5| Evaluation – Content Analysis ​ The main idea behind a product content analysis is to get a look at how students apply their project learning in a practical assignment. Because these additional capstone products are highly individualized for each AHEC, designing and developing a method for evaluating them will also need to be highly individualized. These methods will be solidified in quarter three. Product content analysis will be more of a bonus evaluation measure since pre/post-tests and focus group results are those items that are considered critically informative. ​ Product Content Analysis

6| Evaluation – Lessons Learned Preparing AHEC awardees for the administrative time needed to provide feedback and test/develop tools will be a program improvement for future initiatives. Preparing AHEC awardees for the time needed to programmatically administer formal evaluation tools with participants will be a program improvement for future initiatives. Evaluation efforts of this kind have secondary functions as accountability and monitoring measures. ​ The project coordinator and NLM evaluator have taken steps to protect groups from paper, feedback, and evaluation burnout. For example, asking only for what is necessary, reducing verbiage, keeping numbers of s in check etc.