ATTC Network Evaluation 1 National Evaluation of the Addiction Technology Transfer Center (ATTC) Network: ATTC Event & Activity Reporting Database Review Roy M. Gabriel, Ph.D. Jeffrey R.W. KnudsenMargaret Gwaltney, M.B.A.Richard Finkbiner, Ph.D. RMC Research CorporationAbt Associates, Inc.MANILA Consulting Group Portland, ORBethesda, MDMcLean, VA Presentation to ATTC Directors November 5, 2009

ATTC Network Evaluation 2 Agenda The ATTC Event & Activity Reporting Database:  Review of past action steps  The NO, NET, and the ATTC Network  Implementation to date  Examination of current data  All events vs. GPRA only  Role of these data in overall evaluation  Next steps???

ATTC Network Evaluation 3 Review— Where We’ve Been

ATTC Network Evaluation 4 ATTC Event & Activity Reporting Database and the NET ATTC Event & Activity Reporting Database was augmented to meet multiple reporting needs: National Office/CSAT, NIDA, and the national evaluation. In coordination with ATTC National Office, the NET developed a web-based query system to extract and summarize information from the database. Examples:  Query the entire Network, specific region(s), time periods, event types, etc.  Tally number of events, number of participants, etc., as ATTC “outputs” (both GPRA and non-GPRA).  Cross-tab tallies w/many other event characteristics (e.g., funding source, materials source, GPRA/non-GPRA, collaborator type, delivery mode, etc.).

ATTC Network Evaluation 5 The Underlying Premise... During the design phase of the ATTC Evaluation contract, ATTC Directors clearly indicated that, from a service provision perspective, GPRA data did not tell the entire story. It was clear that other very important services were being provided by ATTC centers, but were not being counted by the GPRA system.

ATTC Network Evaluation 6 The Output is Only as Good as the Input... Data Quality The NET presented a series of sample summary tables at the December 2008 ATTC Directors meeting. It was determined that assuring useful (and accurate) data queries and reports is contingent on the NET and the Network working together:  The ATTC NO and the NET hosted a 2-part webinar series in March 2009 reviewing (a) the purpose of the Event & Activity Reporting Database (from the perspectives of CSAT, NIDA, and the NET) and (b) data entry procedures.  The ATTC Performance Monitoring Subcommittee was working to (a) re-disseminate activity/event definitions and to (b) establish rules of thumb regarding what non-GPRA events should be entered into the system.  The NO provided draft guidelines for review (April 2009).

ATTC Network Evaluation 7 NET Reporting Plans—On Hold The NET prepared templates for quarterly summaries of events/activities to each regional ATTC.  Reports were to feature regional activity data along with a Network-wide summary of the same information.  The NET had hoped such summary reports would increase data utilization within Network.  Production and dissemination were contingent upon more consistent implementation of data system.  The NET does not want to issue reports if numbers are misleading, skewed, or simply inaccurate.

ATTC Network Evaluation 8 Implementation of Event & Activity Reporting Database January 1–June 30, 2009

ATTC Network Evaluation 9 GPRA vs. Non-GPRA Events GPRA?# of Events% of EventsValid % of Events (Not Specified) a 294% Yes44359%62% No27337%38% Total b 745 a Note: This variable is only required once an event has occurred. “Not specified” responses likely represent event records not updated after the event took place. b Note: Unduplicated event count.

ATTC Network Evaluation Illustrative NET Interpretation In a 6-month period, the Network conducted nearly 750 events and activities.  An average of about 50 events/activities per region.  Much variability in these numbers from region to region. Nearly 40% of these are not captured by the GPRA reporting system. “Natural” follow-up questions:  What kind of events?  Provided to whom? With whom?  On what topics? 10

ATTC Network Evaluation 11 Variability in Entering Non-GPRA Events 3 Regional Centers account for 70% of all non-GPRA events entered into database.  These 3 Regional Centers demonstrated a 25% GPRA/ 75% non-GPRA balance.

ATTC Network Evaluation NET Observations/Questions Is it accurate that:  3–4 regions conducted no non-GPRA events in these quarters?  The majority of regions conduct fewer than 5 non-GPRA events in a quarter?  In contrast to the earlier slide that indicated, “Network-wide,” nearly 40% of the events conducted fall in the non-GPRA category. What is the balance between GPRA and non-GPRA events in the Network: 60%/40%? 25%/75%? Something in between? 12

ATTC Network Evaluation 13 A Closer Look at Data: Event Type YR2, Q2–3

ATTC Network Evaluation 14 Event Type (n = 745)

ATTC Network Evaluation 15 Event Type—GPRA Only (n = 443)

ATTC Network Evaluation 16 Event Type—Non GPRA Only (n = 273)

ATTC Network Evaluation 17 Event Types for 3 Centers Consistently Logging Both GPRA & Non GPRA Events (n = 256)

ATTC Network Evaluation 18 A Closer Look at Data: Collaborator Type YR2, Q2–3

ATTC Network Evaluation 19 Collaborator Type (n = 745 events)

ATTC Network Evaluation 20 Collaborator Type—GPRA Only (n = 443)

ATTC Network Evaluation 21 Collaborator Type—Non GPRA Only (n = 273)

ATTC Network Evaluation 22 Collaborator Types for 3 Centers Consistently Logging Both GPRA & Non GPRA Events (n = 256 events)

ATTC Network Evaluation 23 Many More Examples Available... Previous slides provided only as illustrative examples. Many other queries could have also been used to demonstrate the impact of non-GPRA data, for example:  Primary funding source  Tech transfer objective  Delivery mode  Target audience  State-level service saturation

ATTC Network Evaluation 24 What This Means... The inclusion of non-GPRA data changes the overall service delivery “picture” for the Network.  This is not a revelation, nor unexpected; but nonetheless, from the NET’s perspective, very important to the Network. The change in picture is even more apparent when looking at those few Centers who seem to be higher implementers (or reporters) of non-GPRA activities. However, given the variability in reporting, the NET can only confidently put forth the service delivery picture drawn from GPRA.  When 70% of non-GPRA entries are coming from 20% of the Network, the climate for skew exists.

ATTC Network Evaluation Importance of Event & Activity Reporting Database in National Evaluation The NET has designed a comprehensive series of data collection activities to:  Address and assess multiple objectives of tech transfer in the Network.  Obtain a variety of perspectives on the effectiveness and value of the ATTC Network. Across all of these data collection activities, the database is the only one that can capture the sheer volume of work produced by the Network (and volume is valued by many key stakeholders). 25

ATTC Network Evaluation 26 Next Steps The NET will run all developed queries on GPRA events entered into ATTC Event & Activity Reporting Database to learn more about planning processes, partners, funding, materials, objectives, etc. of ATTC provided services. If the NET can determine that entry of non- GPRA events/activities has improved, pool of events targeted for inquiry will be expanded.