UCEDD Response to DDPIE June 1, 2007 Double Tree Eleven Purpose: to summarize the small group feedback on the DDPIE evaluation plan

UCEDD PM Work Group for DDPIE Responses--Purpose Primary audience for this evaluation is OMB, secondary audience may be Congress, programs, public Need to clarify what is needed for the PART process—keep this evaluation focused on that requirement Evaluate DD Act programs—this feedback focuses on the UCEDD network

UCEDD PM Work Group for DDPIE Responses--Approach Recommendations: Base the evaluation on the DD Act; to the extent possible, be cognizant of the work on reauthorization Align the evaluation with compliance with DD legislation (“are programs doing what they are funded to do?”) Evaluate systems impact of the network (“is the legislative intent of the DD Act being accomplished?”)

UCEDD PM Work Group for DDPIE Responses--Approach Clarify what qualifies as “independent” evaluation for OMB (e.g., recognize that independent evaluation may only require primary data collection to answer those questions for which you don’t already have reliable, valid data) In keeping with OMB policy, minimize the burden of data collection on grantees—use existing data sources that we have already invested in (e.g., MTARS, NIRS, annual progress reports).

UCEDD PM Work Group for DDPIE Responses--Approach The present proposal (scope and cost) seems disproportionate to the requirement Use a criterion-based, stratified sampling method of approximately 20-30% of UCEDDs

UCEDD PM Work Group for DDPIE Responses--Methods Add rigorous, qualitative evaluation to tell the story (e.g., multiple case studies, document analysis, focus groups, key informant interviews) Ensure transparency of the process at all levels—communication on steps all along the way Validation needs to occur sooner in the process

UCEDD PM Work Group for DDPIE Responses--Methods Assessment of collaboration needs to include collaboration beyond other ADD-funded partners (i.e. leveraging and systems change requirements of the DD Act)

UCEDD PM Work Group for DDPIE Responses--Measures Reduce the number of measures (benchmarks and performance standards); recommend two for each core function and one for collaboration—select those applicable to most UCEDDs Eliminate measures of structure and process Develop data definition for each measure and evaluate it in terms of applicability of the majority of the network

UCEDD PM Work Group for DDPIE Responses--Measures In developing the benchmarks, use consistent language and framework (e.g., “the network of UCEDDs…”) Focus now on design and benchmarks—wait until there is agreement on benchmarks before reviewing indicators

Next Steps Share recommendations with UCEDD Directors and obtain consensus Discuss recommendations with Westat and ADD Support ongoing work on Benchmarks, Indicators and Performance Standards