Lynn Elinson, Ph.D. Project Director Developmental Disabilities Program Independent Evaluation (DDPIE) Project Lynn Elinson, Ph.D. Project Director

Evaluation Standards* Utility Feasibility Propriety Accuracy *Joint Committee on Standards for Educational Evaluation

Purpose of the Independent Evaluation Demonstrate impact of DD Network programs on: Individuals Families Service providers State systems Provide feedback to ADD to help improve the effectiveness of its programs and policies Promote accountability to the public

Purpose for UCEDDs Identification of accomplishments from external organization Identification of areas that need improvement

DDPIE Project Independent evaluation 2 phases Phase 1 – development and testing of tools Phase 2 – full-scale evaluation Westat – contracted by ADD to implement Phase 1

Evaluation Standards Indicators What do we hope to achieve? What do we observe (measurement of indicators)? Comparison Are there differences/discrepancies? What is the nature and extent of the differences? What action needs to be taken?

Open Systems Model Structure (Input) Process Output (Product) Outcome Effectiveness Structure (Input) Process Output (Product) Outcome Have Hoda re-create this so it looks better on the slide. Inputs are those resources that are needed to set processes in motion and keep them running. Some examples of inputs are staff, policies, resource networks, facilities, and funding. Inputs must be in place before proposed processes can function properly. Processes are those event sequences and arrangements of staff, services, and resources needed to achieve the intended result(s). When inputs are in place and processes are functioning as intended, then outputs and outcomes are produced. Outputs, often referred to as products, are the “units” produced by processes supported by given inputs. An example is the number of staff trained to use particular strategies targeted at meeting the employability needs of people with disabilities. Outcomes refer to the intended results of creating certain outputs/products. While the product is the trained staff, the relevant outcome is the increased capacity of the staff to serve people with disabilities who seek their services. Of course, the ultimate outcome is improvement in the employment outcomes of people with disabilities. Effectiveness is defined as the relationship between the outcomes achieved and the processes used to affect those outcomes. One can ask the question, “Do the unique capacity-building strategies used by a grantee (or the grantees in general) in fact produce the desired outcomes?” The efficiency of capacity-building efforts is evaluated through a comparison of inputs and outputs. Given that outcomes are satisfactory, or remain constant, the relative efficiency can be assessed among different approaches to building system capacity to serve people with disabilities. In this model, efficiency is only relevant if positive outcomes are realized. Efficiency

Basic Evaluation Approach Performance-based approach Development of standards and indicators Measurement of indicators to determine level at which standards are being met Development of measurement matrices that contain standards, indicators, and performance levels (not developed; limited development; adequate development) Determination of overall performance at the national level

Evaluation Tools Measurement matrices - standards - indicators (structures, processes, outputs, outcomes) - performance levels Data collection instruments

INDEPENDENT EVALUATION FRAMEWORK OF INDICATORS DD Act Administered by ADD Programs DD Councils P&As UCEDDs Collaboration KEY FUNCTIONS FRAMEWORK OF INDICATORS STANDARDS PERFOR- MANCE LEVELS Standards will be used as benchmarks to evaluate the indicators, as measured. DD Councils Outreach Informing policymakers UCEDDs Training Comm. serv. Research Dissemination P&As Individual advocacy Outreach/public education Collab. Proj. develop. Proj. implement. Structure Process Output Outcome Level at which a standard is met. Three performance levels for each standard. MEASUREMENT MATRICES Will be developed to organize each program’s key functions, framework of indicators, standards and corresponding indicators, and performance levels for each standard.

Project Tasks and Timing October, 2005 – December, 2006 Collect and review background information. Establish Advisory Panel, Working Groups, and Validation Panels. Develop draft performance standards, indicators, and data collection tools. Work with Validation Panels to finalize matrices for pilot study. Train pilot study staff. January, 2007 – September, 2007 Conduct pilot study in up to 5 states. Write report to ADD with recommendations.

Advisory Panel Self-advocates Family members Representatives from 3 programs - UCEDD: Richard Carroll from Arizona Child/disability advocates Evaluation expert Federal representative (for PAIMI evaluation)

Working Groups 4 Working Groups (UCEDD, P&A, DD Council, Collaboration) Role: To assist Westat in developing draft measurement matrices that will be reviewed and endorsed by Validation Panels Process: In-person and telephone meetings; work offline

Criteria for Selection of UCEDD Working Group Members Director/Associate Director At least one state/jurisdiction has more than one UCEDD program Rural/urban Geographic distribution Reflects different types of UCEDDS (e.g., placement in medical school, school of education, standalone)

Working Group Activities Description of program Discussion and identification of key functions Identification of structures, processes, outputs, and outcomes for each key function Discussion of standards for structures, processes, outputs, and outcomes

What can UCEDDs do? Stay informed Open process - input welcome Participation in pilot study – random selection

UCEDD Working Group Members Carl Calkins, Ph.D. Kansas City, MO calkinsc@umkc.edu Tawara Goode, M.A. Washington, DC tdg2@georgetown.edu Gloria Krahn, Ph.D. Portland, OR krahng@ohsu.edu David Mank, Ph.D. Bloomington, IN dmank@indiana.edu Fred Orelove, Ph.D. Richmond, VA forelove@vcu.edu Fred Palmer, M.D. Memphis, TN fpalmer@utmem.edu Lucille Zeph, Ed.D. Orono, ME Lu.Zeph@umit.maine.edu

Westat Contact Information Lynn Elinson - LynnElinson@westat.com - 412 421-8610 Cynthia Thomas - CynthiaThomas@westat.com - 301 251-4364 Priyanthi Silva - PriyanthiSilva@westat.com - 301 610-5162 Bill Frey - FreyW1@westat.com - 301 610-5198