2006 ANNUAL MEETING Arlington, Virginia Evaluators' Breakout II: EVALUATING THE EVALUATOR 10:00 - 11:15 am Friday, January 13, 2006 2006 ANNUAL MEETING Arlington, Virginia INDUSTRY / UNIVERSITY COOPERATIVE RESEARCH CENTERS PROGRAM NATIONAL SCIENCE FOUNDATION

EVALUATING THE EVALUATOR ERIC SUNDSTROM NSF IUCRC Evaluator University of Tennessee DENIS GRAY NSF IUCRC Evaluation Program North Carolina State University &

Why This Session? Questions from Center Directors at January, 2005, IUCRC meeting, and... After all, we're Evaluators, it only figures that we'd eventually look at evaluating what we do...

Goals for Today Explore the problem of evaluating IUCRC Evaluators Frame the problem, not attempt to propose a solution Address 3 questions: 1) Who evaluates Evaluators? 2) Based on what? 3) Evaluate how? Identify next steps

1. Who evaluates IUCRC Evaluators? Evaluators ultimately report to NSF IUCRC Program Manager, Alex Schwarzkopf. Evaluators also serve multiple customer-groups: - IUCRC Evaluation Program at NCSU (Denis) - Center Director & Site Co-Directors - Center Operations Managers / Administrators - Industry Member Representatives - Center Scientists & Graduate Students - Host University Administrators - Fellow IUCRC Evaluators

Clarifying who evaluates IUCRC Evaluators: What purposes does the IUCRC evaluation effort serve? Who is the primary customer for the IUCRC evaluation?

2. Evaluate based on what? NSF and other counterparts can evaluate an IUCRC Evaluator's performance based on: How the Evaluator performs the role: - Completeness of data-collection & reporting - Timeliness of data-collection, analysis, reporting - Quality of evaluation reports - Etc. What the Evaluator role entails – evaluation Protocol: - Focus & methods of data-collection - Design & validity of questionnaires, other measures - Timing of data-collection, analysis, reporting - Structure & content of evaluation reports

Role of Evaluator - Main Elements Program evaluation: Data-collection, analysis, reporting Liaison & linkage: Communication with constituencies Other duties, adopted or negotiated

Role of NSF IUCRC Evaluator: Current Protocol Program evaluation - Data-collection, analysis, reports: - Annual Evaluator's Report - Industry & Faculty Questionnaires: Collect, analyze, report - LIFE Forms: Manage distributing, transcribing, IAB review - Exit interviews with Industry Member representatives Liaison & linkage - Communication with constituencies: - Center Industrial Advisory Board meetings, Spring & Fall - Annual NSF IUCRC Program meeting, January - Annual NSF IUCRC Evaluators' meeting, June Other duties, adopted or negotiated: - Consultant / Coach - Meeting Facilitator / Leader - Mediator, etc.

How well does the evaluation protocol meet the needs of the “customer” (NSF)? How important and useful for the customer is the information collected via the protocol? Evaluator Report Responses to Industry & Faculty Questionnaires LIFE forms Exit interviews What factors affect the usefulness of the information? Specific information collected Burden of data collection Sources of the information Timeliness of feedback How could the value of the information be enhanced?

3. Evaluate how? Evaluation of Evaluators incorporates content – basis or questions for evaluation – and a process or procedure. Content: - Is the Evaluator doing the right job? (correct protocol) - How well is the Evaluator doing that job? (following protocol) Process: Open question - Must support & align with the NSF IUCRC Program - Involves multiple counterparts - Other features?

Next Steps Review and evaluate the current Evaluator protocol Consider re-designing the protocol Explore options for revising the current role Consider having Evaluators do self-evaluation via an Evaluator Practices Questionnaire Others?

Questions & Discussion?

What could be done to improve the protocol? Make evaluation tools shorter, more efficient and focused Get higher response rates & make feedback to director quicker Industry P/O  Industry feedback tool (one pager) Provide LIFE form analysis: Faculty P/O  Faculty feedback; switch to students after year 5 Create alternative tools: focus group guides Continue to encourage and support periodic research projects that would be implemented program-wide

JUNE EVALUATOR MEETING June 14 - 16, 2006 Holiday Inn Rosslyn / Key Bridge Arlington, VA

Goal for Today: Identify Preferences for... Prospective speaker(s) Session topics Task Forces

Prospective Speakers MARIANN ("SAM") JELINEK - College of William & Mary, Williamsburg, VA. - Innovation, technology & organization; former Director of NSF's Innovation & Organizational Change Program. DAVID ("DAVE") ROESSNER - SRI International, Arlington, VA. - Evaluation of Science & Technology; former NSF IUCRC Evaluator. SCOTT SHANE - Case Western Reserve University - Innovation & Entrepreneurship; speaker at earlier IUCRC Evaluators' meeting. BILL VALDEZ - Dept. of Energy, Washington DC, Director, Dept. of Planning & Analysis. - Founder, Washington Research Evaluation Network (WREN) consortium of R&D evaluators.

Session Topics at June Evaluator Meeting? Evaluation of R&D for Innovation, Advancing Technology, ... Report of 2005 Process / Outcome Results & Trend Analysis Evaluator Research: Results from Current Projects; New Projects What Constitutes Success for a NSF IUCRC? (...for a consortium?) New (Fine-tuned? Upgraded? Redesigned?) Evaluator Protocol Evaluator's Report: Guidelines for Analysis (+ Recommendations?) LIFE Form Procedure: Current Practice & Variations (& next steps?) Evaluator Practices Questionnaire: Draft, process (& 1st Results?) IAB Meeting Structure: Theme & Variations Others?

Proposed Evaluator Task Forces Planning & Program for June Evaluator Meeting (Eric, Denis, ... & Kate Ryan) Evaluator Protocol - Review & Update (Denis, David M., ... ) Evaluator Research Projects & Agenda (Dave T., ...) LIFE Form Practices - Current & Next (Otto, ... ) Evaluator Practices Questionnaire & Process (Craig, Don, ... )