Evaluation Working Group (EWG) Update November 2009

EWG History & Operations Started in 2007, less active in 2008 Reactivated in 2009 with hiring of NNE Teleconference every week (or so): [Monday at noon PDT (3p.m. EDT)] Leads: Rosanne Fortner & Chris Parsons Members: 3 COSEE PIs, 6 COSEE Evaluators, 1 COSEE Advisor, CCO, NNE & NSF

EWG Mission To serve the COSEE Network, its sponsors, Center Evaluators (CE) and National Network Evaluator (NNE) by providing information & advice on: - collective evaluation needs - existing resources and collaborative opportunities that document the extent to which COSEE goals and objectives are achieved, as well as the Network’s impacts

EWG Tasks (1) A forum for identifying collaborative opportunities, which includes: Assist the NNE with identification of and mechanisms for integrating specific evaluation research being conducted in the Centers; Establish the scope of CE input to the portfolio of COSEE evaluation products, research and lessons learned; Establish with NNE the standards for reporting existing or new “returns on investment” data being collected by the CEs; Assist in brokering relationships among CEs and between the NNE and CEs; and Provide logistical support for projects involving multiple CEs and NNE. (2) Periodic review and assessment of NNE plans

“Bins” Document History: A Scheme for Cross-Center Data Gathering Related to Task: Establish with NNE the standards for reporting existing or new “returns on investment” data being collected by the CEs. “Bins” Document: Scheme for Cross-Center Data Gathering - Part I: Definitions and Demographic Data (Scientist/Educator Engagement) - Part II: Bins for Activities & Participants Written, reviewed, discussed, reorganized, revised and some piloting About a year in the making; consideration of Returns on Investment and Business Plan; at least 8 drafts

“Bins” Document – Why? As a network, we need a standardized way of collecting common data across Centers We need this data to be able to: – Answer questions about what we’re doing (internal and external questions) as a network – Provide data (part of our evidence package) for the Decadal Review See page 34 of Briefing Booklet

Information gathered across Centers should… be easy enough to gather result in information and findings that are compelling be credible be cost effective

EWG Request of Council We’re asking you to – Approve the Bins document as COSEE’s framework for cross-Center data collection – Tell us if we missed anything – Are there any glaring errors or omissions in this document – Commit to cross-Center data collection – Allocate time/resources to cross-Center data collection efforts

EWG Next Steps re: Bins Starting work on establishing protocols and implementation plan for cross-Center data gathering Working with NNE to set up data base for collected data

Some questions framing the next phase of EWG “Bins” work When do we start? What is the reporting cycle? What is the data- gathering time frame? Who within COSEE will fill in the data? How do we come up with an agreed upon electronic format? How do we pilot this? How do we train and who do we train to do this? How will data be reported? Who will do the reporting? Who will have access to the reports? How will or could this serve Center needs vs. Network needs over time? Will it be revisited over time? How does the Bins data collection relate to Human Subjects/IRB? How will we know if the data gathering meets the “test” on the first page of the document?