Data Collection Strategies, Methods and Standardization July 24, 2006 Jeff Mossey Assistant Director, Kentucky NSF EPSCoR

Brief Review of Typical Activities Annual Reporting (Fastlane) State Reporting (likely different) Brochure Annual Report Nuggets/Success Stories Newsletters Online Reporting Systems Diversity and Evaluation Reports/Activities Reverse Site Visits

Agenda Update on Evaluation Activities Identify Some (Evaluation) Problems and Possible Solutions Demonstrate the “EPSCoR Connections” Site –Strengths/Weaknesses/Improvements –Is this something that we can do as a program? Open Discussion of Issues/Concerns/Ideas Present it to NSF; Nat. Conf. Breakout

Status of Evaluation Activities Report from Norman Webb coming soon “Three Phases” of activities: Phase I – (Conceptual) Framework Conduct Workshops Establish Indicators Develop Report Phase II – (Detail) Development Establish metrics and standards Find out what data is available Phase III – Implementation and Testing

Six Major Program Indicators 1.Human Resource Development 2.Research Production 3.Research Portfolio Quality 4.Research Investments and Materials 5.Research Collaboration and Networking 6.Research Climate, Culture, and Communications

Norman’s Comments Data availability? Both nationally and at the state-level (e.g. funding/publications) Fastlane/EPSCoR Database Relationship –Common System or Common File Format Strength of the Connection ??? –Attribution to the project (Direct/Indirect) Timing of the Reporting (Fed. Fiscal Year)

Problems We Must Overcome LSAMP Program = –(Goal) Increasing the graduation rate of underrepresented students in the sciences. –(Metric) Did they go to grad school? EPSCoR Program = –(Goal) Research Infrastructure Development –(Goal) Increased Grant Funding (Competitiveness) –(Goal) Diversity –(Goal) Economic Development –(Goal) Outreach and Education (K-12) –(Goal) Networking and Strategic Development with Industrial Governmental, and Academic Partners

Problems We Must Overcome EPSCoR Investment/ Activities Meaningful, Aggregated Success Stories 1.) Use common metrics, stories, and techniques for gathering success indicators 2.) Use common systems for collecting the data 3.) Gather the right metrics/stories

Two Layers of Success Aggregated, Impressive Numbers “Annually over 200,000 people participate in this program” -LSAMP Brochure Nugget Short Stories

Weaknesses Won’t accommodate “other” successes (e.g. NIH awards, Nobel Prizes) Progress toward milestones and other “narrative” updates will still be needed Programs newer to EPSCoR will have fewer stories

Strengths Success stories are program-wide and standardized Tracing backward, or “backing into” a story is easier than constant monitoring “outward” Attribution to EPSCoR can vary Minimal time commitment More dated stories aren’t neglected By definition, connects EPSCoR to successful outcomes … follow-on awards = increased competitiveness Aggregate connections to show an impressive percentage of involvement. Stories (i.e., history of EPSCoR investments) is archived

Next Steps … Be Proactive! Is this something we want to do? –Standardizing the story gathering process? Develop “users manual” Document other evaluation comments Share with Norm/Sherry