Aye-Aye Captain: Navigating Evaluation Data Collection as an External Evaluation Consultant David J. Bernstein, Ph.D. Senior Study Director, Westat Thu, October 27, 2016, 1:00 PM - 1:45 PM International South 7

Introduction This presentation will explore data collection issues faced by external evaluators (contractors) to meet Federal agency evaluation design and data collection needs.

The Client is Always Right Feds work in a complex procurement environments. Program and/or evaluation offices don’t have complete control of contracts, timelines, and budgets. Contractor proposals become part of contract if they win. If data collection issues are not fully spelled out or vetted, it can be expensive and timely for Feds/contractors.

RFP Issues Hard to anticipate client needs that are not always fully specified. Client requirements for data collection must be spelled out in the RFP: Identify potential data sources. Identify population-level data that can be used for statistical modeling. Specify sampling requirements. Include sub-populations that will be required for analysis.

Addressing IRB Issues 45 CFR 46 sets the Federal Policy for the Protection of Human Subjects ('Common Rule'). Applies to most Federal agencies. Preliminary IRB review is required even if a study is exempt from review. Evaluations are exempt from IRB review, but some contractor IRBs may want to be kept informed about data collection activities. IRB review enhances evaluation design quality by protecting human subjects.

Instrument Pre-testing Pre-testing data collection instruments is more than a best practice. In large well-funded studies, pre-testing is likely built into timing and funding. In smaller studies pre-testing may not be included. If included may need to pre-test with 9 or fewer respondents. Why you ask?

PRA and OIRA Issues Paperwork Reduction Act: OMB review process for data collections with 10+ individuals. RFPs and don’t always incorporate time/funding. Evaluation study timelines need to be staged: Year 1-Evaluation Planning: Evaluability assessment, data collection form development, and OMB review in Year 1. Year 2 and Later-Data Collection: Data collection, analysis in the outyears. Build in time for process, implementation reviews. Outcomes-Quite a Bit Later: Allow interventions time needed to produce and demonstrate intended outcomes or… Outcome Proxies: Be prepared to identify proxies to gauge progress towards outcomes.

PRA and OIRA Solutions Realistic discussions during kick-off meetings re: what is doable. Don’t reinvent the wheel. Use previously successful PRA Part A and Part B paperwork as a model for similar data collections. Use blanket clearances when appropriate. Use administrative data (no PRA) if relevant.

Contact Information David J. Bernstein, Ph.D. Senior Study Director, Westat davidbernstein@westat.com (301) 738-3520 @DJBernstein on Twitter