1. Contracting for Evaluation Products and Services by James B. Bell (bell@jbassoc.com)

2. Goal--procure needed evaluation services and products from an appropriately qualified evaluation contractor. Process--competition among prospective contractors or “offerors” that results in a contract award.

3. Five Principles for Successful Evaluation Contracting Principle 1, Establish a solid foundation by creating a feasible, approved concept plan that outlines key elements of the procurement. Principle 2, Ensure that offerors will understand the key elements of the purchase by developing a well- defined, appropriately prescriptive request-for- proposals (RFP). Principle 3, Select a well-qualified evaluator that fulfills the intent of the RFP and adapts to changing circumstances as needed.

4. Five Principles for Successful Evaluation Contracting (Continued) Principle 4, Constructively monitor interim progress to shepherd high performance by the contractor, without unnecessarily interfering or hindering progress. Principle 5, Ensure the quality and usefulness of major products by promoting early review and discussion of emerging findings using internal briefings to conserve resources.

5. Creating a Feasible, Approved Concept Plan Pitfalls Unreasonable expectations and internal disagreement that undercut product quality and usefulness. Tips Decide on key elements-- subject, scope and purpose, main products, delivery schedule, total cost estimate, period of performance and procurement approach Key elements are agreed-upon by prominent internal and external stakeholders (if not sure, use evaluability assessment techniques before the purchase or as a first step of the contractor)

6. Creating a Feasible, Approved Concept Plan (Continued) Tips Ensure products and services are deliverable within the delivery schedule and budget Ensure the evaluation contributes to building evaluation capacity in addition to fulfilling the sponsor’s immediate intent

7. Developing a Well-defined RFP Pitfalls An incomplete, ambiguous, or overly prescriptive RFP that interferes with offerers’ ability to demonstrate optimal ways to fulfill the evaluation intent. Tips Include sufficient detail on key technical elements— unambiguous and no missing key elements. Be appropriately prescriptive--allow offerors to demonstrate creativity to fulfill the intent of the evaluation. Present a few over-arching evaluation questions to help clarify the purpose (not a great number of narrow, disconnected questions that do not correspond to the scope and purpose).

8. Developing a Well-defined RFP (Continued) Tips Emphasize the main products and major tasks. Emphasize the needed substantive and methodological staff qualifications. Require a task-level staff loading chart for at least the first year of a multi-year contract (allocations of labor in out years can be less precise, but a task-level staff loading chart should be created before the start of each year). Include a total cost estimate expressed as dollars or gross amount of labor (total hours or person years).

9. Selecting a Well-qualified Evaluation Contractor Pitfalls The evaluator fails to perform as expected. Tips Provide advance notification of forthcoming evaluations to promote availability of prospective offerors. Establish umbrella contracts, or adopt a phased solicitation process when pressed for sponsor labor to develop an RFP. Request notices of intent to submit a proposal.

10. Selecting a Well-qualified Evaluation Contractor (Continued) Tips Ensure for the proposal review both independence and an appropriate range of expertise. Ensure selected contractor staff demonstrates through performance on similar past assignments the substantive (topical) knowledge and methodological qualifications needed to conduct the evaluation. Ensure the selected offeror’s understanding of problem accurately describes the current state of the subject program. Ensure the technical approach addresses the key evaluation questions with appropriate methods

11. Constructive Interim Progress Monitoring Pitfalls The evaluation proceeds more slowly and expensively than expected and misses chances to avoid problems and capitalize on opportunities. Tips The project officer maintains a constructive collaborative relationship with the contractor’s project director. The project officer supports the contractor, especially on issues where the sponsor exercises greater control such gaining access to documents, databases and people. Create a minimally burdensome progress report format, but require more detail if progress flags. Defend against unrecognized shifts in the evaluation mandate by periodically revisiting and discussing it throughout the project period.

12. Assuring Product Quality and Usefulness Pitfalls Reports do not accurately convey key evaluation findings in concise, clear language and re-writing consumes too much of the available resources. Tips Devote considerable attention to the development of major products. Encourage, and if necessary require, the use of a stepwise report development process to enable sponsor involvement and conserve project resources. Have the contractor start early on analysis, interpretation and shaping findings’ language using partial data