1. Assessing Readiness for Developmental Evaluation within the context of the CTSA Program
Doug Easterling, Ph.D.
Department of Social Sciences and Health Policy
Wake Forest School of Medicine
American Evaluation Association Annual Meeting
Washington, DC
November 10, 2017

2. Organization of the Think Tank Session
Setting the context (15 mins)
Overview of Developmental Evaluation and the CTSA Program
How well does DE fit within CTSA?
“Readiness” issues that warrant attention
The 4 specific issues we’ll explore today
Small groups (45 mins)
Internal factors (within an AMC) that inhibit or facilitate DE (Clara Palfrey)
Larger issues (e.g., CTSA program requirements) that inhibit or facilitate DE (Julie Rainwater)
Developing systems to enhance readiness for DE (Gaurav Dave)
Measuring readiness for DE at an organizational level (Doug Easterling)
Reconvene (30 mins)
Report out of 3-5 major “aha’s” and/or “points of enthusiasm”
What is all this telling us about the appropriateness of DE within CTSA?
Next steps
To further promote the adoption of DE among CTSA hubs
Conceptualizing and assessing “readiness” for DE

3. Developmental Evaluation
Supports the development and adaptation of innovations
within dynamic environments
where the problems to be addressed are complex
where solutions are inherently transitory
Departure from traditional evaluation (summative and formative)
Assumptions as to whether the program model should converge
Dynamic interaction between findings and adaptation to design
Complexity is modeled rather than trying to control the context
Emergent methodology to accommodate emergent strategy
Evaluator is part of innovation team rather than independent arbiter
Patton, 2011

4. Institutional Context
DE raises new and ambitious expectations (e.g., flexibility, adaptability, critical analysis, systems thinking, tolerance for ambiguity)
For the evaluator
For program developers
For the organization that delivers the innovation
Early adopters
Foundations
Social service agencies
Advocacy organizations
Coalitions and networks

5. Academic Medical Centers
DE has the potential to advance the work of AMCs
Discovery, innovation and adaptation are core elements
The environment is complex and dynamic (changes in scientific knowledge, technology, financing, policy, demographics)
There are countervailing factors
RCT is the dominant paradigm for research
Hierarchical structures where knowledge is power
Scarce resources (or at least the perception of scarcity)
Admitting shortcomings can be professionally catastrophic
Not surprisingly, DE (and evaluation more generally) historically has not found a home within AMCs
Clinical and Translational Science Awards (CTSA) program offers a chance to introduce DE into AMCs (i.e., a disruptive opportunity)
Pass it off to Arthur after this slide

6. Clinical and Translational Science Awards (CTSA) Program
CTSA supports a national network of medical research institutions — called hubs ― that work together to improve the translational research process to get more treatments to more patients more quickly.
The hubs receive 5-year grants from NIH to catalyze innovation in training, research tools and processes.
CTSA program is administered nationally by National Center for Accelerating Translational Sciences (NCATS)
First round of CTSA grants were awarded in 2006
~60 hubs are funded at any given point in time
Hubs are required to establish and maintain specific programs, and to meet specific requirements re: metrics

7. Sampling of Programs at a Typical CTSA Hub
Grants for pilot projects
Career development training and funding
Mentor training
Translational workforce development
Biomedical informatics
Biostatistics, Epidemiology and Research Design (BERD) consultation
Clinical Research services and facilities
Integrating special populations into research
Team science
Regulatory knowledge and support
Evaluation

8. Developmental Evaluation within the CTSA Context
In some ways, the perfect context for DE
Continually increasing understanding of what it takes to promote translational science
CTSA programs evolve accordingly
Evaluation is a required element for each hub
Opportunities for cross-program learning within each hub
Remarkable sharing and co-learning across hubs
But there are countervailing factors
Lack of understanding of and appreciation for DE
AMCs hire scientists rather than evaluators (esp developmental evaluators)
DE requires transparency and fluidity – at odds with bureaucracy
NCATS emphasis on metrics and CQI – rather than exploring and adapting strategy
Some hubs are pursuing DE – either explicit DE or “looks and quacks like DE”

9. American Evaluation Association Conference
Conducting Developmental Evaluation within Academic Medical Centers under the Clinical and Translational Science Awards (CTSA) Program
Doug Easterling, Wake Forest University (Chair And Panelist)
Gaurav Dave And Tiffany Young, University Of North Carolina At Chapel Hill (Panelist)
Jillian Harvey, Medical University Of South Carolina (Panelist)
Janice Hogle, University Of Wisconsin-Madison (Panelist)
Arthur Blank, Albert Einstein College Of Medicine (Moderator)
October 27, 2016
American Evaluation Association Conference
Doug Easterling will start off the presentation

10. What will it take to further implement DE within the CTSA context?
Increase the supply of developmental evaluators within AMCs
Increase demand for (appreciation of) a DE approach
Identify and understand barriers and facilitators (e.g., policies, structures, attitudes, culture, leadership) at multiple levels
Institution-wide at CTSA-funded medical centers
The CTSA institute/center
Individual leaders of specific CTSA-funded programs
NCATS
Change organizational context to be more conducive to DE
Endorse DE within policy documents that guide how evaluation is carried out within CTSI hubs

11. Moving the process forward: Key Questions
What are the critical internal factors (within an AMC) that inhibit or facilitate DE (Clara Palfrey)
Culture, personnel, funding streams, institutional leadership, data systems, etc.
What are the larger contextual issues that inhibit or facilitate DE (Julie Rainwater)
CTSA grant requirements, NCATS policies and priorities, common metrics, IOM report, etc.
What steps can proponents of DE take within their institution to increase interest and create a hospitable environment for DE (Gaurav Dave)
Planning processes, support systems, cultivation, etc.
How should we go about measuring an organization’s readiness for DE (Doug Easterling)
Pre-conditions for innovation and organizational learning, measurable constructs, possible tools