The SHP Framework: A New Framework for the Evaluation of Complex Projects Barbara Heath, Ph.D. Aruna Lakshmanan, Ph.D. Catherine Freeman, MPA East Main Evaluation & Consulting, LLC AEA 2013 | October 16, 2013 | 6:10pm

Context The iPlant Collaborative Multi-institutional Multi-disciplinary Virtual Distributed Create a cyberinfrastructure (CI) for plant science that will enable new conceptual advances through integrative, computational thinking

A new type of organization that will… Foster computational thinking in biology Be by, for, and of the community Remain on the leading edge of plant science Constantly reinvent itself and its capabilities

The SHP Framework Systems Highlights Patterns of Action Iterative process that developed organically alongside the evaluand Appropriate for dynamical situations with high uncertainty and high social conflict

Systems: CDE Model Context is complex, dynamical, and includes humans CDE* Three conditions for self-organizing behavior Containers, Differences, Exchanges Container 2 Container 1 * Eoyang, G.H. (2007)

Systems: CDE Model Data sources: documents, interviews, meeting minutes/attendance, surveys, formal reports Disaggregate the data into units Units are analyzed as a set and clustered Containers (C)

Systems: CDE Model Core Services Core Software Engagement Team iPToL Working Group iPG2P Working Group Executive Committee Board of Directors Target Community Admin. Support Determine what drives the action within each container (D) and how containers interact (E) to form the system

Systems: CDE Model System at a specific time point Executive Team Core Software Core Services

System Diagram

System Diagrams April 2009January 2009

System Diagram

Systems Analysis Outcome

Highlights Continued applying systems methods to iPlant Generated a significant amount of data and related analysis Selected to present a list of highlights for each quarter of project implementation Further identified highlights as deliberate events and emergent ideas

Q:YDeliberate Events/ActivitiesEmergent Ideas Q1:09 GCT Selected BOD Meeting to review GCPs Recommendations for GC engagement Leaflet launched Q2:09 NSF Retreat Management structure reorganization: SOT, FAT, CA Advisory Genomics in Education Workshop iPToL kickoff meeting – 6 groups formed Shift from consensus-based management to team- based management Engagement Team and Steering Committee concepts Compensation avenues for GCTs DNA Subway discussion initiated Q3:09 iPG2P kickoff meeting-5 groups formed CI Team meeting at TACC New Board Members inducted Board Chair change EOT Advisory Meeting in Chicago Website redesign Stanzione to TACC – leverage human and computing capital at TACC Shift from management focus to implementation focus Tech talks begin Q4:09 SOT kickoff meeting iPlant/NESCent joint meeting VA Mutual Understanding Meeting PI resigns My-Plant Workflows integrating data constructed via web-based services 1KP Consortium Atmosphere discussion initiated Highlights Excerpt, 2009

Patterns Recognized, 2009 Q:Y Deliberate Events/Activities CommunityImplementationManagement Q1:09 Leaflet launched GCT Selected BOD Meeting to review GCPs Recommendations for GC engagement Q2:09 Genomics in Education Workshop iPToL kickoff meeting – 6 groups formed NSF Retreat Management structure reorganization: SOT, FAT, CA Advisory iPToL kickoff meeting – 6 groups formed Q3:09 Website redesign iPG2P kickoff meeting – 5 groups formed CI Team meeting at TACC New Board Members inducted Board Chair change EOT Advisory Meeting in Chicago Q4:09 iPlant/NESCent joint meeting VA Mutual Understanding Meeting SOT kickoff meeting

Period of organization Heavy tool and application development Increase in workshops, training and outreach Renewal period

Rapid tool and application development Adoption increasing Spread awareness

The SHP Framework Systems Highlights Patterns of Action

SHP: Patterns of Action

Patterns of Action Patterns Examine actions over time Provide a visual representation Theme selection Understand project goals Examine value to client

Patterns of Action Personnel are concerned Restructuring Step CIAT CI-Lead IAB BOD FABCIATBOD IAB SciPlant BOD Project Initiated PI Resigns Direct Supervision Executive Management Management

Patterns of Action WG Seed Atmosphere 1KP SOT IBP DNA Subway MyPlan t TNRS Peak Ranger BSA-GWP Trait Evolution BrachyBio! GCW GCT2 GCT1 Reorganization Team BasedChaosProject Based Scientific Portfolio Development

Patterns of Action Personnel are concerned Restructuring Step CIAT CI-Lead IAB BOD FABCIATBOD IAB SciPlant BOD Project Initiated PI Resigns WG Seed Atmosphere 1KP SOT IBP DNA Subway MyPlant TNRS Peak Ranger BSA-GWP Trait Evolution BrachyBio! GCW GCT2 GCT1 Reorganization Scientific Portfolio Development Management Chaos

Why SHP? Captures the dynamic nature of the system For the Evaluator Identify and understand emergent themes Adjust data collection and focus Frame appropriate recommendations For the Client Helps visualize history of behavior, actions, and decisions Helps identify what worked and why Encourages proactivity

The SHP Framework Systems Highlights Patterns of Action

References Eoyang, G. H. (2007). Human systems dynamics: Complexity-based approach to a complex evaluation. In B. Williams, and I. Imam (Eds.) Systems concepts in evaluation: An expert anthology (pp ). Point Reyes, CA: EdgePress of Inverness. Mintzberg, H. (2007). Tracking strategies: Toward a general theory of strategy formation. New York, NY: Oxford University Press.

Acknowledgements The iPlant Collaborative (NSF Award: DBI ) Courtney Humphreys, EMEC Contact Information Barbara P. Heath, Ph.D. Aruna Lakshmanan, Ph.D. Catherine M. Freeman, MPA East Main Evaluation & Consulting, LLC Wilmington, North Carolina