1. The SHP Framework: A New Framework for the Evaluation of Complex Projects Barbara Heath, Ph.D. Aruna Lakshmanan, Ph.D. Catherine Freeman, MPA East Main Evaluation & Consulting, LLC AEA 2013 | October 16, 2013 | 6:10pm

2. Context The iPlant Collaborative Multi-institutional Multi-disciplinary Virtual Distributed Create a cyberinfrastructure (CI) for plant science that will enable new conceptual advances through integrative, computational thinking

3. A new type of organization that will… Foster computational thinking in biology Be by, for, and of the community Remain on the leading edge of plant science Constantly reinvent itself and its capabilities

4. The SHP Framework Systems Highlights Patterns of Action Iterative process that developed organically alongside the evaluand Appropriate for dynamical situations with high uncertainty and high social conflict

5. Systems: CDE Model Context is complex, dynamical, and includes humans CDE* Three conditions for self-organizing behavior Containers, Differences, Exchanges Container 2 Container 1 * Eoyang, G.H. (2007)

6. Systems: CDE Model Data sources: documents, interviews, meeting minutes/attendance, surveys, formal reports Disaggregate the data into units Units are analyzed as a set and clustered Containers (C)

7. Systems: CDE Model Core Services Core Software Engagement Team iPToL Working Group iPG2P Working Group Executive Committee Board of Directors Target Community Admin. Support Determine what drives the action within each container (D) and how containers interact (E) to form the system

8. Systems: CDE Model System at a specific time point Executive Team Core Software Core Services

9. System Diagram

10. System Diagrams April 2009January 2009

11. System Diagram

13. Systems Analysis Outcome

14. Highlights Continued applying systems methods to iPlant Generated a significant amount of data and related analysis Selected to present a list of highlights for each quarter of project implementation Further identified highlights as deliberate events and emergent ideas

15. Q:YDeliberate Events/ActivitiesEmergent Ideas Q1:09 GCT Selected BOD Meeting to review GCPs Recommendations for GC engagement Leaflet launched Q2:09 NSF Retreat Management structure reorganization: SOT, FAT, CA Advisory Genomics in Education Workshop iPToL kickoff meeting – 6 groups formed Shift from consensus-based management to team- based management Engagement Team and Steering Committee concepts Compensation avenues for GCTs DNA Subway discussion initiated Q3:09 iPG2P kickoff meeting-5 groups formed CI Team meeting at TACC New Board Members inducted Board Chair change EOT Advisory Meeting in Chicago Website redesign Stanzione to TACC – leverage human and computing capital at TACC Shift from management focus to implementation focus Tech talks begin Q4:09 SOT kickoff meeting iPlant/NESCent joint meeting VA Mutual Understanding Meeting PI resigns My-Plant Workflows integrating data constructed via web-based services 1KP Consortium Atmosphere discussion initiated Highlights Excerpt, 2009

16. Patterns Recognized, 2009 Q:Y Deliberate Events/Activities CommunityImplementationManagement Q1:09 Leaflet launched GCT Selected BOD Meeting to review GCPs Recommendations for GC engagement Q2:09 Genomics in Education Workshop iPToL kickoff meeting – 6 groups formed NSF Retreat Management structure reorganization: SOT, FAT, CA Advisory iPToL kickoff meeting – 6 groups formed Q3:09 Website redesign iPG2P kickoff meeting – 5 groups formed CI Team meeting at TACC New Board Members inducted Board Chair change EOT Advisory Meeting in Chicago Q4:09 iPlant/NESCent joint meeting VA Mutual Understanding Meeting SOT kickoff meeting

17. Period of organization Heavy tool and application development Increase in workshops, training and outreach Renewal period

18. Rapid tool and application development Adoption increasing Spread awareness

19. The SHP Framework Systems Highlights Patterns of Action

20. SHP: Patterns of Action

21. Patterns of Action Patterns Examine actions over time Provide a visual representation Theme selection Understand project goals Examine value to client

22. Patterns of Action Personnel are concerned Restructuring Step CIAT CI-Lead IAB BOD FABCIATBOD IAB SciPlant BOD Project Initiated PI Resigns Direct Supervision Executive Management 2008200920102011 Management

23. Patterns of Action 2008200920102011 WG Seed Atmosphere 1KP SOT IBP DNA Subway MyPlan t TNRS Peak Ranger BSA-GWP Trait Evolution BrachyBio! GCW GCT2 GCT1 Reorganization Team BasedChaosProject Based Scientific Portfolio Development

24. Patterns of Action Personnel are concerned Restructuring Step CIAT CI-Lead IAB BOD FABCIATBOD IAB SciPlant BOD Project Initiated PI Resigns 2008200920102011 2008200920102011 WG Seed Atmosphere 1KP SOT IBP DNA Subway MyPlant TNRS Peak Ranger BSA-GWP Trait Evolution BrachyBio! GCW GCT2 GCT1 Reorganization Scientific Portfolio Development Management Chaos

25. Why SHP? Captures the dynamic nature of the system For the Evaluator Identify and understand emergent themes Adjust data collection and focus Frame appropriate recommendations For the Client Helps visualize history of behavior, actions, and decisions Helps identify what worked and why Encourages proactivity

26. The SHP Framework Systems Highlights Patterns of Action

27. References Eoyang, G. H. (2007). Human systems dynamics: Complexity-based approach to a complex evaluation. In B. Williams, and I. Imam (Eds.) Systems concepts in evaluation: An expert anthology (pp. 123-140). Point Reyes, CA: EdgePress of Inverness. Mintzberg, H. (2007). Tracking strategies: Toward a general theory of strategy formation. New York, NY: Oxford University Press.

28. Acknowledgements The iPlant Collaborative (NSF Award: DBI-1265383) Courtney Humphreys, EMEC Contact Information Barbara P. Heath, Ph.D. bheath@emeconline.com Aruna Lakshmanan, Ph.D. alakshmanan@emeconline.com Catherine M. Freeman, MPA cfreeman@emeconline.com www.emeconline.com East Main Evaluation & Consulting, LLC Wilmington, North Carolina