1. Impact Panel
SI^2 PIs Meeting

2. Software Cycle How does the use of your software generate impact?
How does your impact generate resources, for sustainability?

3. What impacts are SI2 projects aiming for?
How ought those impacts be measured?
How ought those impacts be communicated to others?
Successful infrastructure might be invisible, but it didn’t become infrastructure by being invisible.

4. Survey summary (metrics)
Level 0
Haven’t thought much about it yet
Level 1 (our activity)
Our publications
Speed of development activity (bugs tracked/fixed)
Level 2 (others’ activity)
Count ‘em (Downloads, citations)
Direct and Indirect impact (software and papers)
Level 4? (activity in context)
Finding tipping points (e.g. activity of others exceeds ours)
Describing novel science that is facilitated
Passive vs Active collection.

5. Panel members Michael McLennan, HubZero Jason Priem, ImpactStory.org
Doug Thain, Cooperative Computing Tools (SI^2)
Jim Jagielski, Apache Foundation/Incubator
Travel delay
Robert van de Geijn, BLIS (SI^2)

6. HUBzero: Measuring the Impact of Scientific Works
Michael McLennan Director, HUBzero® Platform for Scientific Collaboration
Purdue University
NSF SI2 PI Meeting, January 17-18, 2013

7. Lesser General Public License, LGPL-3.0
What is HUBzero?
Lesser General Public License, LGPL-3.0
Linux/Apache/MySQL/PHP
Download:
40+ sites worldwide
NSF SI2 PI Meeting, January 17-18, 2013

8. Metrics Reported for Resources
NSF SI2 PI Meeting, January 17-18, 2013

9. Metrics Reported for Contributors
NSF SI2 PI Meeting, January 17-18, 2013

10. Metrics Reported about Community
NSF SI2 PI Meeting, January 17-18, 2013

11. NSF SI2 PI Meeting, January 17-18, 2013
Tracking Citations
NSF SI2 PI Meeting, January 17-18, 2013

12. Jason Priem

15. Doug Thain

16. Douglas Thain, Todd Tannenbaum, and Miron Livny,
How to Measure a Large Open Source Distributed System,
Concurrency and Computation: Practice and Experience, 18(15) 2006.

17. SI2-SSE: Connecting Cyberinfrastructure with the Cooperative Computing Tools Douglas Thain, University of Notre Dame
Public Global Namespace
Strategy: Create optional online interactions that have a direct benefit to the end user and to the service provider alike.
Offline Data Analysis Service
Real Time Service Monitoring

18. Apache Incubation and Impact
Apache Incubation Process
Real focus on external contributors
Because
Contribution indicates Impact
Money/Time where the mouth is.

19. Techniques for measuring impact
Traces on your website/infrastructure:
Registrations, Downloads, Clickstreams
Tracing support interactions (discussions)
Traces in publications
Citations (and other mentions)
Characteristic artifacts (esp. figures)
Traces in execution
Software that reports its own usage
Studies of use (e.g., productivity)
Evidence in workflows

20. Downloads vs Installed Base
Wiggins, A., Howison, J., & Crowston, K. (2009). Heartbeat: Measuring Active User Base and Potential User Interest in FLOSS Projects. In IFIP OSS Conference

21. Downloads vs Installed Base
Wiggins, A., Howison, J., & Crowston, K. (2009). Heartbeat: Measuring Active User Base and Potential User Interest in FLOSS Projects. In IFIP OSS Conference

22. Robert van de Geijn

23. Measure 1 (Paolo Bientinesi)
 “The impact of software S for application C should be the ratio between the time it would take to complete the application without the use of software S and the time it would take to complete it with the software.” (paraphrased)
In other words, it is the productivity multiplication factor.
(I am told) Economists call this the "Marginal Utility Price Ratio" -- the ratio of the marginal utility of the software, to its cost.
By this measure, the value of the GotoBLAS would be about
1.0/0.9 = 1.1.
John Stanton, a computational chemist, immediately applauded this measure. So, to users of software this seems to make sense.

24. Measure 2 (Chris Bischof)
“The number of refereed publications in good journals or conferences that credit the use of this resource, and the transitive impact of those publications.”
Google Scholar: 500 citations to papers by Goto in the last 5 years.
ACM DL: ~70 citations in high quality papers in the last 5 years.
I was not able to determine how many papers mention the GotoBLAS without citing the papers, which is really what Chris meant, I think.
Hard to measure the transitive effect…

25. Resources Apache Foundation Incubation Guidelines
incubator.apache.org/guides/
“Buzzing Communities” (Richard Milligan)
Feverbee.com
Producing Open Source Software (Karl Fogel)
producingoss.com
Impact Story
Impact-Story.org

26. Counting citations How are you going to count / list publications?
One, clear, citation that is recognizable to engines (Google Scholar, Citeseer, Web of Science) Vs
Adding authors to reward contributions, new publications to describe innovations
When are you going to ask?
At registration, at upgrade, at execution?
What are the standards in your field?
Perhaps 10% of users cite correctly