1. Badges to Acknowledge Open Practices
Andrew Sallans
Center for Open Science
IASSIST 2014

2. Openness is a core value of scientific practice

3. Challenges: Perceived norms
Communality
Open Sharing
Universalism
Evaluate research on own merit
Disinterestedness
Motivated by knowledge and discovery
Organized skepticism
Consider all new evidence, even against
one’s prior work
Quality
Counternorms
Secrecy
Closed
Particularlism
Evaluate research by reputation
Self-interestedness
Treat science as a competition
Organized dogmatism
Invest career promoting one’s own
theories, findings
Quantity
Merton’s norms of science (1942) -
We can understand the nature of the challenge with existing psychological theory. For example:
1. The goals and rewards of publishing are immediate and concrete; the rewards of getting it right are distal and abstract (Trope & Liberman)
2. I have beliefs, ideologies, and achievement motivations that influence how I interpret and report my research (motivated reasoning; Kunda, 1990). And, even if I am trying to resist this motivated reasoning. I may simply be unable to detect it in myself, even when I can see those biases in others.
3. And, what biases might influence me. Well, pick your favorite. My favorite in this context is the hindsight bias.
4. What’s more is we face these potential biases in a context of minimal accountability. What you know of my laboratory work is only what you get in the published report. …
5. Finally, even if I am prepared to accept that I have these biases and am motivated to address them so that I can get it right. I am busy. So are you. If I introduce a whole bunch of new things that I must now do to check and correct for my biases, I will kill my productivity and that of my collaborators. So, the incentives lead me to think that my best course of action is to just to the best I can and hope that I’m doing it okay.
Communality – open sharing with colleagues; Secrecy
Universalism – research evaluated only on its merit; Particularism – research evaluated by reputation/past productivity
Disinterestedness – scientists motivated by knowledge and discovery, not by personal gain; self-interestedness – treat science as a competition with other scientists
Organized skepticism – consider all new evidence, theory, data, even if it contradicts one’s prior work/point-of-view; organized dogmatism – invest career in promoting one’s own most important findings, theories, innovations
Quality – seek quality contributions; Quantity – seek high volume

4. Anderson, Martinson & DeVries, 2007
3,247 mid- and early-career scientists who had research funding from NIH.
ideal to which most scientists subscribe
scientists perceptions of their own behavior
scientists perceptions of their peer’s behaviors
self-regulation, substantiall autonomy, the complexity of scientific projects, professional expertise, innovative work on cutting-edge problems, and a system of largely voluntary compliance with regulation and codes of ethics all point to the futility and inadvisability of direct administrative control over scientists’ behavior
Anderson MS, Martinson BC, De Vries R. Normative dissonance in science: Results from a national survey of U.S. scientists. Journal of Empirical Research on Human Research Ethics.2007;2(4):3–14.
Anderson, Martinson & DeVries, 2007

5. Challenges Perceived norms (Anderson, Martinson & DeVries, 2007)
Motivated reasoning (Kunda, 1990)
Minimal accountability (Lerner & Tetlock, 1999)
I am busy (Everyone)
We can understand the nature of the challenge with existing psychological theory. For example:
1. The goals and rewards of publishing are immediate and concrete; the rewards of getting it right are distal and abstract (Trope & Liberman)
2. I have beliefs, ideologies, and achievement motivations that influence how I interpret and report my research (motivated reasoning; Kunda, 1990). And, even if I am trying to resist this motivated reasoning. I may simply be unable to detect it in myself, even when I can see those biases in others.
3. And, what biases might influence me. Well, pick your favorite. My favorite in this context is the hindsight bias.
4. What’s more is we face these potential biases in a context of minimal accountability. What you know of my laboratory work is only what you get in the published report. …
5. Finally, even if I am prepared to accept that I have these biases and am motivated to address them so that I can get it right. I am busy. So are you. If I introduce a whole bunch of new things that I must now do to check and correct for my biases, I will kill my productivity and that of my collaborators. So, the incentives lead me to think that my best course of action is to just to the best I can and hope that I’m doing it okay.

6. How might this change?

7. Focus on simple interventions that researchers can actually adopt.
Bottom-up
Exposing users to good practices
Training sessions
Workshops
Spreading the message through active publishing
Going to where the user is now
Consultants serving as “Shock troops”
Top-down
Committees
Journal Editors
Federal Agencies
University administrators
think about very simple interventions that people can adopt
top down versus bottom up
community bottom up
exposing users to good practices
training, workshops,
active in community with publishing,
going to where the user is now,
stat consultant “shock troops”,
help reduce barriers and connect people to OSF tools that help them get there
community top down
committees,
journals,
feds,
etc.
Focus on simple interventions that researchers can actually adopt.

9. How to get a badge... Disclosure
--- author provides public statement that they’ve met badge criteria
Peer Reviewed
--- independent review of author’s public statement and meeting of criteria

10. What are the criteria?
Persistent path to the data, materials, pre registration, etc.
Sufficient information for an independent person to reproduce the results
Other more specialized items...

11. Who endorses these badges?

12. Which journals are adopting?

15. Going further…baking badges

19. What comes next? Improve the badge issuing workflow
Refine integration into peer review process
Gain adoption of more journals/organizations
Measure effect

20. Lastly, this is a community effort...
Ben B. Blohowiak | Johanna Cohoon | Lee de-Wit | Eric Eich | Frank J. Farach | Roger Giner-Sorolla | Fred Hasselman | Alex O. Holcombe | Macartan Humphreys | Melissa Lewis | Brian A. Nosek | Jonathan Peirce | Andrew Sallans | Jeffrey R. Spies | Chris Seto | Sara Bowman
Note: bold are non-COS

21. Thanks! Find out more and get involved here: https://osf.io/tvyxz/
me:
Twitter: @asallans