Data Sharing and Secondary Use of Scientific Data: What can collaboratories learn from ecology? Ann Zimmerman USGS Great Lakes Science Center Ann Arbor, MI SOC Seminar September 11, 2003

Ecology The study of interrelationships between the earth’s organisms and their environment

Photos in slides 3-5 are from: Klett, Albert T., et al Techniques for studying nest success of ducks in upland habitats in the Prairie Pothole Region. U.S. Fish and Wildlife Service, Resource Publication 158.

Why ecological data?  Ecologists work at small spatial and temporal scales  Data sets are small and highly diverse  Standard methods are difficult to achieve  Ecology is a craft science  There is a high level of data ownership

Data sharing is necessary in order to address many environmental problems. The destruction of rainforests influences weather patterns in other parts of the world. Airborne pollutants from one country affect the health of another nation’s water supply.

No one could use my data! They wouldn’t understand them! We must have your data to save the planet!

Intriguing Questions  Why are some data easier/harder to share than others?  Do standards really facilitate data sharing? If so, when? If so, how?  How do secondary users judge data quality?

Answers are relevant to…  Design of data resources  Standards development  Policy  Education

Existing Research  The affect of databases on the practice and communication of science  Scientists’ attitudes toward data sharing  Research-related information that scientists share  Expected returns for sharing  Data withholding

RQ: What are the experiences of ecologists who use shared data?  How do ecologists locate data and assess their quality?  What are the characteristics of the data they receive?  What information do ecologists depend on to use the data?  What challenges do they face throughout the process?

Qualitative Research Methods  Effective when important variables are unclear and empirical information is scarce  Useful for understanding processes as well as outcomes  Interviewing is a useful method to study past events and when participants cannot be observed

Qualitative Research Limitations  Imprecise measurement  Vulnerability to bias  Weak generalizability of findings

Key Definitions Data  Scientific data Scientific or technical measurements…and observations or facts that can be represented by numbers…and that can be used as a basis for reasoning or further calculation (NRC, 1997).

Ecologists  Members of ESA, or  Self-identification, or  Affiliation or title contains ecolog*

Data Sharing  The voluntary provision of information from one individual or institution to another for purposes of legitimate scientific research (Boruch, 1985)  My study is limited to shared data used for ecological research.

Secondary Use of Data  The use of data collected for one purpose to study a new problem  Includes data gathered to address a specific research question & data used to describe biological or physical phenomena

Data Collection  Method: Semi-structured, in-depth interviews  Primary subjects: 13 ecologists who reused data (selected from 2 key ecological journals)  Secondary subjects: 4 data managers

Data Analysis  Primary data: Interview transcripts  Developed a coding scheme and analyzed data following suggestions from Miles & Huberman*  Reliability: Detailed descriptions of subject selection, data collection, and data analysis; reporting of bias and values; member checks  Validity: Use of diverse sources to study the same phenomenon; member checks * Qualitative Data Analysis: An Expanded Sourcebook, 2 nd ed.

Conceptual Framework Overcoming Distance

Overcoming D i s t a n c e Potential Distances: Cultural, Epistemological, Methodological, or Terminological Temporal or Spatial Personal Social Exchange Standards Informal Knowledge

Standards as Distance Spanners: Making Local Knowledge Public Measurement as a social technology (Porter) * Quantification as a technology of distance * Standards as a substitute for trust based on personal knowledge Porter, T. M. (1999). Quantification and the accounting ideal in science. In M. Biagioli (Ed.), The science studies reader (pp ). New York: Routledge. Porter, T. M.(1995). Trust in numbers: The pursuit of objectivity in science and public life. Princeton, NJ: Princeton University Press.

Standards Reduce & Amplify  Standard measurements involve a loss of information (reduction).  Reduction turns local knowledge into public knowledge (amplification). Latour, B. (1999). Circulating reference: Sampling the soil in the Amazon forest. In Pandora’s hope: Essays on the reality of science studies (pp ). Cambridge, MA: Harvard University Press.

Circulating Reference The ability of standards to bring the world closer, yet also to push it away Inscriptions

Key Findings Overcoming Distances in the Secondary Use of Data

Gathering One’s Own Data Helps with Reuse Ecologists' experiences as collectors of their own data in the field or laboratory plays the most important role in their secondary use of data.

Data Gathering Provides:  Expertise to understand the critical link between the purpose, the research methods chosen, and the data that result  Ability to recognize the data limitations  Ability to visualize potential points of error  A ‘sense’ for data

Research purpose Methods Data What frog species live here?How many frogs live here?

Charles: “In some ways it is just very simple. Someone saw an animal on such and such a date at such and such a location. That’s basically it. And you can explain that to six-year-old. The only tricky thing…. and, you know, in some ways it is not that hard conceptually, but I see people making the mistake all the time… What does the absence of a record mean? And the absence of a record doesn’t mean the absence of a species. It may just mean a lack of survey effort. And you see biological reports all the time that people consult the state biodiversity database and say, “Oh, we have no endangered species on this piece of property. It’s okay; go ahead and turn it into a shopping mall.”

Susan: “Well, where the different sources of error can come in-- things like getting water samples, or running the equipment and running the machines that actually analyze water chemistry, and how where you sample within a lake might influence dissolved organic carbon. So, you just get a better idea of all the different things that could influence the final number.” Visualizing Potential Points of Error

Factors that Influence Research Methods  The scientific question  The environment  The taxa  Practical considerations such as time, money, and skill

Nancy: “When you're in the field, most of what you learn is not the data points you're collecting -- it's just that sense.” Michael: “The more you actually go out and do those things the more.... You are sort of more critical of the data.” Gaining a ‘sense’ for data

Standards of Scientific Practice Ecologists recognize the informal knowledge they gain in the field, but it is not discussed publicly in the context of “real science” Formal notions about norms of scientific practice guide the gathering of data for reuse and frame ecologists’ experiences

Hindrances to Sharing & Reuse  Challenge of locating and integrating data collected for many different purposes and at varying temporal and spatial scales  Ecologists’ idiosyncratic methods of organizing data

Re-circulating Reference Ecologists attempt to reconstruct the original collection of the data they seek to reuse. Inscriptions

Ellen: “If honestly I could not figure out what they had done, then I just would not use that data point.”

Nathan: “One person could have a table that has a column of species and density. Another person could have a table that says Species I Density, Species II Density, and Species III Density. Those sorts of schema differences when you scale them up to 10, 20, 30 data sets -- and we would like to get to 100, 200, 1000 data sets -- become extremely limiting in your ability to integrate the data and to utilize them in a particular framework.”

Key Findings: Their applicability, significance for collaboratories, and suggestions for future research

Factors Influencing Data Reuse  Scientific questions  Existence of formal data sharing systems  Data characteristics  Presence of standards  Reuse potential  Intermediaries  Computational and statistical capacity

Christine: “In a field like molecular ecology, you grind up a sample, extract the DNA, and sequence it. It's the same thing over and over regardless of the material, and so it's relatively easy to standardize that. Of course, the more I work in molecular ecology, the more I realize that there are many sources of error, many points of decision making, etc. that can and do make standardization difficult.”

Charles : “The economics data is often much more organized and processed. In economics, typically people are working with a shared data set. There are hundreds of people that work with the current population survey, for example, and you can go and find out, "Well, what are the problems with this data set?" Everyone can tell you, "Oh yeah, ’79 was a really bad year, and there’s a glitch, and you are going to have reprocess this field if you want to use it. … But ecology data is not like that. Typically it never gets re- analyzed. And so you are on your own and kind of starting from scratch working with, untested and unverified, unvalidated, and unchecked out data most of the time.”

Most scientific data are not simple “measurements” Taken from Paul Avery’s SOC seminar – May 8, 2003 Data Grids for 21 st Century Data Intensive Science Available at:

“…the analysis of protemics data is currently informal and relies heavily on expert opinion. Databases and software tools developed for the analysis of molecular sequences and microarrays are helpful, but are limited owing to the unique attributes of proteomics data and differing research goals.” Boguski, Mark S. and Martin W. McIntosh Biomedical informatics and proteomics. Nature 422:

Ecological Circuitry Collaboratory “At an individual level, we would like all students in this program to be better able to build, use, and understand models while at the same time have firm grounding in the practices of field- and lab-based empirical science.”

Special thanks to…  Doctoral committee (Margaret Hedstrom, Chair)  Study participants  Scientist friends and colleagues  USGS Great Lakes Science Center  UM School of Information  Rackham Graduate School