1. Cyber-Infrastructure Challenges & Ecoinformatics: An Ecologist’s Perspective William Michener LTER Network Office Department of Biology University of New Mexico

2. Today’s Road Map Science Cyber-Infrastructure Challenges Ecoinformatics

3. Today’s Road Map Science Cyber-Infrastructure Challenges Ecoinformatics

4. Most studies use a single scale of observation -- Commonly 1 m 2 The literature is biased toward single and small scale results

5. Time (yrs) Variable Change transition from one state or condition to another

6. Space Parameters Time Thinking Outside the “Box” LTER Biocomplexity ???? Increase in breadth and depth of understanding.....

7. 24 7 11 10 8 6 5 4 3 2 20 1 9 18 16 15 17 14 13 12 23 22 21 19 LTER 24 NSF LTER Sites in the U.S. and the Antarctic: > 1500 Scientists; 6,000+ Data Sets— different themes, methods, units, structure, ….

9. Today’s Road Map Science Cyber-Infrastructure Challenges Ecoinformatics

10. Knowledge Data Information Phenomena

11. Knowledge Data Information Phenomena

12. Abstraction of phenomena

13. Knowledge Data Information Phenomena

14. Yon Hither Hunter-gatherers

15. A Paradigm Shift Taxon 1 Taxon 2 Taxon 3 Taxon 4 Abioticfactors Integrated,InterdisciplinaryDatabases Harvesters

16. Information Content Time Time of publication Specific details General details Accident Retirement or career change Death (Michener et al. 1997) Data Entropy

17. A B Schema transform Coding transform Taxon Lookup Semantic transform Imagine scaling!! C A B C Semantics

18. Semantics—Linking Taxonomic Semantics to Ecological Data Rhynchospora plumosa s.l. Elliot 1816 Gray 1834 Kral 1998 Peet 2002? Chapman 1860 R. plumosa R. Plumosa v. intermedia R. plumosa v. plumosa R. Plumosa v. interrupta R. plumosa R. intermedia R. pineticola R. plumosa v. plumosa R. plumosa v. pinetcola R. sp. 1  Taxon concepts change over time (and space)  Multiple competing concepts coexist  Names are re-used for multiple concepts from R. Peet ABC

19. Knowledge Data Information Phenomena

20. Characteristics of Ecological Data Complexity/Metadata Requirements Satellite ImagesDataVolume(perdataset) Low High High Soil Cores PrimaryProductivity GIS Population Data BiodiversitySurveys Gene Sequences Business Data Weather Stations Most Ecological Data MostSoftware

21. What Users Really Want…

22. Data Collection Analysis Translation Use By Non-Scientists Publish For Other Scientists

23. Today’s Road Map Science Cyber-Infrastructure Challenges Ecoinformatics

24. Ecological Informatics A broad interdisciplinary science that incorporates both conceptual and practical tools for the understanding, generation, processing, and propagation of ecological data and information.

25. Data Design and Metadata Data Acquisition and Quality Control Access and Archiving Analysis and Interpretation Data Manipulation and Quality Assurance Project Initiation Publication

26. Experimental Design Methods Data Design Data Forms Data Entry Field Computer Entry Electronically Interfaced Field Equipment Electronically Interfaced Lab Equipment Raw Data File Quality Assurance Checks Data Contamination Data verified? Data Validated Archive Data File Archival Mass Storage Magnetic Tape / Optical Disk / Printouts Access Interface Off-site Storage Secondary Users Publication Synthesis Investigators Summary Analyses Quality Control Metadata Research Program Investigators Studies yes no

27. “Ecological Informatics” Activities Project / experimental design Data design [Porter & McCartney] Data acquisition QA/QC [Vanderbilt] Data documentation (metadata) [Michener] Data archival

28. “Ecological Informatics” Activities Project / experimental design Data design Data acquisition QA/QC Data documentation (metadata) Data archival

29. Project / Experimental Design ExperimentalDesign Analyses Data / Database Design

30. Project / Experimental Design Some Classic References –Green, R.H. 1979. Sampling Design and Statistical Methods for Environmental Biologists. John Wiley & Sons, Inc., New York. –Resetarits, Jr., W.J. and J. Bernardo (eds.). 1998. Experimental Ecology. Oxford University Press, New York. –Scheiner, S.M. and J. Gurevitch (eds.). 1993. Design and Analysis of Ecological Experiments. Chapman & Hall, New York. –Sokal, R.R. and F.J. Rohlf. 1995. Biometry. W.H. Freeman & Company, New York. –Underwood, A.J. 1997. Experiments in Ecology: Their Logical Design and Interpretation Using Analysis of Variance. Cambridge University Press, Cambridge, UK.

31. “Ecological Informatics” Activities Project / experimental design Data design Data acquisition QA/QC Data documentation (metadata) Data archival

32. Data Design Conceptualize and implement a logical structure within and among data sets that will facilitate data acquisition, entry, storage, retrieval and manipulation.

33. Data Set Design: Best Practices Assign descriptive file names Use consistent and stable file formats Define the parameters Use consistent data organization Perform basic quality assurance Assign descriptive data set titles Provide documentation (metadata) from Cook et al. 2000

34. 1. Assign descriptive file names File names should be unique and reflect the file contents Bad file names –Mydata –2001_data A better file name –Sevilleta_LTER_NM_2001_NPP.asc Sevilleta_LTER is the project name NM is the state abbreviation 2001 is the calendar year NPP represents Net Primary Productivity data asc stands for the file type--ASCII

35. 2. Use consistent and stable file formats Use ASCII file formats – avoid proprietary formats Be consistent in formatting –don’t change or re-arrange columns –include header rows (first row should contain file name, data set title, author, date, and companion file names) –column headings should describe content of each column, including one row for parameter names and one for parameter units –within the ASCII file, delimit fields using commas, pipes (|), tabs, or semicolons (in order of preference)

36. 3. Define the parameters Use commonly accepted parameter names that describe the contents (e.g., precip for precipitation) Use consistent capitalization (e.g., not temp, Temp, and TEMP in same file) Explicitly state units of reported parameters in the data file and the metadata (SI units are recommended) Choose a format for each parameter, explain the format in the metadata, and use that format throughout the file –e.g., use yyyymmdd; January 2, 1999 is 19990102

37. 4. Use consistent data organization (one good approach) StationDateTempPrecip Units YYYYMMDDCmm HOGI19961001120 HOGI19961002143 HOGI1996100319-9999 Note: -9999 is a missing value code for the data set

38. 4. Use consistent data organization (a second good approach) StationDateParameterValueUnit HOGI19961001Temp12C HOGI19961002Temp14C HOGI19961001Precip0mm HOGI19961002Precip3mm

39. 5. Perform basic quality assurance Assure that data are delimited and line up in proper columns Check that there no missing values for key parameters Scan for impossible and anomalous values Perform and review statistical summaries Map location data (lat/long) and assess errors Verify automated data transfers For manual data transfers, consider double keying data and comparing 2 data sets

40. 6. Assign descriptive data set titles Data set titles should ideally describe the type of data, time period, location, and instruments used (e.g., Landsat 7). Titles should be restricted to 80 characters. Data set title should be similar to names of data files –Good: “Shrub Net Primary Productivity at the Sevilleta LTER, New Mexico, 2000-2001” –Bad: “Productivity Data”

41. 7. Provide documentation (metadata) To be discussed in detail

42. Database Types File-system based Hierarchical Relational Object-oriented Hybrid (e.g., combination of relational and object-oriented schema) Porter 2000

43. File-system-based Database Directory Files Porter 2000

44. Hierarchical Database Project Data setsInvestigators VariablesLocations CodesMethods Porter 2000

45. Relational Database Projects Data sets Locations Location_idData_id Location_id Porter 2000

46. “Ecological Informatics” Activities Project / experimental design Data design Data acquisition QA/QC Data documentation (metadata) Data archival

47. High-quality data depend on: Proficiency of the data collector(s) Instrument precision and accuracy Consistency (e.g., standard methods and approaches) –Design and ease of data entry Sound QA/QC Comprehensive metadata (e.g., documentation of anomalies, etc.)

48. How are data to be acquired? Automatic Collection ? Tape Recorder Data Sheet Field entry into hand-held computer

49. PlantLife Stage ______________ _______________ What’s wrong with this data sheet?

50. Important questions How well does the data sheet reflect the data set design? How well does the data entry screen (if available) reflect the data sheet?

51. PlantLife Stage ardiP/G V B FL FR M S D NP arpuP/G V B FL FR M S D NP atcaP/G V B FL FR M S D NP bamuP/G V B FL FR M S D NP zigrP/G V B FL FR M S D NP P/G V B FL FR M S D NP PHENOLOGY DATA SHEET Rio Salado - Transect 1 Collectors:_________________________________ Date:___________________ Time:_________ Notes: _________________________________________ _______________________________________________ P/G = perennating or germinatingM = dispersing V = vegetatingS = senescing B = buddingD = dead FL = floweringNP = not present FR = fruiting

52. PHENOLOGY DATA SHEET Rio Salado - Transect 1 Collectors Troy Maddux Date: 16 May 1991 Time: 13:12 Notes: Cloudy day, 3 gopher burrows on transect ardiP/G V B FL FR M S D NP Y N arpuP/G V B FL FR M S D NP Y N asbrP/G V B FL FR M S D NP Y N deobP/G V B FL FR M S D NP Y N

53. “Ecological Informatics” Activities Project / experimental design Data design Data acquisition QA/QC Data documentation (metadata) Data archival

54. Experimental Design Methods Data Design Data Forms Data Entry Field Computer Entry Electronically Interfaced Field Equipment Electronically Interfaced Lab Equipment Raw Data File Quality Assurance Checks Data Contamination Data verified? Data Validated Archive Data File Archival Mass Storage Magnetic Tape / Optical Disk / Printouts Access Interface Off-site Storage Secondary Users Publication Synthesis Investigators Summary Analyses Quality Control Metadata Research Program Investigators Studies yes no Brunt 2000 Generic Data Processing

55. “Ecological Informatics” Activities Project / experimental design Data design Data acquisition QA/QC Data documentation (metadata) Data archival

56. “Ecological Informatics” Activities Project / experimental design Data design Data acquisition QA/QC Data documentation (metadata) Data archival

57. Traditional Fates of Data Post-Publication Paper to filing cabinets Data to floppy disks or tape Data and information lost over time entropy

58. Data Archive A collection of data sets, usually electronic, stored in such a way that a variety of users can locate, acquire, understand and use the data. Examples: –ESA’s Ecological Archive –NASA’s DAACs (Distributed Active Archive Centers)

59. Planning Problem Analysis and modeling Cycles of Research “A Conventional View” Collection Publication s Data

60. Cycles of Research “A New View” Planning Problem Definition (Research Objectives) Analysis and modeling Planning Collection Selection and extraction Archive of Data Original Observations Secondary Observations Publication s

61. Start small and keep it simple – building on simple successes is much easier than failing on large inclusive attempts.  Involve scientists - ecological data management is a scientific endeavor that touches every aspect of the research program. Scientists should be involved in the planning and operation of a data management system.  Support science – data management must be driven by the research and not the other way around, a data management system must produce the products and services that are needed by the community. Keys to Success

62. Data Value Time Serendipitous Discovery Inter-site Synthesis Gradual Increase In Data Equity Methodological Flaws, Instrumentation Obsolescence Non-scientific Monitoring Increasing value of data over time

63. LTER Data Access Policy 1)There are two types of data: Type I (data that is freely available within 2-3 years) with minimum restrictions and, Type II (Exceptional data sets that are available only with written permission from the PI/investigator(s)). Implied in this timetable, is the assumption that some data sets require more effort to get on-line and that no "blanket policy" is going to cover all data sets at all sites. However, each site would pursue getting all of their data on-line in the most expedient fashion possible. 2) The number of data sets that are assigned TYPE II status should be rare in occurrence and that the justification for exceptions must be well documented and approved by the lead PI and site data manager. Some examples of Type II data may include: locations of rare or endangered species, data that are covered by copyright laws (e.g. TM and/or SPOT satellite data) or some types of census data involving human subjects.

64. Reasons to Not Share Data: Fear of getting scooped Number of publications will decrease People will find errors Someone will misinterpret my data

65. Benefits of Data Sharing Publicity, accolades, media attention Renewed or increased funding Teaching: long-term data sets adapted for teaching & texts Archival: back-up copy of critical data sets Research: new synthetic studies peer-reviewed publications Document global and regional change Conservation and resource management: species and natural areas protection new environmental laws

66. Brunt (2000) Ch. 2 in Michener and Brunt (2000) Porter (2000) Ch. 3 in Michener and Brunt (2000) Edwards (2000) Ch. 4 in Michener and Brunt (2000) Michener (2000) Ch. 7 in Michener and Brunt (2000) Cook, R.B., R.J. Olson, P. Kanciruk, and L.A. Hook. 2000. Best practices for preparing ecological and ground-based data sets to share and archive. (online at http://www.daac.ornl.gov/cgi- bin/MDE/S2K/bestprac.html)

67. Thanks !!!