1. Metadata Understanding the Value and Importance of Proper Data Documentation Michael Moeller Metadata Specialist NOAA Coastal Services Center

2. What is Metadata?

3. Simply put, metadata is information about your data. What is Metadata?

4. This is the metadata for this. What’s Missing? Emily and Madison

5. This is the metadata for this. While the card-catalog entry is a form of metadata, it does not address topics such as quality, accuracy, or scale. Well-written geospatial metadata describes these and many more aspects of the data. Rodale's illustrated encyclopedia of herbs ISBN: 087596964x (pbk.) : $17.95 ISBN: 0878576991 : $24.95 Title: Rodale's illustrated encyclopedia of herbs / Claire Kowalchik & William H. Hylton, editors ; writers, Anna Carr... [et al.]. Publication info: Emmaus, Pa. : Rodale Press, c1987. Physical descrip: vi, 545 p. : ill. (some col.) ; 24 cm. General Note: Includes index. Subject term: Herbs. Subject term: Herbs--Utilization. Subject term: Herb gardening. Subject term: Herbs--History. Subject term: Herbals. Added author: Kowalchik, Claire. Added author: Hylton, William H. Added author: Carr, Anna, 1955- Added author: Rodale Press. Rodale's illustrated encyclopedia of herbs ISBN: 087596964x (pbk.) : $17.95 ISBN: 0878576991 : $24.95 Title: Rodale's illustrated encyclopedia of herbs / Claire Kowalchik & William H. Hylton, editors ; writers, Anna Carr... [et al.]. Publication info: Emmaus, Pa. : Rodale Press, c1987. Physical descrip: vi, 545 p. : ill. (some col.) ; 24 cm. General Note: Includes index. Subject term: Herbs. Subject term: Herbs--Utilization. Subject term: Herb gardening. Subject term: Herbs--History. Subject term: Herbals. Added author: Kowalchik, Claire. Added author: Hylton, William H. Added author: Carr, Anna, 1955- Added author: Rodale Press.

6. This is the metadata for this.

7. We often use metadata without even knowing it.

8. Metadata contains vital information. Imagine, if you will… You are given two identical cans without labels. One contains cat food, the other contains tuna (dolphin-safe, of course). You must choose one these cans, and then eat the contents.

9. Entity Attributes A Common, Everyday Example

10. Other Examples? What is Metadata?

11. Metadata as a Component of Data

12. Properly documented data provides vital information to interested parties. A Component of Data

13. Metadata is that component of data which describes it. Environmental Sensitivity Index Data A Component of Data RARNUM - unique combination of species, concentration, and seasonality CONC (concentration) = Density species is found at location Season_ID = seasonality code like to the seasonal table Element - Biology group Metadata

14. Metadata describes… CONTENT CONDITION QUALITY Characteristics of the data Characteristics of the data A Component of Data

15. Metadata can describe a variety of data. Title Scale Source Content Location Publication Access Title Scale Source Content Location Publication Access MetadataMetadata A Component of Data GIS files Imagery Geospatial Databases GPS data Biological data In situ data GIS files Imagery Geospatial Databases GPS data Biological data In situ data Data set

16. A Component of Data Because metadata provides vital information about a dataset, it should never be viewed or treated as a separate entity. Metadata Non-spatial or attributes Spatial Take Home Message Metadata is a critical and integral component of any complete data set. Metadata is a critical and integral component of any complete data set.

17. The Value of Metadata

18. The Value of Metadata The Current Concept Primary external value Discovery Assessment Access Use

19. The Value of Metadata The Current Concept Primary internal value “Inheritance” “Properly documenting a data set is the key to preserving its usefulness through time.”

20. The Value of Metadata An Emerging Concept An aid to data management Internal value Discovery Assessment Access Use

21. Additional data management benefits An Emerging Concept Data Currency Date of last edit/update Age of source files Data Utility Track source file usage Track distribution frequency

22. Monitoring Data Development Data processing steps Status of development Estimating Development Costs Data processing – time and extent Source file availability Additional data management benefits An Emerging Concept

23. To realize the full potential of metadata under this new concept, metadata creation must become integral to the data development process. The question is “How?” Make metadata part of the process

24. Approach metadata development from a business perspective Build administrative support Preserves data investment Limits liability Helps manage data resources Aids in external data acquisition Facilitates data access and transfer Provides for efficient data distribution Make metadata part of the process

25. Stress the individual benefits of metadata Build technical support Reduces workload over the long term Field fewer data inquiries Provides a means of documenting personal contributions Facilitates sharing of reliable information Make metadata part of the process

26. Develop strong staff support Incorporate metadata expectations into job descriptions and performance standards Build technical support Provide staff development opportunities The three “T’s”  Training  Tools  Time Make metadata part of the process

27. Develop templates to facilitate efficient and consistent metadata creation Build organizational support Identify pertinent fields within the metadata structure Populate fixed fields  Use standardized language  Define distribution methods  Cite standards used Build source and contact libraries Make metadata part of the process

28. Map metadata fields to the work flow Distribute the effort  Technicians – lineage  Analysts – process and methodology  Field Scientists – accuracy assessments  I.T. Managers – tools, automated collection methods, information management Make metadata part of the process Establish and assign responsibilities

29. Mandate the use of standards and templates Develop boilerplate metadata-deliverable language for data contractors Require publication of metadata Create and publish standard operating procedures to document metadata policies and procedures Establish standard policies Make metadata part of the process

30. Standardized Metadata

31. Why Have a Standard? Think for a moment how hard it would be to… … bake a cake without standard units of measurement. … put gas into your car without standard nozzle sizes. … plug a lamp into a socket without standard electrical outlets.

32. The standard for metadata ensures a level of consistency in data documentation. Standards ensure consistency. Why Have a Standard?

33. Establishing a Standard The Federal Geographic Data Committee (FGDC) was organized in 1990 under the Office of Management and Budget to promote the coordinated use, sharing, and dissemination of geospatial data on a national basis. The FGDC was tasked with creating a metadata standard to meet these objectives. The Metadata Standard

34. "... each agency shall document all new geospatial data it collects or produces, either directly or indirectly, using the standard under development by the FGDC, and make that standardized documentation electronically accessible to the Clearinghouse network." The Content Standard for Digital Geospatial Metadata (CSDGM) Executive Order 12906, 1994 Establishing a Standard

35. This “Content Standard” serves as a uniform summary description of the data set. Establishing a Standard

36. The Content Standard utilizes... Common terms Common definitions Common language Common structure Access constraints Citation currentness entity attribute domain lineage Process step Establishing a Standard

37. The Content Standard… Establishes names of compound elements and data elements. Defines the information about the values that are to be provided for the data elements. Type=“free text” Domain=Real currentness entity attribute Citation lineage Establishing a Standard

38. The Content Standard helps the user determine... If a set of geospatial data is available and fit for a particular use. How to access and transfer the data set. Establishing a Standard

39. Who Who collected the data? Who processed the data? Who wrote the metadata? Who to contact for questions? Who to contact to order? Who owns the data? Where Where were the data collected? Where were the data processed? Where are the data located? What What are the data about? What project were they collected under? What are the constraints on their use? What is the quality? What are appropriate uses? What parameters were measured? What format are the data in? When When were the data collected? When were the data processed? How How were the data collected? How were the data processed? How do I access the data? How do I order the data? How much do the data cost? How was the quality assessed? Why Why were the data collected? Metadata written using the Content Standard answers these important questions: Establishing a Standard

40. Metadata as a Data Discovery Tool

41. The FGDC metadata clearinghouse is a decentralized system of Internet servers you can use to search for available geospatial data. Discovering Data Through Metadata Client FGDC Gateway Servers housing metadata

42. The descriptive information that fuels the FGDC clearinghouse is metadata, which is collected in a standard format to facilitate query and consistent presentation across the multiple participating sites. Discovering Data Through Metadata

43. A Brief Look at the FGDC Clearinghouse The FGDC has six gateways to its clearinghouse system, with access to over 250 metadata servers. www.fgdc.gov/clearinghouse/clearinghouse.html

44. Searches can be performed by using the NSDI Search Wizard, or by using a map interface with place names, or by place names alone. A Brief Look at the FGDC Clearinghouse

45. The new NSDI “Smart Select” Search Wizard bins servers by the types of metadata they house. A Brief Look at the FGDC Clearinghouse

46. Searches can be performed using a map interface that allows the user to define an area of interest. An area of interest can be defined by dragging an area of interest box on the map interface. A Brief Look at the FGDC Clearinghouse

47. The selected area defines the bounding coordinates that will be used in the search. A Brief Look at the FGDC Clearinghouse

48. You can search all the servers listed, or you can select only those that interest you. A Brief Look at the FGDC Clearinghouse

49. Select individual servers of interest to your search. A Brief Look at the FGDC Clearinghouse

50. Search criteria can be further refined by time period of content and keywords. A Brief Look at the FGDC Clearinghouse

51. The status of each selected node is displayed as the search is conducted. A Brief Look at the FGDC Clearinghouse

52. When the search is complete, the status window lets you know if you were successful in discovering metadata that matched your search criteria. A Brief Look at the FGDC Clearinghouse

53. Select a server to see what metadata is available. A Brief Look at the FGDC Clearinghouse

54. Metadata discovered by the search is shown by title. A Brief Look at the FGDC Clearinghouse

55. Metadata record returned in HTML format. Links take you to each of the seven main sections of the record. A Brief Look at the FGDC Clearinghouse

57. The Coastal Information Directory (CID) at the NOAA Coastal Services Center is similar to the FGDC gateway interface, but the CID searches only those spatial data servers that house metadata of a coastal nature. www.csc.noaa.gov/CID/ A Brief Look at the FGDC Clearinghouse

58. For more information on the clearinghouse system, visit the FGDC Web site (www.fgdc.gov). Here you can find information on how to establish your own clearinghouse node using free Isite  software. On-line tutorials provide assistance for setting up and configuring this software. A Brief Look at the FGDC Clearinghouse

59. The Hunt for Good Metadata Exercise 1

60. Reading a Metadata File Exercise 2 Identification_Information: Citation: Citation_Information: Originator: NOAA Coastal Services Center Publication_Date: 200302 Title: Farallone Islands NMS Polyline File Edition: Geospatial_Data_Presentation_Form: Map Publication_Information: Publication_Place: Charleston, South Carolina Publisher: NOAA Coastal Services Center Other_Citation_Details: National Marine Sanctuary boundaries are defined both textually and through a listing of geographic coordinates. This information provided within the CFR was used to develop digital boundaries for the marine sanctuaries.

61. Review and Questions

62. Day Two

63. Let’s Review Metadata is a document that describes your data’s content, condition, and quality. Metadata is an integral component of your data and should never be viewed as a separate entity. Metadata is also a powerful tool that an organization can utilize to build a strong internal data management structure. Metadata’s traditional external value includes data discovery, assessment, access, and use. In order to be successful, this internal management structure requires building support at the administrative, technical, and organizational levels. Standardized metadata ensures consistency in documentation, and aids in the efficient discovery, access, transfer, and use of data.

64. Using the Content Standard

65. FGDC’s Content Standard for Digital Geospatial Metadata Workbook

66. The Content Standard is organized using numbered chapters called “sections.” Organization of the Content Standard Each section is organized into series of elements that define the information content for metadata to document a set of digital geospatial data.

67. Organization of the Content Standard Each section provides: Section name and definition Compound element names Compound element definitions Data element names Data element definitions

68. The workbook presents the production rules for the standard in a graphical format. These graphics illustrate: Organization of the Content Standard Structure of the standard Element groupings Element conditionality Element repeatability

69. Data Quality Information Spatial Data Organization Information Spatial Reference Information Entity and Attribute Information 4526731 Metadata The Three Supporting Sections 9 Time Period Information 10 Contact Information 8 Citation Information Distribution Information Metadata Reference Information Identification Information Structure of the Content Standard The Seven Main Sections

70. Interpreting the Graphical Production Rules Section Sections are depicted by this symbol. Compound Element Compound elements are depicted using a 2-dimensional box. Data Element Data elements are depicted using a 3-dimensional box.

71. Interpreting the Graphical Production Rules Data Element A data element is a logically primitive item of data. Data elements are the things that you “fill in.” The form for the definition of a data element is: Data element name -- definition. Type: (choice of “integer”, “real”, “text”, “date”, or “time”) Domain: (describes valid values that can be assigned) An example of the definition of a data element is: Abstract -- a brief narrative summary of the data set. Type: text Domain: free text Note: Data element definitions are contained in the text of the Content Standard, not in the graphical production rules.

72. Mandatory – Information for this element must be provided. Conditionality Definitions Element Conditionality Mandatory if Applicable – Information for this element must be provided if the data set exhibits the defined characteristic. Optional – Information for this element is provided at the discretion of the data set producer.

73. Compound Element 1 Compound Element 1.1 Data Element 1.1.1 Data Element 1.1.2 Data Element 1.2 How Elements Are Grouped Compound elements are composed of other compound and data elements. The composition is represented by nested boxes.

74. Compound Element 1 Compound Element 1.1 Data Element 1.1.1 Data Element 1.1.2 Data Element 1.2 Compound Element 1 is composed of Compound Element 1.1 and Data Element 1.2. Compound Element 1.1 is composed of Data Element 1.1.1 and Data Element 1.1.2. How Elements Are Grouped

75. If an element can be repeated independently from other elements, it will be indicated as such below the element name. Repeating Elements Compound Element 1 (can be repeated unlimited times) Compound Element 1.1 Data Element 1.1.1 Data Element 1.1.2 Data Element 1.2 This group of elements would repeat. Compound Element 1 Compound Element 1.1 Data Element 1.1.1 Data Element 1.1.2 Data Element 1.2

76. Conditionality Examples All elements are colored yellow, so all are mandatory and must be reported. Compound Element 1 Compound Element 1.1 Data Element 1.1.1 Data Element 1.1.2 Data Element 1.2

77. If yes, Data Elements 1.1.1 and 1.1.2 are mandatory. If no, do not report Compound Element 1.1, Data Element 1.1.1 or 1.1.2, and skip to Data Element 1.2. Compound Element 1 Data Element 1.1.1 Data Element 1.1.2 Data Element 1.2 Compound Element 1.1 Conditionality Examples Compound Element 1 is mandatory. Compound Element 1.1 is optional. Data Element 1.2 is mandatory.

78. Data Element 1.2 is optional. Compound Element 1 Compound Element 1.1 Data Element 1.1.1 Data Element 1.1.2 Data Element 1.2 Conditionality Examples Compound Element 1 is mandatory. Compound Element 1.1 is mandatory. Data Element 1.1.1 is mandatory. Data Element 1.1.2 is mandatory if applicable.

79. Compound Element 1 Compound Element 1.1 Data Element 1.1.1 Data Element 1.1.2 Data Element 1.2 Conditionality Examples Compound Element 1 is mandatory if applicable. If not applicable to the data set, do not report any elements. If applicable, it is mandatory and: Compound Element 1.1 is mandatory. Data Element 1.1.1 is mandatory if applicable. Data Element 1.1.2 is mandatory. Data Element 1.2 is optional.

80. Exercise 3 Using the Workbook This stuff is great!

81. Writing Metadata

82. First records are the hardest. Not all fields may need to be filled in. Tools are available. Training classes can be taken. Can often be produced automatically. Can (and should) be reviewed for updates. It’s not so bad!

83. Writing Metadata Organize your information Write your metadata file Review your file Have someone else review ReviseRevise PublishPublish Six Steps for Writing Quality Metadata.

84. Before you begin writing, get organized. Document your data as you go. Write so others can understand. Always review your document. Writing Metadata

85. Items required Sense of Humor! Chocolate FGDC Workbook Metadata entry tool Coffee

86. Write simply but completely. Document for a general audience. Be consistent in style and terminology. Keep your readers in mind. Writing Metadata

87. Define all acronyms. Avoid using jargon. Clearly state data limitations. Keep your readers in mind. Writing Metadata

88. Write a complete title that includes: What Where When Scale Who Writing Metadata

89. The title is critical in helping others find your data. Which is better? Greater Yellowstone Rivers from 1:126,700 Forest Visitor Maps (1961-1983) Rivers Writing Metadata

90. Be specific. Quantify when you can. Vague: We checked our work and it looks complete. Specific: We checked our work using 3 separate sets of check plots reviewed by 2 different people. We determined our work to be 95% complete based on these visual inspections. Writing Metadata

91. Select your key words wisely. Do not use ambiguous words. Use descriptive words. Fully qualify geographic locations. Writing Metadata

92. Have someone else read it. If you’re the only reviewer, put it away and read it again later. Check for clarity and omissions. Review your final product. Writing Metadata

93. Can a novice understand what you wrote? Are your data properly documented for posterity? When you review your work, ask: Writing Metadata

94. Does the documentation present all the information needed to use or reuse the data? Are any pieces missing? When you review your work, ask: Writing Metadata

95. Your audience may be very diverse. Consider writing your metadata to reflect this diversity.

96. Metadata Creation and Validation Tool Time

97. Tools for metadata creation NOAA CSC MetaScribe Allows you to create a template record that can be used to create large numbers of similar records. NOAA CSC ArcView Metadata Collector Extension for ArcView 3.x that allows user to capture metadata specific to a particular ArcView data set. TKME Straightforward text editor for metadata. ESRI ArcGIS metadata tool Found within the new ArcGIS application, this tool allows you to generate metadata for any ArcGIS data set.

98. Metadata Creation Tools TKME An editor for formal metadata, TKME is intended to simplify the process of creating metadata that conform to the standard.

99. NOAA CSC ArcView ® Metadata Collector The ArcView® Metadata Collection Tool was developed by the National Oceanic and Atmospheric Administration (NOAA) Coastal Services Center in ArcView using the Avenue® scripting language. This tool collects and compiles Federal Geographic Data Committee (FGDC)-compliant metadata for ARC/INFO coverage's, shapefiles, grids and supported image formats. Metadata Creation Tools

100. MetaScribe This new tool was also developed by the National Oceanic and Atmospheric Administration (NOAA) Coastal Services Center to aid in the creation of multiple sets of metadata that exhibit a high degree of redundancy. Metadata Creation Tools

101. ArcGIS metadata collector Found in ArcCatalog, this tool allows the user to write metadata within the Arc environment. Metadata Creation Tools

102. CNS (“Chew ‘n Spit”) A pre-parser for formal metadata designed to assist metadata managers convert records that cannot be parsed by mp into records that can be parsed by mp. Metadata Validation Tools MP (Metadata Parser) A compiler to parse formal metadata, checking the syntax against the FGDC Content Standard for Digital Geospatial Metadata and generating output suitable for viewing with a web browser or text editor.

103. TKME, CNS, and MP are available as free downloads from the United States Geological Survey (USGS) Website. (geology.usgs.gov/tools/metadata) TKME will run from a shortcut on the desktop, but both MP and CNS must be run from a command line in MS-DOS or UNIX. Metadata Validation Tools

104. Finally... Remember, metadata is an integral component of your data, and is vital for data sharing between organizations. Metadata can also provide many benefits at various levels within organizations by making the process of data management more streamlined and efficient.

105. Exercise 4 Time to Write!

106. Review and Questions

107. Michael Moeller Mike.Moeller@noaa.gov www.csc.noaa.gov/metadata/ Metadata Don’t Duck