1. ArcView Metadata Training: American Samoa James Byrne Josh Murphy NOAA Coastal Services Center July 29-30, 2003

2. Course Outline Part I: What Is Metadata? Part II: Metadata Creation Part III: Class Exercise Part IV: Clearinghouses

3. First…. Some Terms Metadata - Documentation of geospatial data written in a consistent manner FGDC - Federal Geographic Data Committee CSDGM - Content Standard for Digital Geospatial Metadata, referred to commonly as “The Standard” or “The Content Standard” Clearinghouse - A distributed catalog of metadata Geospatial - refers to a geographic location

4. Part I: What Is Metadata?

5. Simply put, metadata is information about your data. What is Metadata?

6. This is the metadata for this. What’s Missing? Aunu`u

7. Author(s) Boullosa, Carmen. Title(s) History of American Samoa / by Carmen Boullosa Place New York : Grove Press, 1997. Physical Descr viii, 180 p ; 22 cm. Subject(s) Oceana Format Reference Author(s) Boullosa, Carmen. Title(s) History of American Samoa / by Carmen Boullosa Place New York : Grove Press, 1997. Physical Descr viii, 180 p ; 22 cm. Subject(s) Oceana Format Reference This is the metadata for this. While the card-catalog entry is a form of metadata, it does not address topics such as quality, accuracy, or scale. Well-written geospatial metadata describes these and many more aspects of the data.

8. Identification_Information: Citation: Citation_Information: Originator: United States Department of Commerce, National Oceanic and Atmospheric Administration (NOAA), National Ocean Service (NOS), Coastal Services Center (CSC) Publication_Date: 20020923 Title: Vectorized Shoreline of Hawaii, Derived from Landsat ETM, 2002 Edition: first Geospatial_Data_Presentation_Form: vector digital data Publication_Information: Publication_Place: Charleston, SC Publisher: United States Department of Commerce, Identification_Information: Citation: Citation_Information: Originator: United States Department of Commerce, National Oceanic and Atmospheric Administration (NOAA), National Ocean Service (NOS), Coastal Services Center (CSC) Publication_Date: 20020923 Title: Vectorized Shoreline of Hawaii, Derived from Landsat ETM, 2002 Edition: first Geospatial_Data_Presentation_Form: vector digital data Publication_Information: Publication_Place: Charleston, SC Publisher: United States Department of Commerce, This is the metadata for this. What is Metadata?

9. Metadata contains vital information. Imagine, if you will… You are given two identical cans without labels. One contains cat food, the other contains tuna (dolphin-safe, of course). You must choose one these cans, and then eat the contents.

10. Other Examples? What is Metadata?

11. Properly documented data provides vital information to interested parties.

12. Metadata is that component of data which describes it. Environmental Sensitivity Index Data Metadata RARNUM - unique combination of species, concentration, and seasonality CONC (concentration) = Density species is found at location Season_ID = seasonality code link to the seasonal table Element - Biology group What is Metadata?

13. It’s data about a data set. Title Scale Source Content Location Publication Access Title Scale Source Content Location Publication Access MetadataMetadata GIS files Imagery Geospatial databases GPS data GIS files Imagery Geospatial databases GPS data Data set What is Metadata?

14. Metadata describes… CONTENT CONDITION QUALITY Characteristics of the data Characteristics of the data What is Metadata?

15. Metadata Non-spatial or attributes Spatial Because metadata provides vital information about a dataset, it should never be viewed or treated as a separate entity. Take Home Message Metadata is a critical and integral component of any complete data set. Metadata is a critical and integral component of any complete data set.

16. Two similar paintings by Picasso up for auction sold for vastly different prices. Why? One had metadata......One didn’t. The Value of Metadata

17. Metadata should be updated to reflect changes in the data.1970HEW Western Samoa British Honduras Trust Territory Cape Hatteras Light Arpanet Mt. St. Helen West Germany 2000 HHS & HUD SamoaBelize U.S. Commonwealth Cape Hatteras Light Internet Mt. St. Helen Germany

18. Metadata has other value associated with it.

19. Avoid duplication Avoid duplication Share reliable information Share reliable information Publicize efforts Publicize efforts Reduce workload Reduce workload For data developers, metadata... The Value of Metadata

20. Facilitates understanding Facilitates understanding Focuses on key elements Focuses on key elements Enables discovery — inside and outside of organizations Enables discovery — inside and outside of organizations For data users, metadata... The Value of Metadata

21. For organizations, metadata... The Value of Metadata Protect investment in data Create an institutional memory Counter personnel changes Allow sharing of data with other agencies Reduce costs Limit potential liability Save time and money

22. Metadata as a “data discovery” tool This saves time and money. If it’s geospatial data you need, metadata helps Find data of interest Determine the usefulness of the data Determine how to access the data The Value of Metadata

23. Metadata Standards Think for a moment how hard it would be to… … bake a cake without standard units of measurement. … put gas into your car without standard nozzle sizes. … plug a lamp into a socket without standard electrical outlets.

24. The standard for metadata ensures a level of consistency in data documentation. Standards ensure consistency. Metadata Standards

25. The Federal Geographic Data Committee (FGDC) was organized in 1990 under the Office of Management and Budget to promote the coordinated use, sharing, and dissemination of geospatial data on a national basis. The FGDC was tasked with creating a metadata standard to meet these objectives. The Metadata Standard Metadata Standards

26. "... each agency shall document all new geospatial data it collects or produces, either directly or indirectly, using the standard under development by the FGDC, and make that standardized documentation electronically accessible to the Clearinghouse network." The Content Standard for Digital Geospatial Metadata (CSDGM) Executive Order 12906, 1994 Metadata Standards

27. The Content Standard utilizes... Common terms Common terms Common definitions Common definitions Common language Common language Common structure Common structure Access constraints Citation currentness entity attribute domain lineage Process step Metadata Standards

28. The Content Standard helps the user determine... If a set of geospatial data is available and fit for a particular use. How to access and transfer the data set. Metadata Standards

29. FGDC’s Metadata Workbook Defines the 334 metadata elements. Metadata Standards

30. What do I use “The Workbook” for? It is the definitive resource for applying the FGDC Content Standard. However, it does not define the production rules. It describes element domain values, which are valid values that can be assigned to the data element. It provides section and element definitions.

31. Interpreting the Metadata Workbook A data element is a logically primitive item of data. Data elements are the things that you “fill in.” The form for the definition of a data element is: Data element name -- definition. Type: (choice of “integer”, “real”, “text”, “date”, or “time”) Domain: (describes valid values that can be assigned) An example of the definition of a data element is: Abstract -- a brief narrative summary of the data set. Type: text Domain: free text Note: Data element definitions are contained in the text of the Content Standard, not in the graphical production rules.

32. It is a quick reference for production rules and structure. Use the “Graphical Representation” for quick access. You will still need to use the workbook to find the definition of a particular element and its domain. Organization of the Content Standard

33. Data Quality Information Spatial Data Organization Information Spatial Reference Information Entity and Attribute Information 4526731 Metadata The Three Supporting Sections 9 Time Period Information 10 Contact Information 8 Citation Information Distribution Information Metadata Reference Information Identification Information Organization of the Content Standard The Seven Main Sections

34. Mandatory - must be provided. Meaning Data Element Compound Element What’s Mandatory? What’s Not? Mandatory if Applicable - must be provided if the data set exhibits the defined characteristic. Optional - provided at the discretion of the data set producer.

36. Remember, metadata is a legacy document that concisely sums up your data or data set. Without metadata, your data set is incomplete.

37. FOR MORE INFORMATION Michael Moeller NOAA CSC Metadata Specialist Mike.Moeller@noaa.gov www.csc.noaa.gov/metadata/

38. Part II: Metadata Creation

39. Writing Metadata is not THAT bad! First records are the hardest. Not all fields may need to be filled in. Tools are available. Training classes can be taken. Can often be produced automatically. Can (and should) be reviewed for updates.

40. Templates can help!! Contain information that is specific to your project or organization Allow you to enter and save pertinent information for use at later date Save time and effort! No special tools needed! Like any other metadata, can (and should) be reviewed for updates. Writing Metadata

41. Before you begin writing, get organized. Writing Metadata

42. Document your data as you go. Writing Metadata

43. Write so others can understand. Writing Metadata

44. Define all acronyms. Avoid using jargon. Clearly state data limitations. Writing Metadata Keep your readers in mind.

45. Always review your document. Writing Metadata

46. Items required Chocolate FGDC Workbook Metadata entry tool

47. Tool Time A sample of some of the available tools for metadata creation, validation, and publication. CNS and MP “Chew ‘n spit,” checks and corrects structural errors, and “Metadata Parser”, which checks for errors in element compliance. NOAA CSC MetaScribe Allows you to create a template record that can be used to create large numbers of similar records. NOAA CSC ArcView Metadata Collector Extension for ArcView 3.x. TKME Text editor used for metadata entry. ArcCatalog Metadata Editor Metadata creation in ArcGIS

48. TKME An editor for formal metadata. TKME is intended to simplify the process of creating metadata that conform to the content standard

49. Metadata Collector for ArcView 3.2 A free, downloadable extension for ArcView 3.x users Enters information for certain elements automatically Allows user to save individual elements for future use

50. At the right side of most of the screens will be the option to Retrieve and Save the information you have entered into the fields. This allows you to save frequently used information such as contact and similar abstracts to be used repeatedly. These are generally saved as either.dbf or.txt files in the working directory specified. On each screen, there is a Section Help and a Section Example button. The Section Help button opens a window containing definitions of the metadata elements for that section. The Section Example button provides an example of an FGDC-compliant metadata record. Metadata Collector for ArcView 3.2

51. Certain sections are read directly from the data set by the ArcView Metadata Collector Metadata Collector for ArcView 3.2

52. When you have finished all of the metadata sections, you will be presented with the option to save the metadata file as both a text and an HTML file. Metadata Collector for ArcView 3.2

53. MetaScribe A tool for the creation of multiple, similar metadata records Template driven

54. Metadata Parser (mp) DOS-based tool that compiles and checks the syntax of “raw” metadata Input must be either text, XML, or SGML Creates outputs in multiple formats (HTML, TXT, XML, SGML) Also creates an error file

55. ArcCatalog Metadata Editor Included with basic ArcGIS 8.x software Simple to use: Projection parameters automatically captured (if.prj file present) Can use template to automatically fill in contact, distribution information

56. Using the AV Metadata Collector Extension For the remainder of this section, the instructor will demonstrate metadata creation using the FGDC Content Standard and the ArcView Metadata Collector extension.For the remainder of this section, the instructor will demonstrate metadata creation using the FGDC Content Standard and the ArcView Metadata Collector extension.

57. Section 1 – Identification Information Section 2 – Data Quality Information Section 3 – Spatial Data Organization Information Section 4 – Spatial Reference Information Section 5 – Entity and Attribute Information Section 6 – Data Distribution Information Section 7 – Metadata Reference Information Review: The Seven Main Sections of the Content Standard Using the AV Metadata Collector Extension

58. Data Set Example Tutuila Shoreline

59. Section 1 – Identification Information Basic Information about the data set.

60. Section 1 – Identification Information Metadata

61. The title is critical in helping others find your data. Which is better? Greater Yellowstone Rivers from 1:126,700 Forest Visitor Maps (1961-1983) Section 1

62. The title is the first thing a user sees when searching for data The title helps a user quickly determine the usefulness of data Include topic, time and place! Bad title (user has to read description): “Roads” Better title: “Roads Shapefile for Guam - 2000” Title, title, title Section 1

63. Bounding Coordinates West_Bounding_Coordinate: 145.69 East_Bounding_Coordinate: 145.83 North_Bounding_Coordinate: 15.29 South_Bounding_Coordinate: 15.09 15.29, 145.69 15.09, 145.83

64. Select your key words wisely. Use unambiguous words. Use descriptive words. Fully qualify geographic locations. Key Words

65. Theme: Theme_keyword_thesaurus: None Theme_keyword: major roads Theme_keyword: highways Theme_keyword: transportation network Theme_keyword: thoroughfares Theme_keyword: vector digital data Theme_keyword: geographic information system Key Words

66. Access and Use Constraints Essentially liability statements Example: Section 1 Use_Constraints: Data set is not for use in litigation. While efforts have been made to ensure that these data are accurate and reliable within the state of the art, NOAA, cannot assume liability for any damages, or misrepresentations, caused by any inaccuracies in the data, or as a result of the data to be used on a particular system. NOAA makes no warranty, expressed or implied, nor does the fact of distribution constitute such a warranty. Use_Constraints: Data set is not for use in litigation. While efforts have been made to ensure that these data are accurate and reliable within the state of the art, NOAA, cannot assume liability for any damages, or misrepresentations, caused by any inaccuracies in the data, or as a result of the data to be used on a particular system. NOAA makes no warranty, expressed or implied, nor does the fact of distribution constitute such a warranty.

67. Link to on-line data Other_Citation_Details: Online_Linkage: http://www.csc.noaa.gov/hurricane_tracks.shp

68. Section 1 – Identification Information Metadata

69. Section 2 – Data Quality Information A general assessment of the quality of the data set.

70. Section 2 – Data Quality Information A general assessment of the quality of the data set.

71. How thoroughly and correctly the features in the data set are described. What features have been omitted ? What non-existent features are represented ? How correct is their classification ? How can you test and report Attribute Accuracy? Field Verification Visual comparison of source data to hard copy Attribute Accuracy

72. Attributes How accurate are these attributes? Attribute Accuracy

73. DATA_QUALITY_INFORMATION Attribute_Accuracy Attribute_Accuracy_Report: A team of field investigators participated in data verification exercises June, 1996 and December, 1996. Data validation teams consisted of personnel from Oak Ridge National Laboratory, Moss Landing Marine Laboratories, Ray L. Harris, Jr., and the Coastal Services Center. The team was equipped with a portable color laptop computer linked to a Global Positioning System (GPS). The field station runs software that supports the classified data as a raster background with the road network as a vector overlay with a simultaneous display of live GPS coordinates. Accuracy assessment points were generated with Erdas Imagine software using a stratified random sample. To make the acquisition of the field reference data more practical, a sixteen pixel buffer area around roads (i.e. 8 pixels on each side of the road) including logging trails was created. Seven thousand random points were generated within this area for the accuracy assessment. See the accuracy assessment table for results of data verification exercise. Attribute Accuracy Example Attribute Accuracy Report

74. A closer look Do these footprints accurately represent these? Attribute_Accuracy: Attribute_Accuracy_Report: Attribute accuracy was tested by manual comparison of the source with hardcopy printouts of the digital building footprint data. The attributes were visually compared to attributes in the source data (digital orthophoto). Attribute Accuracy

75. Addresses geometric problems in your dataset -overshoots -undershoots -broken polygons -missing or duplicate labels A typical Logical Consistency Statement: Logical_Consistency_Report: These data are believed to be logically consistent, though no tests were performed. There are no overshoots, undershoots or broken polygons. Line geometry is topologically clean. Logical Consistency

76. Geometric/Logical Consistency Problems

77. How the coordinate descriptions of features compare to the actual locations of those features on the ground. How far away is a map feature from its actual location in the world? How can you test for and report Positional Accuracy? Comparison of your data to more accurate data (survey data) GPS accuracy Using the National Map Accuracy Standards (positional accuracy for paper maps only) Positional Accuracy

78. Q: How horizontally accurate is the position of this shoreline? A: As accurate as the data it was derived from. Positional Accuracy

79. Examples Using National Map Accuracy Standards Using GPS accuracy

80. Be specific. Quantify when you can. Vague: We checked our work and it looks complete. Specific: We checked our work using 3 separate sets of check plots reviewed by 2 different people. We determined our work to be 95% complete based on these visual inspections. Process Step

81. Section 2 – Data Quality Information Metadata

82. Section 3 – Spatial Data Organization Information The mechanism used to represent spatial Information in the data set.

84. Spatial Data Organization Information What is the Structure of the Data? Raster Vector Row and Column count Vector Object Count

85. Section 3 – Spatial Data Organization Information Metadata

86. Section 4 – Spatial Reference Information The description of the reference frame for, and The means to encode coordinated in the data set.

88. Section 4 - Spatial Reference Information Three Choices under Horizontal Coordinate System Geographic Latitude and Longitude Planar Map Projection Grid Coordinate System Local Planar Local

89. Geographic Coordinate System Latitiude and longitude measurements define a position on the Earth. Units of measurement can be decimal degrees decimal minutes decimal seconds degrees and decimal minutes degrees minutes and seconds radians grads

90. Planar Systems Map Projection - representation of the earth’s surface East-West Projections: Lambert, Albers (Conic) Used in Florida, Massachusetts, and Ohio North-South Projections: Transverse Mercator (Cylindrical) Used in Florida and California Grid Coordinate Systems - State Plane and UTM Used by states as common coordinate systems Based on map projections

91. Datums A datum is a set of parameters defining a coordinate system. Typically, your data’s projection will be referenced to either North American Datum NAD 83 World Geodetic System 1984 North American Datum NAD 27

92. Section 4 – Spatial Reference Information Metadata

93. Section 5 – Entity and Attribute Information Information about the content of the data set, including the entity types, their attributes, and the attribute values that may be assigned.

95. Attribute Table

96. attributes attribute values Attribute Table

97. Detailed Attribute Description Not only do you have to describe your attributes but you also have to describe the values Attribute Domains: domains are used to describe attribute values 1) Range Domain: a sequence of values - if the attribute is “water_depth” and the values are between 0 and 150, a range domain is suitable. 2) Enumerated Domain: comprised of a list of values - if the attribute is “landcover_code” and the list of values is between 1 16, then an enumerated domain should be used. 3) Codeset Domain: data values are defined by a set of codes - if the attribute is “wetland_id#” and the codes are defined in a standard or code book, then the codeset domain should be used. 4) Unrepresentable Domain: the set of data values cannot be represented - numbers assigned by software program that are meaningless

98. Enumerated Domain To understand this legend, each value in this enumerated domain will have to be defined.

99. Section 5 – Entity and Attribute Information Metadata

100. Section 6 – Distribution Information Information about the distributor of and options for obtaining the data set.

102. Section 6 – Distribution Information Metadata

103. Section 7 – Metadata Reference Information Information on the currentness of the metadata information, and the responsible party.

105. Section 7 – Metadata Reference Information Metadata

106. Have someone else read it. If you’re the only reviewer, put it away and read it again later. Check for clarity and omissions. Review your final product. Reviewing Metadata

107. Can a novice understand what you wrote? Are your data properly documented for posterity? When you review your work, ask: Reviewing Metadata

108. Does the documentation present all the information needed to use or reuse the data? Are any pieces missing? When you review your work, ask: Reviewing Metadata

109. Part III: Class Exercise

112. Part IV: Clearinghouses

113. Clearinghouses A metadata clearinghouse is a location — typically accessed through the Internet — to search for spatial data sets

114. A clearinghouse is a decentralized system of Internet servers you can search Servers with metadata ClientClearinghouses

115. Clearinghouses make metadata records easy to find Clearinghouses

116. The National Geospatial Data Clearinghouse has more than 100 spatial data servers with digital geographic data Clearinghouses

117. The NGDC is a set of information services that use hardware, software, and telecommunications networks to provide searchable access to information Clearinghouses

118. FGDC Geospatial Data Clearinghouse Montana State Library Natural Resource Information System GIS Nebraska Geospatial Data Clearinghouse Wisconsin Land Information Clearinghouse Some specific examples: Clearinghouses

119. You can search all or part of the community in a single session xClearinghouses

120. Clearinghouses www.fgdc.gov

121. You can define your criteriaClearinghouses

122. You can select a serverClearinghouses

123. You can view your search resultsClearinghouses

124. The FGDC Clearinghouse selected the search and retrieve software ANSI Z39.50-1995 (ISO 10163-1995) Clearinghouse Implementation

125. How can you participate? Set up your own node Send metadata to an existing node Have another organization host your node Put metadata on a web page for future “harvesting” to a node Clearinghouse Implementation

126. Why set up a node? You You control your metadata It’s easy to set up You choose the software The software operates on typical web server platforms Clearinghouse Implementation

127. In setting up a node, most work is in preparing metadata for available data sets Store these data in a structured form, according to the standard Clearinghouse Implementation

128. Before you can index your metadata into a clearinghouse, you must check for completeness, accuracy, and quality Clearinghouse Implementation

129. Your metadata must meet clearinghouse requirements: Proper format Proper field names and values Appropriate formats: text, HTML, and fgfSGML Clearinghouse Implementation

130. Parsing and validation tools check for structural or content errors errorreports metadatadocuments metadata parser software mp Clearinghouse Implementation

131. Run cns output through mp to make text, SGML, HTML files YES Correct original file Run original file through cns Errors in error.txt?YES Create Metadata Create Metadata Run cns output through mp NO Correct original file Place 3 files on searchable system Errors in error.txt? Errors in error.txt? NO Clearinghouse Implementation

132. mp produces metadata in HTML, SGML, and TEXT mpmp TEXTTEXT HTMLHTML SGMLSGML Clearinghouse Implementation

133. 1989 Land Cover/Land Use on the Upper Mississippi River System 1989 Land Cover/Land Use on the Upper Mississippi River System Metadata: Identification_Information Data_Quality_Information Spatial_Data_Organization_Informati on 1989 Land Cover/Land Use on the Upper Mississippi River System 1989 Land Cover/Land Use on the Upper Mississippi River System Metadata: Identification_Information Data_Quality_Information Spatial_Data_Organization_Informati on mp HTML file Clearinghouse Implementation

134. mp HTML file Clearinghouse Implementation

135. Kurt P. Kowalski and Douglas A. Wilcox Great Lakes Science Center Biological Resources Division 1997 Coastal Wetland Vegetation Analysis (Metzger Marsh) Ann Arbor, MI Kurt P. Kowalski and Douglas A. Wilcox Great Lakes Science Center Biological Resources Division 1997 Coastal Wetland Vegetation Analysis (Metzger Marsh) Ann Arbor, MI mp SGML file Clearinghouse Implementation

136. Identification_Information: Citation: Citation_Information: Originator: Environmental Management Technical Center Publication_Date: 19950829 Title: 1989 Land Cover/Land Use on the Upper Mississippi River System Geospatial_Data_Presentation_Form: Map Publication_Information: Publication_Place: Environmental Management Technical Center Publisher: Environmental Management Technical Center (EMTC) Other_Citation_Details: White, B. M. and T. W. Owens. 1991. Geographic Information System Pilot Project For The Upper Mississippi River System. U.S. Fish and Wildlife Service, National Ecology Research Center, Fort Collins, Colorado, June 1991. LTRMP 91-05. 48 pp. + Appendix. Identification_Information: Citation: Citation_Information: Originator: Environmental Management Technical Center Publication_Date: 19950829 Title: 1989 Land Cover/Land Use on the Upper Mississippi River System Geospatial_Data_Presentation_Form: Map Publication_Information: Publication_Place: Environmental Management Technical Center Publisher: Environmental Management Technical Center (EMTC) Other_Citation_Details: White, B. M. and T. W. Owens. 1991. Geographic Information System Pilot Project For The Upper Mississippi River System. U.S. Fish and Wildlife Service, National Ecology Research Center, Fort Collins, Colorado, June 1991. LTRMP 91-05. 48 pp. + Appendix. mp TEXT file Clearinghouse Implementation

138. For more information on clearinghouse implementation: Clearinghouse Implementation www.fgdc.gov/clearinghouse/tutorials/howto.html OR Please contact John Ulmer NOAA CSC Computer Specialist John.Ulmer@noaa.gov

139. Any Questions? Clearinghouse Implementation