XML and related technologies tutorial Developed using material at

Topics covered XML (.xml)– describe data XML Schema (.xsd)– validate data XPATH – navigate data XSLT (.xsl) – transform data

XML (.xml)– describe data XML stands for Extensible Markup Language. XML stands for Extensible Markup Language. XML is a markup language much like HTML XML is a markup language much like HTML XML was designed to describe data XML was designed to describe data XML tags are not predefined. You must define your own tags XML tags are not predefined. You must define your own tags XML uses an XML Schema(.xsd) to validate the data (or Document Type Definition (.dtd), …) XML uses an XML Schema(.xsd) to validate the data (or Document Type Definition (.dtd), …) XML with a XML Schema is designed to be self- descriptive XML with a XML Schema is designed to be self- descriptive XML can be used to create other XML based languages XML can be used to create other XML based languages

XML pros/cons Format Pro - Data and data description tags are written in text so they are portable and not dependent on proprietary formats or conversion processes for use. Pro - Data and data description tags are written in text so they are portable and not dependent on proprietary formats or conversion processes for use. Con – because data is verbosely described, larger datasets(e.g. model outputs) or binary formats(e.g. images) can be poor candidates for pure xml adoption. The common solution is to leave these types of data in their raw formats with use of some xml to describe the useful metadata(observation type, temporal/spatial range,…) of the file. Con – because data is verbosely described, larger datasets(e.g. model outputs) or binary formats(e.g. images) can be poor candidates for pure xml adoption. The common solution is to leave these types of data in their raw formats with use of some xml to describe the useful metadata(observation type, temporal/spatial range,…) of the file.Structure Pro - XML structure can be easily extended with the addition of elements/attributes as needed. Pro - XML structure can be easily extended with the addition of elements/attributes as needed. Con – deciding on the initial XML structure is driven by the application use of the data which can vary widely. Con – deciding on the initial XML structure is driven by the application use of the data which can vary widely.

XML Syntax <to>John</to><from>Jane</from><heading>Reminder</heading> Don't forget me this weekend! Don't forget me this weekend! </note> is the ‘root’ element of the document. is a parent element, is a child element and is a sibling element. is the ‘root’ element of the document. is a parent element, is a child element and is a sibling element. id=“100” is an attribute of the element. Attribute use should be limited, but is generally considered ok when referring to element metadata. id=“100” is an attribute of the element. Attribute use should be limited, but is generally considered ok when referring to element metadata. All elements must have a closing tag and be properly nested. All elements must have a closing tag and be properly nested. Tags are case sensitive. Tags are case sensitive. Attribute values must be quoted. Attribute values must be quoted.

Element Naming XML elements must follow these naming rules: Names can contain letters, numbers, and other characters Names can contain letters, numbers, and other characters Names must not start with a number or punctuation character Names must not start with a number or punctuation character Names must not start with the letters xml (or XML or Xml …) Names must not start with the letters xml (or XML or Xml …) Names cannot contain spaces(substitute underscore(_) instead) and should not use the colon(:) or dash(-) characters Names cannot contain spaces(substitute underscore(_) instead) and should not use the colon(:) or dash(-) characters

XML Schema (.xsd)– validate data defines elements that can appear in a document defines elements that can appear in a document defines attributes that can appear in a document defines attributes that can appear in a document defines which elements are child elements defines which elements are child elements defines the order of child elements defines the order of child elements defines the number of child elements defines the number of child elements defines whether an element is empty or can include text defines whether an element is empty or can include text defines data types for elements and attributes defines data types for elements and attributes defines default and fixed values for elements and attributes defines default and fixed values for elements and attributes

John John Smith Smith Several elements can refer to the same complex type

XPath – navigate data XPath is a syntax for defining parts of an XML document XPath is a syntax for defining parts of an XML document XPath uses path expressions to navigate in XML documents XPath uses path expressions to navigate in XML documents XPath contains a library of standard functions XPath contains a library of standard functions XPath is a major element in XSLT XPath is a major element in XSLT

#file obs_system.xml #file obs_system.xml no no yes yes bar bar yes yes yes yes bar bar

#!perl #!perl use strict; use strict; use XML::XPath; use XML::XPath; my $xp = XML::XPath->new(filename => 'obs_system.xml'); my $xp = XML::XPath->new(filename => 'obs_system.xml'); foreach my $element ($xp- )) { foreach my $element ($xp- )) { print $element->string_value()."\n"; print $element->string_value()."\n"; }

XSLT (.xsl) – transform data XSL stands for EXtensible Stylesheet Language. XSL stands for EXtensible Stylesheet Language. XSLT stands for XSL Transformations XSLT stands for XSL Transformations XSLT is the most important part of XSL XSLT is the most important part of XSL XSLT transforms an XML document into another XML document or output(see xsl:output method="text") XSLT transforms an XML document into another XML document or output(see xsl:output method="text") XSLT uses XPath to navigate in XML documents XSLT uses XPath to navigate in XML documents

Empire Burlesque Empire Burlesque Bob Dylan Bob Dylan USA USA Columbia Columbia

<xsl:stylesheet version="1.0" <xsl:stylesheet version="1.0" xmlns:xsl=" xmlns:xsl=" My CD Collection My CD Collection Title Title Artist Artist

Salinity Workshop Possible Web services which return data IOOS XML Schema for data returned IOOS XML Schema for data returned Salty Slim – more documentation at rkshop rkshop rkshop Same set of possible web services could also be used by generalized observing systems(NEON or GEOSS) or as example for USGS, NDBC, OBIS, etc web services

Possible web services Possible web services ##tell me the services you offer ##tell me the services you offer # GetCapabilities returns a list of methods available with their associated input vars and outputs. returns a list of methods available with their associated input vars and outputs. ##give me reference handles to the platforms, their position and what they collect ##give me reference handles to the platforms, their position and what they collect # GetPlatformList returns list of platform_id's and their associated geographic position and observation type(standard_names) which they collect returns list of platform_id's and their associated geographic position and observation type(standard_names) which they collect

##### ##### ##give me all the data ##give me all the data # GetLatest(optional: platform_id/bounding_box) returns a list of platform_id's and their corresponding latest observations(optionally for a specific platform_id/within the selected geographic bounding_box) returns a list of platform_id's and their corresponding latest observations(optionally for a specific platform_id/within the selected geographic bounding_box) ##give me just the observations requested ##give me just the observations requested # GetLatestByObservation(observation_standard_name[list?], optional: platform_id/bounding_box) returns a list of platform_id's and only the selected observation[list?] for the latest data(optionally for a specific platform_id/within the selected geographic bounding_box) returns a list of platform_id's and only the selected observation[list?] for the latest data(optionally for a specific platform_id/within the selected geographic bounding_box) ##### ##### ##instead of just the latest data, give me for the specified date range ##instead of just the latest data, give me for the specified date range ##give me all the data ##give me all the data # GetByDateRange(optional: platform_id/bounding_box, start_datetime, end_datetime) returns a list of platform_id's and and observations for data within the date range(optionally for a specific platform_id/within the selected geographic bounding_box) returns a list of platform_id's and and observations for data within the date range(optionally for a specific platform_id/within the selected geographic bounding_box) ##give me just the observations requested ##give me just the observations requested # GetByDateRangeByObservation(observation_standard_name[list?], optional: platform_id/bounding_box, start_datetime, end_datetime) returns a list of platform_id's and only the selected observation[list?] for data within the date range(optionally for a specific platform_id/within the selected geographic bounding_box) returns a list of platform_id's and only the selected observation[list?] for data within the date range(optionally for a specific platform_id/within the selected geographic bounding_box)

Salty Slim – more documentation at Salty Slim – more documentation at <ioos_data <ioos_data xmlns=" xmlns=" xmlns:xsi=" xmlns:xsi=" xsi:schemaLocation=" ioos_sst.xsd"> xsi:schemaLocation=" ioos_sst.xsd"> ndbc ndbc NameOfReferenceSystem NameOfReferenceSystem meters meters NGVD88 NGVD88 NAD83 NAD83

<platform> ndbc_41004 ndbc_41004 fixed_point fixed_point <url>

sstPrimary sstPrimary ndbc_41004_sstPrimary ndbc_41004_sstPrimary sea_surface_temperature sea_surface_temperature degree_Celsius degree_Celsius

*fixed_profile(wls,adcp,ctd) - with free_depth represented on a per attribute basis. omitted. *fixed_profile(wls,adcp,ctd) - with free_depth represented on a per attribute basis. omitted *fixed_depth(ships, floaters) - with free_latitude, free_longitude represented on a per attribute basis., omitted. *fixed_depth(ships, floaters) - with free_latitude, free_longitude represented on a per attribute basis., omitted *free(subs, tagged species) - 'free_latitude', 'free_longitude', 'free_depth' represented on a per attribute basis.,, omitted. *free(subs, tagged species) - 'free_latitude', 'free_longitude', 'free_depth' represented on a per attribute basis.,, omitted Technically speaking, everything could be represented in the 'free' type format, but it might be useful from a metadata/processing standpoint to know what the collection type is. Technically speaking, everything could be represented in the 'free' type format, but it might be useful from a metadata/processing standpoint to know what the collection type is.

#the following URL (which is the same as 'GetLatest') supports the Carolinas coast website latest observations #the following URL (which is the same as 'GetLatest') supports the Carolinas coast website latest observations ,22,-71.5,36.5&typename=latest_in_situ_obs ,22,-71.5,36.5&typename=latest_in_situ_obs #returns the following XML document (only one platform listing shown) #returns the following XML document (only one platform listing shown) , , , , , , , , , ,

carocoops_CAP1_wls carocoops_CAP1_wls :00: :00: :54: :54: :54: :54: m m ur_range=24&station_id=carocoops_CAP1_wls&lon= &lat=32.86&air_pressure_table=air_pressure_prod&archive_flag=&time_stamp=2005_09_16_17_54_00&pressure_ units=MB ur_range=24&station_id=carocoops_CAP1_wls&lon= &lat=32.86&air_pressure_table=air_pressure_prod&archive_flag=&time_stamp=2005_09_16_17_54_00&pressure_ units=MB in Hg (0 deg 3m in Hg (0 deg 3m itu.phtml?hour_range=24&station_id=carocoops_CAP1_wls&lon= &lat=32.86&air_pressure_table=air_pressure_prod&archive_flag=&time_stamp=2005_09_16_17_54_00&pressure_ units=INCHES_MERCURY itu.phtml?hour_range=24&station_id=carocoops_CAP1_wls&lon= &lat=32.86&air_pressure_table=air_pressure_prod&archive_flag=&time_stamp=2005_09_16_17_54_00&pressure_ units=INCHES_MERCURY :54: :54: deg 3m deg 3m ?hour_range=24&station_id=carocoops_CAP1_wls&lon= &lat=32.86&air_temperature_table=air_temperature_prod&archive_flag=&time_stamp=2005_09_17_03_54_00&de gree_units=C ?hour_range=24&station_id=carocoops_CAP1_wls&lon= &lat=32.86&air_temperature_table=air_temperature_prod&archive_flag=&time_stamp=2005_09_17_03_54_00&de gree_units=C deg 3m deg 3m tml?hour_range=24&station_id=carocoops_CAP1_wls&lon= &lat=32.86&air_temperature_table=air_temperature_prod&archive_flag=&time_stamp=2005_09_17_03_54_00&de gree_units=F tml?hour_range=24&station_id=carocoops_CAP1_wls&lon= &lat=32.86&air_temperature_table=air_temperature_prod&archive_flag=&time_stamp=2005_09_17_03_54_00&de gree_units=F......

SEACOOS XML Services epositorySeacoosXMLServices epositorySeacoosXMLServices Using an XML descriptor file to describe ASCII column oriented data for later processing Web forms simplify the process of creating needed XML Service currently exists for fixed point data which converts ASCII to SEACOOS netCDF Data scout currently converts netCDF to SQL for relational database population, but future efforts may skip netCDF step entirely

time,wind_speed,wind_from_direction,sea_surface_ temperature time,wind_speed,wind_from_direction,sea_surface_ temperature :00:00+00_SEP_5.0_SEP_ _SEP_ :00:00+00_SEP_5.0_SEP_ _SEP_ :00:00_SEP_6.0_SEP_125.0_SEP_ :00:00_SEP_6.0_SEP_125.0_SEP_ :00:00_SEP_7.0_SEP_130.0_SEP_ :00:00_SEP_7.0_SEP_130.0_SEP_ :00:00_SEP_8.0_SEP_135.0_SEP_ :00:00_SEP_8.0_SEP_135.0_SEP_15.0

CF-1.0 CF-1.0 SEACOOS-NETCDF-2.0 SEACOOS-NETCDF-2.0 SEACOOS-XML-1.0 SEACOOS-XML-1.0 <!-- format_category list <!-- format_category list [fixed-point,fixed-profiler,fixed-map,moving-point-2D,moving-point-3D,moving-profiler] [fixed-point,fixed-profiler,fixed-map,moving-point-2D,moving-point-3D,moving-profiler] --> --> fixed-point fixed-point Jeremy Cothran Jeremy Cothran Baruch Institute, University of South Carolina at Columbia Baruch Institute, University of South Carolina at Columbia carocoops carocoops CAP2 CAP2 buoy buoy

e_data/latest 2 _SEP_ e_data/latest 2 _SEP_

<dependent_variables> 2 wind_speed m s-1 2 wind_speed m s