+ IRIS Data Services Shortcourse December 15, 2014 MOMA Room, Palomar Hotel

+ Agenda 8:00Welcome and IntroductionsTim Ahern 8:10Rick Benson, DMC Operations What Types of Data Does the DMC Manage Focus on raw seismic data How data are referenced in SEED using naming conventions, UTC Times, Virtual Networks 8:25Chad Trabant, Products and Services Fundamentals of Web Services Review of Web Services at the DMC Exercise using web services, URL builders Web Services beyond the browser: Fetch scripts, Java, MATLAB, ObsPy Exercise Using FetchData 9:10Manoch Bahavar, Product Specialist Event related data access with WILBER3, a browser based web services client 9:30Rick Benson Other DMC Services: SeedLink and BreqFast 9:45Mary Templeton, Quality Assurance Analyst The IRIS MUSTANG Quality Assurance System MUSTANG system MUSTANG Data Browser LASSO Automated scripts for advanced QA 10:30Tim Ahern Research Ready Data Sets Answering your questions

+ What Data Does DMC Have?

+ Virtual Networks: Currently 54

+ USArray Virtual Networks Virtual NetworkDescription _US-TA USArray Transportable Array _US-REF USArray Reference Network _US-FAUSArray Flexible Array _US-MTUSArray Magnetotelluric _US-ALLAll USArray Components _PBO Plate Boundary Observatory _SAFODSAFOD

+ An overview of web services service.iris.edu Web services use HTTP to get information from IRIS to your computer. Next: the basics and what the IRIS DMC has to offer.

+ Web Service Requests Client URL (with parameter list..) Header & Content Web Services

+ Communicate using client Client URL (with parameter list..) Header & Content Web Services

+ Communicate using client URL (with parameter list..) Header & Content Web Services Browser cURL Fetch irisFetch Your Own

+ Request is passed as URL Client Header & Content Web Services URL (with parameter list..) URL (with parameter list..)

+ Web Service executes request Client Header & Content URL (with parameter list..) URL (with parameter list..) Web Services

+ Content is returned Client URL (with parameter list..) URL (with parameter list..) Header & Content Web Services

+ Web Service Valid Request ? Data Exists? ✔ ✔ 400 – Invalid parameter PCLOADLETTER 200 – Data Data Data Data Data 404 (204) – Not Found ✖ ✖ Get Data A web service in action

+ Available via services… service.iris.edu

+ FDSN standardization Common set of interfaces: fdsnws-dataselect fdsnws-station fdsnws-event What you learn here applies to other datacenters.

service.iris.edu

+ Each service is fully documented URL Builder Description Query Parameters Examples Detailed Parameter Descriptions Relevant Links

+ Quickly access and explore web service data using the URL Builders

+ Web Services exercise Plot 35 minutes of 3-Component data with event From network IU and station(s) A?M? For the LH? channels Starting 5 min before P wave arrival For the largest shallow (<= 10km) event occurring within the last month. Rotated into RADIAL and TRANSVERSE components Use web services to do it. service.iris.edu

+ Web Services exercise Event – get event time, lat, lon, depth for largest shallow ( <10km depth) event in last 30 days. 2. Station – get station name, lat & lon for net: IU, station: A?M? 3. Distaz – get distance & azimuth 4. Traveltime – Find first P arrival 5. Rotation – plot a response corrected, ZRT rotated seismogram for 5 min before P arrival to 30 minutes after P arrival. for LH? channels. Taking notes will be helpful service.iris.edu

+ Web Services exercise Exercise 1. Event – get event time, lat, lon, depth for largest shallow ( <10k depth) event in last 30 days. 2. Station – get station name, lat & lon for net: IU, station: A?M? 3. Distaz – get distance & azimuth 4. Traveltime – Find first P arrival 5. Rotation – plot a response corrected, ZRT rotated seismogram for 5 min before P arrival to 30 minutes after P arrival. for LH channels. Hints. Builders help… EVENT Text output order by magnitude change dates & times max depth don’t need ALL events STATION text output, station level TRAVELTIME Only need min P phase traveltime ROTATION Rotate into ZRT service.iris.edu

+ About using the Browser… Interactive Builders only create URLs. Faster to modify/create the URL by hand Browser can only handle a certain amount of data. Lots of XML may crash your browser Requires your direct interaction

+ Web services: Beyond the Browser - via HTTP clients, such as wget, curl, etc These often are already on many systems - via DMC-provided interfaces: - command line: Fetch family of Perl scripts - Java: IRIS Java Web Service Library (IRIS-WS library) - MATLAB: irisFetch.m - via Community developed solutions: - ObsPy, SOD, Waveform Suite, EMERALD, SeisFile, etc. - via WILBER3 the JWEED and future desktop applications

+ Automate access with FETCH scripts

+ Fetch clients... Allow command-line access to IRIS-held data Have options that map to the service’s parameters Fetch clients and mseed2sac are available from: More usage details are available from:

+ The DMC’s Fetch web service clients FetchData: Fetch miniSEED data, simple metadata, and instrument responses (SEED RESP & SAC PZs) FetchEvent: Fetch Event (earthquake) information as text or XML FetchMetadata: Fetch primary channel metadata (coordinates, etc.) as text or XML service.iris.edu/clients

+ Usage: FetchEvent [options] -v More verbosity, may be specified multiple times (-vv, -vvv) -s starttime Limit to origins after time (YYYY-MM-DD,HH:MM:SS.sss) -e endtime Limit to origins before time (YYYY-MM-DD,HH:MM:SS.sss) --lat min:max Specify a minimum and/or maximum latitude range --lon min:max Specify a minimum and/or maximum longitude range --radius lat:lon:maxradius[:minradius] Specify circular region with opt. minimum radius --depth min:max Specify a minimum and/or maximum depth in kilometers --mag min:max Specify a minimum and/or maximum magnitude --magtype type Specify a magnitude type for magnitude range limits --cat name Limit to origins from specific catalog (e.g. ISC, PDE, GCMT) --con name Limit to origins from specific contributor (e.g. ISC, NEIC) --ua date Limit to origins updated after date (YYYY-MM-DD,HH:MM:SS) --allorigins Return all origins, default is only primary origin per event --allmags Return all magnitudes, default is only primary magnitude per event --orderbymag Order results by magnitude instead of time --evid id Select a specific event by DMC event ID --orid id Select a specific event by DMC origin ID -X xmlfile Write raw returned XML to xmlfile -A appname Application/version string for identification -o outfile Write event information to specified file, default: console --help

+ Example: Quick access to metadata $ FetchMetadata -N TA -S ELFS 29  The existence of metadata does not guarantee that time series data exists Received KB of metadata in 0.1 seconds (796.2 KB/s) Processed metadata for 114 channel epochs in 0.4 seconds (316.7 KB/s) #net|sta|loc|chan|lat|lon|elev|depth|azimuth|dip|instrument|scale|scalefreq|scaleunits|samplerate|start|end TA|ELFS|--|ACE| | |1553.0|0.0|0.0|0.0|Quanterra 330 Linear Phase Composite||0||0| T21:01:00| T18: TA|ELFS|--|BHE| | |1553.0|0.0|84.9|0.0|Streckeisen STS-2 G3/Quanterra 330 Linear Phase Co| E8|0.2|M/S|40|2007 TA|ELFS|--|BHN| | |1553.0|0.0|354.9|0.0|Streckeisen STS-2 G3/Quanterra 330 Linear Phase Co| E8|0.2|M/S|40|200 TA|ELFS|--|BHZ| | |1553.0|0.0|0.0|-90.0|Streckeisen STS-2 G3/Quanterra 330 Linear Phase Co| E8|0.2|M/S|40|200 TA|ELFS|--|LCE| | |1553.0|0.0|0.0|0.0|Quanterra 330 Linear Phase Composite| |0|S|1| T21:01:00|2007 TA|ELFS|--|LCQ| | |1553.0|0.0|0.0|0.0|Quanterra 330 Linear Phase Composite|1.0|0|PERCENT|1| T21:01:00|2007 TA|ELFS|--|LHE| | |1553.0|0.0|84.9|0.0|Streckeisen STS-2 G3/Quanterra 330 Linear Phase Co| E8|0.03|M/S|1|2007 TA|ELFS|--|LHN| | |1553.0|0.0|354.9|0.0|Streckeisen STS-2 G3/Quanterra 330 Linear Phase Co| E8|0.03|M/S|1|200 TA|ELFS|--|LHZ| | |1553.0|0.0|0.0|-90.0|Streckeisen STS-2 G3/Quanterra 330 Linear Phase Co| E8|0.03|M/S|1|200 TA|ELFS|--|LOG| | |1553.0|0.0|0.0|0.0|Quanterra 330 Linear Phase Composite||0||0| T21:01:00| T18: TA|ELFS|--|OCF| | |1553.0|0.0|0.0|0.0|Quanterra 330 Linear Phase Composite||0||0| T21:01:00| T18: TA|ELFS|--|UHE| | |1553.0|0.0|84.9|0.0|Streckeisen STS-2 G3/Quanterra 330 Linear Phase Co| E7|0.002|M/S|0.01|

+ FetchData exercise Request 1 hour of GSN long-period vertical (LHZ) data and simple metadata for M8.8 Chilean earthquake: $ FetchData -N _GSN –L 00 –C 'LHZ' -s ,06:34:00 -e ,07:34:00 -o /data/Chile-GSN-LHZ.mseed -m /data/Chile-GSN-LHZ.metadata Convert the miniSEED to SAC with metadata $ mseed2sac Chile-GSN-LHZ.mseed –m Chile-GSN- LHZ.metadata -E '2010,058,06:34:11/ / /22.9'

+ Broad support for FDSN services

+ Integrating & customizing for your own use #!/usr/bin/bash # usage:./fetchingscript minmag:maxmag startday endday a=$(./FetchEvent --mag ${1} --s ${2} --s ${3} --orderbymag --limit 1) # parse output a=${a%*|*|*|*|*|*} # strip off the end a=${a//[[:space:]]} # remove all spaces IFS=' ' read -r evid evdt evlat evlon evdep <<< ${b//|/ } # split fields # wrestle dates evdate=${evdt:0:10} evtime=${evdt:10} evendtime=$(date -j -f "%T" -v+10M ${evtime} "+%H:%M:%S") #add 10 minutes evstart="${evdate}T${evtime}" evend="${evdate}T${evendtime}"./FetchData -N _GSN -C BHZ -s ${evstart} -e ${evend} -o data.mseed -m data.meta -A test --radius ${evlat}:${evlon}:85:95./mseed2sac data -m data.meta data.mseed

+ WILBER3 Event-related data access Purpose: Leverage IRIS/FDSN web services to request event-related data Scenario: Need waveforms relating to a specific event Stations with spread Choose distances & azimuths Preview ability Interface: Web based, all you need is a browser ds.iris.edu/wilber3

+ Waveform access, by need

+ Batch REQuest FAST (BREQ_FAST) Purpose: Request large volume of SEED information asynchronously Scenario: You’re away, and need to get data. Not at your computer Slow internet connection Need large volume of data Don’t/can’t need to work on it right away

+ Order SEED now, process later with BREQ_FAST

+ BREQ_FAST via SeismiQuery

+ BREQ_FAST via SeismiQuery

+ BREQ_FAST via SeismiQuiery

+ BREQ_FAST via SeismiQuiery

+ Request from BREQ_FAST Via Request SEED : Request SEED metadata: Request miniSEED : Manual: Joe Seismologist. FTP.LABEL Earthquake1.QUALITY B.END GRFO IU SHZ ANTO IU SH? AFI IU BH? 00 SEE CD SHZ CASY IU BHZ 10 KMI CD BHZ SSE CD B?? SHZ PAS TS BH? SHZ L??

+ Real-Time Data Import:

+ Seedlink: Real-Time Data Export

+ Buffer of Uniform Data: BUD

+ How IRIS DS Serves “SeedLink” Data

+ Connection and Expectation (We do not regard this as High Availability)

+ Stations: Total vs RealTime

+ Seedlink Shipments are Significant

+ The IRIS MUSTANG Quality Assurance System What is MUSTANG? Modular Utility for STAtistical kNowledge Gathering Suite of web services that calculate, store and retrieve data quality measurements Modular – new metrics can easily be contributed and implemented Able to house contributed measurements from other institutions Queries for MUSTANG measurements are in the form of a URL

+ A few terms Metric – an algorithm that calculates some value related to data quality. Target – a data channel described by its SEED network, station, location and channel IDs, plus its SEED quality code (e.g. IU.ANMO.00.BHZ.Q). Measurement – a value calculated by a data quality metric for a single target and time window.

+ The MUSTANG system

+ What does MUSTANG measure? Current metrics

+ What does MUSTANG measure? Current network coverage (mainly BH channels) Coming next: G, GE, MN, NL & NZ

+ Where do you find MUSTANG? Main MUSTANG web page Tutorial: Getting Started With MUSTANG

+ How do you access MUSTANG measurements? Existing clients Builder LASSO Databrowser Write a client customized to your needs Any language that can use HyperText Transfer Protocol (HTTP) to send commands and receive data can be used to write a client.

+ IRIS DMC: Research Ready Data Sets MUSTANG Metric Estimators Gaps, overlaps, completeness, signal to noise, power density, pdf mode changes, Glitches, (~24 metrics in phase 2) MUSTANG Metric Estimators Gaps, overlaps, completeness, signal to noise, power density, pdf mode changes, Glitches, (~24 metrics in phase 2) PostgreSQL Database Data Quality Technician Domestic & Non-US Network Operators Domestic & Non-US Network Operators Researcher Specifies Required Data Metric Constraints DMC Filters Data Request Using Defined Constraints Filtered Data Request Returned to Researcher Archived and Real Time Data Research Ready Data Sets

+ IRIS Data Services Posters Data Access: GC13F-0716: Developing Federated Services within Seismology: IRIS’ involvement in the CoopEUS Project Monday, Dec. 15, 2014, 01:40 PM – 06:00 PM, Moscone West S43A-4536: Federated data access and other services offered by the IRIS DMC Thursday, Dec. 18, 2014, 01:40 PM – 06:00 PM, Moscone South Data Products: S33A-4478: The IRIS DMC’s Earth Model Collaboration (EMC) Wednesday, Dec. 17, 2014, 01:40 PM - 06:00 PM, Moscone South S43A-4506: A highlight of data products from IRIS Data Services Thursday, Dec. 18, 2014, 01:40 PM - 06:00 PM, Moscone South Quality Assurance: S13C-4471: The Future of Seismic Data Quality Assurance at the IRIS DMC Monday, Dec. 15, 2014, 01:40 PM – 06:00 PM, Moscone South S43A-4525: MUSTANG, A Community-Facing Web Service to Improve Seismic Data Quality Awareness Through Metrics Thursday, Dec. 18, 2014, 01:40 PM - 06:00 PM, Moscone South

+ Time for your Questions