1. ES 3 Takes the Provenance Challenge J AMES F REW Donald Bren School of Environmental Science and Management University of California, Santa Barbara frew@bren.ucsb.edu

2. Provenance Challenge 2 Frew / Provenance Challenge Workshop / DC / 2006-09-13 SCA fraction, Sierra Nevada, 07 Mar 2004

3. Provenance Challenge 3 Frew / Provenance Challenge Workshop / DC / 2006-09-13 Model structure: MODIS SCA / albedo

4. Provenance Challenge 4 Frew / Provenance Challenge Workshop / DC / 2006-09-13 From Wrappers to Probulators Wrappers: Active Lineage  +  Complete control over what gets recorded  Single language/API for all wrapped events  Not tied to execution –You can even lie about what happened  -  Must explicitly script everything  Scripts can drift from reality –You can even lie about what happened

5. Provenance Challenge 5 Frew / Provenance Challenge Workshop / DC / 2006-09-13 From Wrappers to Probulators Probulators: Passive Lineage  +  Record what actually happened –Not just what you think happened –Not what didn’t happen  Automatic: don’t have to write new scripts for everything  -  Different flavors for different environments –Can’t just do everything in {…insert favorite language here…}

6. Provenance Challenge 6 Frew / Provenance Challenge Workshop / DC / 2006-09-13 Probulator flavors  Instrumentation  Insert lineage capture instructions directly into science codes –e.g. “I just created file ‘foo’”  Typical implementation: preprocessor/precompiler  Overriding  Replace standard routines/libraries with lineage-capturing versions –e.g. open(…) → snoopy_open(…)  Typical implementation: modify execution environment –environment variables –configuration files  Passive monitoring  Trace program execution –e.g. “called open() with args = foo, bar, …”  Typical implementation: strace’d shell

7. Provenance Challenge 7 Frew / Provenance Challenge Workshop / DC / 2006-09-13 ES 3 Lineage Architecture

8. Provenance Challenge 8 Frew / Provenance Challenge Workshop / DC / 2006-09-13 Probulating IDL: Instrumenting the code ;edit pro modscag_cleanse,prefix=prefix,ns=ns,nl=nl HELP, NAMES="*", OUTPUT=ES3_ENVIROMENT & ES3_LOG, $ ENTER="modscag_cleanse", ENVIROMENT=ES3_ENVIROMENT ;clean up {under,over}flow of MODSCAG run ; ;Input:prefix = prefix for all of the MODSCAG output filenames ;ns = number of samples ;nl = number of lines ;Output:rewrite of the MODSCAG files ; ;t.h.painter / 1.19.2005 ; open snow file ES3_openr,1,string(prefix,'snow.pic') snow=fltarr(ns,nl) readu,1,snow [ blah blah blah ] HELP, NAMES="*", OUTPUT=ES3_ENVIROMENT & ES3_LOG, LEAVE="modscag_cleanse", $ ENVIROMENT=ES3_ENVIROMENT END ; modscag_cleanse

9. Provenance Challenge 9 Frew / Provenance Challenge Workshop / DC / 2006-09-13 Probulating IDL: Results <init time="20050522T234606Z” pid="31002" stime="20050522T234604Z" pstime="20050522T234256Z" ppid="30920" language="idl" user="haavar" hostname="spitting-duck.bren.ucsb.edu"> <variable name="!PATH" value="/home/haavar/probulator//idl: /home/rsi/idl_6.1/lib/hook: […] /home/rsi […] /home/haavar/painter/data/tillsnow.pic ]

10. Provenance Challenge 10 Frew / Provenance Challenge Workshop / DC / 2006-09-13 Probulating a shell script: Example  http://twiki.ipaw.info/bin/view/Challenge/ES3#Example http://twiki.ipaw.info/bin/view/Challenge/ES3#Example