Kepler, Opal and Gemstone Amarnath Gupta University of California San Diego

2 ISSGC06 – Ischia, Italy Changing Needs for Scientific Process Observe  Hypothesize  Conduct experiment  Analyze data  Compare results and Conclude   Predict Traditional Scientific Process (before computers) Yesterday…at least for some of us!

3 ISSGC06 – Ischia, Italy What’s different in today’s science? Today’s scientific process Observe  Hypothesize  Conduct experiment  Analyze data  Compare results and Conclude   Predict More to add to this picture: network, Grid, portals, +++ Observing / Data: Microscopes, telescopes, particle accelerators, X-rays, MRI’s, microarrays, satellite-based sensors, sensor networks, field studies… Analysis, Prediction / Models and model execution: Potentially large computation and visualization

4 ISSGC06 – Ischia, Italy A Brief Recap What are Scientific Workflow Systems trying to achieve? –Creation of a problem solving environment over distributed and mostly autonomous platforms –Seamless access to resources and services –Service composition and reuse –Scalability –Detached execution and yet, user interaction –Reliability and fault tolerance –“Smart” re-runnability –Reproducibility of execution –Information discovery as an aid to workflow design

5 ISSGC06 – Ischia, Italy What is Kepler? Derived from an earlier scientific data-flow system called Ptolemy-II, which is –Designed to model heterogeneous, concurrent systems for engineering applications –An actor-based workflow paradigm Kepler adds to Ptolemy-II –New components for scientific workflows –Structural and Semantic type management –Semantic annotation and annotation propagation mechanisms –Distributed execution capabilities Execution in a grid framework –…

6 ISSGC06 – Ischia, Italy Promoter Identification Workflow Source: Matt Coleman (LLNL)

Promoter Identification Workflow

8 ISSGC06 – Ischia, Italy

9

10 ISSGC06 – Ischia, Italy Enter initial inputs, Run and Display results

11 ISSGC06 – Ischia, Italy Custom Output Visualizer

12 ISSGC06 – Ischia, Italy Kepler System Architecture Authentication GUI Vergil SMS Kepler Core Extensions Ptolemy …Kepler GUI Extensions… Actor&Data SEARCH Type System Ext Provenance Framework Kepler Object Manager Documentation Smart Re-run / Failure Recovery

13 ISSGC06 – Ischia, Italy What is an Actor-based Workflow? An actor-based workflow is a graph with three components –Actors: passive (parameterized) programs are specified by their input and output signatures Ports: an actor has a set of input and output ports that are specified by the signature of the data tokens passing through that port No call semantics Attributes –Dataflow connections: a connectivity specification that designates the flow of data from one actor to another Relation: an intermediate data holding station –Director: an execution control model that coordinates the execution behavior of a workflow

14 ISSGC06 – Ischia, Italy Composite Actors Composite actor AW –A pair (W,Σ W ) comprising a subworkflow W and a set of distinguished ports Σ W  freeports(W), the i/o- signature of W –The i/o-signatures of the subworkflow W and of the composite actor AW containing W match, i.e., Σ W = ports(AW) –An actor can be “refined” by treating it as a workflow and adding other restrictions around it Workflow abstraction –One can substitute a subworkflow as a single actor –The subworkflow may have a different director than the higher-level workflow

15 ISSGC06 – Ischia, Italy Mineral Classification Workflow

16 ISSGC06 – Ischia, Italy PointInPolygonalgorithm

17 ISSGC06 – Ischia, Italy Execution Model Actors –Asynchronous: Many actors can be ready to fire simultaneously Execution ("firing") of a node starts when (matching) data is available at a node's input ports. Locally controlled events – Events correspond to the “firing” of an actor Actor: – A single instruction – A sequence of instructions Actors fire when all the inputs are available Directors are the WF Engines that –Implement different computational models –Define the semantics of execution of actors and workflows interactions between actors Process Network (PN) Director –Each actor executes as a separate thread or process –Data connections represent queues of unbounded size. Actors can always write to output ports, but may get suspended (blocked) on input ports without a sufficient number of data tokens. –Performs buffer management, deadlock detection, allows data forks and merges

18 ISSGC06 – Ischia, Italy The Director Execution Phases –pre-initialize method of all actors Run once per workflow execution Are the data types of all actor ports known? Are transport protocols known? –type-check Are connected actors type compatible? –run* initialize –Executed per run –Are all the external services (e.g., web services) working? –Replace dead services with live ones… iteration* –pre-fire »Are all data in place? –fire* –post-fire »Any updates for local state management? –wrap-up

19 ISSGC06 – Ischia, Italy Polymorphic Actors: Components Working Across Data Types and Domains Actor Data Polymorphism : –Add numbers (int, float, double, Complex) –Add strings (concatenation) –Add complex types (arrays, records, matrices) –Add user-defined types Actor Behavioral Polymorphism: –In dataflow, add when all connected inputs have data –In a time-triggered model, add when the clock ticks –In discrete-event, add when any connected input has data, and add in zero time –In process networks, execute an infinite loop in a thread that blocks when reading empty inputs –In CSP, execute an infinite loop that performs rendezvous on input or output –In push/pull, ports are push or pull (declared or inferred) and behave accordingly –In real-time CORBA*, priorities are associated with ports and a dispatcher determines when to add By not choosing among these when defining the component, we get a huge increment in component re- usability. But how do we ensure that the component will work in all these circumstances? Source: Edward Lee et al.

20 ISSGC06 – Ischia, Italy GEON: Geosciences Network Multi-institution collaboration between IT and Earth Science researchers Funded by NSF “large” ITR program GEON Cyberinfrastructure provides: –Authenticated access to data and Web services –Registration of data sets and tools, with metadata –Search for data, tools, and services, using ontologies –Scientific workflow environment –Data and map integration capability –Visualization and GIS mapping

21 ISSGC06 – Ischia, Italy R. Haugerud, U.S.G.S D. Harding, NASA Point Cloud x, y, z n, … Survey Process & Classify Analyze / “Do Science” Interpolate / Grid LiDAR Introduction

22 ISSGC06 – Ischia, Italy LiDAR Difficulties Massive volumes of data –1000s of ASCII files –Hard to subset –Hard to distribute and interpolate Analysis requires high performance computing Traditionally: Popularity > Resources

23 ISSGC06 – Ischia, Italy A Three-Tier Architecture GOAL: Efficient LiDAR interpolation and analysis using GEON infrastructure and tools –GEON Portal –Kepler Scientific Workflow System –GEON Grid Use scientific workflows to glue/combine different tools and the infrastructure Portal Grid

24 ISSGC06 – Ischia, Italy Lidar Workflow Process Configuration phase Subset: DB2 query on DataStar Portal Grid Subset Analyze moveprocess Visualize moverenderdisplay Interpolate: Grass RST, Grass IDW, GMT… Visualize: Global Mapper, FlederMaus, ArcIMS Scheduling/ Output Processing Monitoring/ Translation

25 ISSGC06 – Ischia, Italy Lidar Processing Workflow (using Fledermaus ) Subset Analyze moveprocess Visualize moverenderdisplay Arizona Cluster NFS Mounted Disk IBM DB2 Datastar NFS Mounted Disk d1 d2 (grid file) d2 d1 iView3D/ Browser Create Scene file Fledermaus sd

26 ISSGC06 – Ischia, Italy Lidar Workflow Portlet 1.User selections from GUI 1.Translated into a query and a parameter file 2. Uploaded to remote machine 2.Workflow description created on the fly 3.Workflow response redirected back to portlet

27 ISSGC06 – Ischia, Italy Render Map DB2 Spatial query Client/ GEON Portal NFS Mounted Disk ArcInfo Compute Cluster x,y,z and attribute raw data process output KEPLER WORKFLOW Map Parameters Grass Functions submit Parameter xml Create Workflow description ArcSDE ArcIMS Map onto the grid (Pegasus) Grass surfacing algorithms: Spline IDW block mean … Download data Binary grid ASCII grid Text file Tiff/Jpeg/Gif ASCII grid LIDAR POST-PROCESSING WORKFLOW PORTLET

28 ISSGC06 – Ischia, Italy Portlet User Interface - Main Page

29 ISSGC06 – Ischia, Italy Portlet User Interface - Parameter Entry 1

30 ISSGC06 – Ischia, Italy Portlet User Interface - Parameter Entry 2

31 ISSGC06 – Ischia, Italy Portlet User Interface - Parameter Entry 3

32 ISSGC06 – Ischia, Italy Behind the Scenes: Workflow Template

33 ISSGC06 – Ischia, Italy Filled Template

34 ISSGC06 – Ischia, Italy Example Outputs

35 ISSGC06 – Ischia, Italy With Additional Algorithms

36 ISSGC06 – Ischia, Italy Kepler System Architecture Authentication GUI Vergil SMS Kepler Core Extensions Ptolemy …Kepler GUI Extensions… Actor&Data SEARCH Type System Ext Provenance Framework Kepler Object Manager Documentation Smart Re-run / Failure Recovery

37 ISSGC06 – Ischia, Italy The Hybrid Type System Every portal of an actor has a type signature –Structural Types Any type system admitted by the actor –DBMS data types, XML schema, Hindley-Milner type system … –Semantic Types An expression in a logical language to specify what a data object means In the SEEK project, such a statement is expressed in a DL over an ontology –MEASUREMENT  ITEM_MEASURED.SPECIES_OCCURRENCE A workflow is well-typed if –For every pair of connected ports –The structural type of the output port is a subtype of that of the input port –The semantic type of the output port is logically subsumed by that of the input port

38 ISSGC06 – Ischia, Italy Hybridization Constraints A hybridization constraint –a logical expression connecting instances of a structural type with instances of the corresponding semantic type for a port –For a relational type r(site, day, spp, occ) I/O Constraint –A constraint relating the input and output port signatures of an actor Propagating hybridization constraints Having a tuple in r implies that there is a measurement y of the type speciesoccurrence corresponding to x occ

39 ISSGC06 – Ischia, Italy

40 ISSGC06 – Ischia, Italy How can my (grid) application become a Kepler actor? By making it a web service –For applications that have a command line interface –OPAL can convert the application into a web service What is Opal? –a Web services wrapper toolkit Pros: Generic, rapid deployment of new services Cons: Less flexible implementation, weak data typing due to use of generic XML schemas

41 ISSGC06 – Ischia, Italy Condor poolSGE Cluster PBS Cluster Globus Application Services Security Services (GAMA) State Mgmt GemstonePMV/VisionKepler Opal is an Application Wrapping Service

42 ISSGC06 – Ischia, Italy The Opal Toolkit: Overview Enables rapid deployment of scientific applications as Web services (< 2 hours) Steps –Application writers create configuration file(s) for a scientific application –Deploy the application as a Web service using Opal’s simple deployment mechanism (via Apache Ant) –Users can now access this application as a Web service via a unique URL

43 ISSGC06 – Ischia, Italy Opal Architecture Tomcat Container Axis Engine Opal WS Cluster/Grid Resources Container Properties Service Config Scheduler, Security, Database Setups Binary, Metadata, Arguments

44 ISSGC06 – Ischia, Italy Service Operations Get application metadata: Returns metadata specified inside the application configuration Launch job: Accepts list of arguments and input files (Base64 encoded), launches the job, and returns a jobID Query job status: Returns status of running job using the jobID Get job outputs: Returns the locations of job outputs using the jobID Get output as Base64: Returns an output file in Base64 encoded form Destroy job: Uses the jobID to destroy a running job

45 ISSGC06 – Ischia, Italy MEME+MAST Workflow using Kepler

46 ISSGC06 – Ischia, Italy Kepler Opal Web Services Actor

47 ISSGC06 – Ischia, Italy Opal and Gemstone

48 ISSGC06 – Ischia, Italy Opal Summary Opal enables rapidly exposing legacy applications as Web services –Provides features like Job management, Scheduling, Security, and Persistence More information, downloads, documentation: –

49 ISSGC06 – Ischia, Italy Kepler System Architecture Authentication GUI Vergil SMS Kepler Core Extensions Ptolemy …Kepler GUI Extensions… Actor&Data SEARCH Type System Ext Provenance Framework Kepler Object Manager Documentation Smart Re-run / Failure Recovery

50 ISSGC06 – Ischia, Italy Joint Authentication Framework Requirements: –Coordinating between the different security architectures GEON uses GAMA which requires a single certificate authority. SEEK uses LDAP with has a centralized certificate authority with distributed subordinate CAS –To connect LDAP with GAMA – Coordinating between 2 different GAMA servers –Single sign-on/authentication at the initialize step of the run for multiple actors that are using authentication This has issues related to single GAMA repository vs multiple, and requires users to have accounts on all servers. Kepler needs to be able to handle expired certificates for long- running workflows and/or for users who use it for a long time. A trust relation between the different GAMA servers must be established in order to allow for single authentication.

51 ISSGC06 – Ischia, Italy Functional Prototype Completed APIs and tests cases in place More work required on certificate renewal and multiple server access

52 ISSGC06 – Ischia, Italy Vergil is the GUI for Kepler Actor ontology and semantic search for actors Search -> Drag and drop -> Link via ports Metadata-based search for datasets Actor Search Data Search

53 ISSGC06 – Ischia, Italy Back to Kepler - Actor Search Kepler Actor Ontology Used in searching actors and creating conceptual views (= folders) Currently 160 Kepler actors added!

54 ISSGC06 – Ischia, Italy Data Search and Usage of Results Kepler DataGrid – Discovery of data resources through local and remote services SRB, Grid and Web Services, Db connections – Registry of datasets on the fly using workflows

55 ISSGC06 – Ischia, Italy Vergil Updates To make it more useful to the user –Updated actor icons –Menu redesign Improve readability Develop cohesive visual language Follow standard HCI principles Improve organization Composite DB Query Computation or Operation Transformation Filter File Operation Web Service

56 ISSGC06 – Ischia, Italy Kepler Archives Purpose: Encapsulate WF data and actors in an archive file –… inlined or by reference –… version control More robust workflow exchange Easy management of semantic annotations Plug-in architecture (Drop in and use) Easy documentation updates A jar-like archive file (.kar) including a manifest All entities have unique ids (LSID) Custom object manager and class loader UI and API to create, define, search and load.kar files

57 ISSGC06 – Ischia, Italy KAR File Example

58 ISSGC06 – Ischia, Italy Kepler Object Manager Designed to access local and distributed objects Objects: data, metadata, annotations, actor classes, supporting libraries, native libraries, etc. archived in kar files Advantages: –Reduce the size of Kepler distribution Only ship the core set of generic actors and domains –Easy exchange of full or partial workflows for collaborations –Publish full workflows with their bound data Becomes a provenance system for derived data objects => Separate workflow repository and distributions easily

59 ISSGC06 – Ischia, Italy Initial Work on Provenance Framework Provenance –Track origin and derivation information about scientific workflows, their runs and derived information (datasets, metadata…) Need for Provenance –Association of process and results –reproduce results –“explain & debug” results (via lineage tracing, parameter settings, …) –optimize: “Smart Re-Runs” Types of Provenance Information: –Data provenance Intermediate and end results including files and db references –Process (=workflow instance) provenance Keep the wf definition with data and parameters used in the run –Error and execution logs –Workflow design provenance (quite different) WF design is a (little supported) process (art, magic, …) for free via cvs: edit history need more “structure” (e.g. templates) for individual & collaborative workflow design

60 ISSGC06 – Ischia, Italy Kepler Provenance Recording Utility Parametric and customizable –Different report formats –Variable levels of detail Verbose-all, verbose-some, medium, on error –Multiple cache destinations Saves information on –User name, Date, Run, etc…

61 ISSGC06 – Ischia, Italy Provenance: Possible Next Steps Kepler Provenance –Deciding on terms and definitions –.kar file generation, registration and search for provenance information –Possible data/metadata formats –Automatic report generation from accumulated data –A GUI to keep track of the changes –Adding provenance repositories –A relational schema for the provenance info in addition to the existing XML

62 ISSGC06 – Ischia, Italy What other system functions does provenance relate to? Failure recovery Smart re-runs Semantic extensions Kepler Data Grid Reporting and Documentation Authentication Data registration Re-run only the updated/failed parts Guided documentation generation an updates

63 ISSGC06 – Ischia, Italy Where Kepler Meets the Grid Abstract Grid workflow actors –Stage-execute-fetch (sub-)workflows Copy files from one resource to computation node Perform execution – possibly through a grid job scheduler Get the result files back and continue with the rest of the workflow –Actors Authenticate actor –over Globus Grid, SRB and databases Copy actor –for both stage and fetch Job executor actor –special wrappers for ssh-based execution, web service- clients, Grid job runner proxies, and actors for Nimrod- and APST-based submissions

64 ISSGC06 – Ischia, Italy Where Kepler Meets the Grid Monitoring actor –Light monitoring: user notified on actor failure (e.g. NIMROD) upon completion of actor failure –Medium monitoring: same with immediate notification –Heavy monitoring: notifies every communication including immediate actor failure Filter actor –Filtering and subsetting remote data of different formats Data Discovery actor Service Discovery actor Storage actor Transformation and Query actors –Shim generation –Querying of databases and mediators

65 ISSGC06 – Ischia, Italy Hot Topics in Kepler

66 ISSGC06 – Ischia, Italy To Sum Up … is an open-source system and collaboration – is a ~3 year-old project –grows by application pull from contributors –most topics are designed jointly –is developed by multiple developers under different projects in different countries –Is now being used in actual scientific research The screen shots were results of initial success! There is a lot more to cover and work on… –New foci at SDSC-Kepler around provenance and distributed computing

67 ISSGC06 – Ischia, Italy Amarnath Gupta +1 (858) Questions… Thanks!