Building a KEPLER Extension using Ptolemy II KEPLER Collaboration Staff Presented by: Ilkay Altintas and Efrat San Diego Supercomputer Center, UCSD

2/20/2004Building a KEPLER Extension Using Ptolemy II2 Outline Overview of Ptolemy II and Kepler collaboration Ptolemy II Ptolemy II Architecture Download, Make and Install The Vergil user interface and application Directors and Actors Ports and Parameters Manager Kepler Overview of pre-built Kepler actors and demos Ingredients to build a Kepler extension Building you first actor HelloWorld revisited…

2/20/2004Building a KEPLER Extension Using Ptolemy II3 The KEPLER System for Scientific Workflows … A framework for design, execution and deployment of scientific workflows Caters specifically to the domain scientist Builds on Ptolemy II (next slide... :-) Application pull from various projects

2/20/2004Building a KEPLER Extension Using Ptolemy II4 … based on Ptolemy II A set of Java packages for heterogeneous, concurrent modeling, design and execution. Strengths include: Precisely defined models of computation and component interaction e.g. Process Networks (PN) – data-flow oriented An intuitive GUI that lets rapid workflow composition A modular, reusable and extendable object-oriented environment An XML based workflow definition – MoML Workflows defined in Ptolemy II MoML XML schema Easily exchangeable

2/20/2004Building a KEPLER Extension Using Ptolemy II5 Actor-Oriented Design Reference: PtolemyII Design Document-1, Chapter 1

2/20/2004Building a KEPLER Extension Using Ptolemy II6 Focus on Actor-Oriented Design Adapted from the *.ppt slides by Edward A. Lee (See References) Object orientation: class name data methods call return What flows through an object is sequential control Actor orientation: actor name data (state) ports Input data parameters Output data What flows through an object is streams of data

2/20/2004Building a KEPLER Extension Using Ptolemy II7 Layered Software Architecture Ptolemy II packages have carefully constructed dependencies and interfaces PN CSP CT DE FSM SDF Kernel Data ActorMath Graph Adapted from the *.ppt slides by Edward A. Lee (See References)

2/20/2004Building a KEPLER Extension Using Ptolemy II8 Ptolemy II Architecture Core packages (actor, data, kernel, math, util) The data model(--abstract syntax) of the models Abstract semantics User Interface (UI) packages MoML and visual interface support Library packages Domain polymorphic actors Domain packages Implementations of different models of computation

2/20/2004Building a KEPLER Extension Using Ptolemy II9 Overview of the Key Classes

2/20/2004Building a KEPLER Extension Using Ptolemy II10 Installing Ptolemy II User version Version Ptolemy II – Web Start Installer Source download Precompiled (…downloaded before…) Source-only code – 4 steps to make cd “PTII” rm –f config.*./configure make fast install

2/20/2004Building a KEPLER Extension Using Ptolemy II11 Abstract Syntax of PTII Models Abstract syntax choices: Hierarchy is tree structured (like XML). A relation mediates connections. Ports can link multiple relations and relations can link multiple ports. Ports mediate connections across levels of the hierarchy (no statecharts-style level-crossing links) … Abstract syntax defines the structure of a model, but says little about what it means. Hierarchical Entities, Ports, Connections and Attributes Adapted from the *.ppt slides by Edward A. Lee (See References)

2/20/2004Building a KEPLER Extension Using Ptolemy II12 Models of Computation Semantic interpretations of the abstract syntax Different models Different semantics Different execution Are actors active? passive? reactive? Are communications timed? synchronized? buffered? Models of Computation: continuous-time dataflow rendezvous discrete events synchronous time-driven publish/subscribe … process { … read(); … } process { … write(); … } channel port receiver One Class of Semantic Models: Producer / Consumer Adapted from the *.ppt slides by Edward A. Lee (See References)

2/20/2004Building a KEPLER Extension Using Ptolemy II13 Director Governs the execution of a composite entity. Scheduling, dispatching threads, generate code, etc. A composite entity is called opaque if it doesn’t have a local director. An opaque composite entity inherits the director of its container as its executive director.

2/20/2004Building a KEPLER Extension Using Ptolemy II14 Vergil User Interface The graph editor Actor library Director library Utilities Ports Parameters Quick tour

2/20/2004Building a KEPLER Extension Using Ptolemy II15 Using Vergil Executing a pre-built model from the graph editor Building a model Choosing actors Making connections Choosing a director Building composite actors Saving and running the model Start Exercise-0

2/20/2004Building a KEPLER Extension Using Ptolemy II16 MoML Modeling Markup Language A primary persistent XML file format for Ptolemy II. ptolemy filename.xml ptexecute filename.xml vergil filename.xml moml configuration.xml filename.xml

2/20/2004Building a KEPLER Extension Using Ptolemy II17 An Example MoML File <!DOCTYPE entity PUBLIC "-//UC Berkeley//DTD MoML 1//EN" " <property name="director" class="ptolemy.domains.sdf.kernel.SDFDirector"/> SDF domain Top-level entity Reference: PtolemyII Design Document-1, Chapter 6

2/20/2004Building a KEPLER Extension Using Ptolemy II18 Type System

2/20/2004Building a KEPLER Extension Using Ptolemy II19 Building Kepler Extensions FOCUS: How to build actors? Exercises are under: Please complete the preparation for the programming exercises… 10 minute break…

2/20/2004Building a KEPLER Extension Using Ptolemy II20 Ingredients We have: Java Cygwin for Developers (for Windows users) Ant 1.5 or higher Downloaded: Compiled Ptolemy II source code Kepler extensions (a version from CVS) Completed the preparation for the hands-on programming exercises. Coding-style guidelines are at: Please use them!

2/20/2004Building a KEPLER Extension Using Ptolemy II21 Adding Actors Domain and/or data polymorphic actors Use object-oriented inheritance to avoid code duplication 3 base classes: Source, Sink, Transformer To use the actors in Vergil Add them to one of the actor libraries Most libraries are under $PTII/ptolemy/actor/lib Libraries are XML files The basic structure of an actor: See

2/20/2004Building a KEPLER Extension Using Ptolemy II22 Actor Interfaces: Ports & Parameters input ports output port p1p1 p2p2 p3p3 parameters: a 1 = value a 2 = value input/output port Example:

2/20/2004Building a KEPLER Extension Using Ptolemy II23 Anatomy of an Actor: Ports Used for “message transport”, can be an input, an output, or both. IOPort Key class: IOPort (Can be connected to other IOPort instances via IORelations.) TypedIOPort DEIOPort Use TypedIOPort in order to benefit from the type system! (Domain specific: DEIOPort) Receiver and Sender interfaces depending on the usage of the port. Public members of the actors!!!

2/20/2004Building a KEPLER Extension Using Ptolemy II24 Ports: Introduction to the API public TypedIOPort portName; //Definition //Create the port portName = new TypedIOPort(this, “portName”, true, false); portName.setMultiport(true); //Can support more than one link int width = portName.getWidth(); //0 or 1 if single port //Reading and Writing portName.send(channelNumber, token); Token token = portName.get(channelnumber); //Setting the type of the port portName.setTypeEquals(BaseType…a type in the type system…); portName.setTypeAtLeast(…must be a port or parameter…);

2/20/2004Building a KEPLER Extension Using Ptolemy II25 Anatomy of an Actor: Parameters Public members of the actors! Similar API with ports… public Parameter parameterName; //Definition //Creation and setting the initial value: 2-ways parameterName = new Parameter (this, “parameterName”, new Token(…token value…)); OR parameterName = new Parameter (this, “parameterName”); parameterName.setExpression(…tokenValue…);

2/20/2004Building a KEPLER Extension Using Ptolemy II26 Anatomy of an Actor: Constructors The major task: To create and configure ports and parameters super(container, name); Carries the NameDuplication and IllegalAction exceptions from the super class. The icon for the actor can be set here.

2/20/2004Building a KEPLER Extension Using Ptolemy II27 Exercise-1 Create an actor with An output port A parameter Introduction to the fire method. Adding your actors into the actor library Using your actors in a workflow Break for 10 minutes when finished…

2/20/2004Building a KEPLER Extension Using Ptolemy II28 Execution of an actor initialize() Iterations wrapUp() prefire fire() postfire()

2/20/2004Building a KEPLER Extension Using Ptolemy II29 Action Methods initialize() initialize(): Initialize the state variables of an actor. prefire prefire(): Returns a boolean which indicates if the actor wants to fire. Can also be used to perform an operation that will happen exactly once per iteration. fire fire(): The main point of execution. For reading inputs, producing outputs, read the current parameter values.

2/20/2004Building a KEPLER Extension Using Ptolemy II30 Action Methods (cont.) postfire postfire(): Has two tasks: Updating persistent state Determining whether the execution of an actor is complete. wrapUp wrapUp(): For displaying final results.

2/20/2004Building a KEPLER Extension Using Ptolemy II31 Things to remember when implementing a fire() method Use the methods of the Token class for arithmetic whenever possible (to get data polymorphism) If data-polymorphism is not necessary, set the type to a base type then cast the token to that type. Cannot assume that there is be data available at all the input ports (for domain-polymorphism) Do not update the persistent state in fire() (use postfire())

2/20/2004Building a KEPLER Extension Using Ptolemy II32 Implementing Polymorphism Class: PortParameterFunction A PortParameterFunction object will be returned as a function of two objects. Set the type of the output equal to the type of this object. Type system will compute the type of the PortParameterFunction object and use it as the type of the output when necessary. We’ll see this in the example 2!

2/20/2004Building a KEPLER Extension Using Ptolemy II33 Exercise-2 Getting data through an input port. Understanding the action methods. Designing a polymorphic actor. Using inner classes Break for 10 minutes when finished…

2/20/2004Building a KEPLER Extension Using Ptolemy II34 The manager Controls the overall execution of a model. Interacts only with the “top-level composite actor” startRun() -> run() -> execute() ExecutionListener interface provides the manager with info on the events generated during execution. Hand-outs... show how Ptolemy II handles mixing models of computations hierarchically. --Using opaque composite actors!

2/20/2004Building a KEPLER Extension Using Ptolemy II35 Exceptions A uniform mechanism for reporting errors Base class: KernelException Exception chaining re-implemented since Java versions < 1.4 doesn’t support it. The detail message includes the detail message from the cause argument. A protected _setCause() method is implemented, but not the public initCause() method that JDK1.4 has. Non-severe exceptions: IllegalActionException, NameDuplicationException, NameDuplicationException. Severe-exceptions: KernelRuntimeException, InvalidStateException, InternalErrorException.

2/20/2004Building a KEPLER Extension Using Ptolemy II36 Exercises-3&4 Stopping a model using an actor Utilizing the manager functions Handling exceptions Using the postfire method Creating a customized icon for an actor Using an existing code/application in Kepler. Break for 10 minutes when finished…

2/20/2004Building a KEPLER Extension Using Ptolemy II37 Existing Kepler Actors Actor prototyping tool Generic Web and Grid Service RDBMS Connection and Querying Generic User Interface and Transformation Biological Service and Data Access Rock Classification EML Data Ingestion GARP Native Species Pipeline (Using JNI to utilize C++ code)

2/20/2004Building a KEPLER Extension Using Ptolemy II38 Actor Prototyping Tool Allows “place-holder” actors to be defined on the fly by non-programmers during workflow creation Scientists can thereby create workflows without programming knowledge Workflows created with these actors can be executed once their functionality is implemented by a programmer

2/20/2004Building a KEPLER Extension Using Ptolemy II39 Distributed SWFs in KEPLER Web and Grid Service plug-ins WSDL, GWSDL ProxyInit, GlobusGridJob, GridFTP, DataAccessWizard WS Harvester Imports all the operations of a specific WS (or of all the WSs in a UDDI repository) as Kepler actors WS-deployment interface (…ongoing work…) XSLT and XQuery transformers to link non-fitting services together

2/20/2004Building a KEPLER Extension Using Ptolemy II40 A Generic Web Service Actor Given a WSDL and the name of an operation of a web service, dynamically customizes itself to implement and execute that method. Configure - select service operation

2/20/2004Building a KEPLER Extension Using Ptolemy II41 Set Parameters and Commit Set parameters and commit

2/20/2004Building a KEPLER Extension Using Ptolemy II42 WS Actor after Instantiation

2/20/2004Building a KEPLER Extension Using Ptolemy II43 Web Service Harvester Imports the web services in a repository into the actor library. Has the capability to search for web services based on a keyword.

2/20/2004Building a KEPLER Extension Using Ptolemy II44 Composing 3 rd -Party WSs Output of previous web service User interaction & Transformations Input of next web service

2/20/2004Building a KEPLER Extension Using Ptolemy II45 Current Grid Actors

2/20/2004Building a KEPLER Extension Using Ptolemy II46 Providing RDBMS Access Database connection. Opening a database connection and passing it to all actors accessing this database. Database Querying. A generic actor that queries a database and provides its result. DBConnection type and DBConnectionToken. A new IOPort type and a token to distinguish a database connection from any general type.

2/20/2004Building a KEPLER Extension Using Ptolemy II47 Database Connection OpenDBConnection actor: Input: database connection information Output: A DBConnectionToken, a reference to a database connection instance, through a DBConnection output port.

2/20/2004Building a KEPLER Extension Using Ptolemy II48 Querying a Database DatabaseQuery actor: Input: A query string (SQL) and a database connection reference. Parameters: output type – XML, Record or String. Process: Execute query. Produce results according to parameters.

2/20/2004Building a KEPLER Extension Using Ptolemy II49 Using Bio-Services Actors: PIW

2/20/2004Building a KEPLER Extension Using Ptolemy II50 Sub-workflow: Gene Sequence Processing Data transformations between the tasks

2/20/2004Building a KEPLER Extension Using Ptolemy II51 Sub-workflow: Transfac Detail

2/20/2004Building a KEPLER Extension Using Ptolemy II52 Classifying Igneous Rocks

2/20/2004Building a KEPLER Extension Using Ptolemy II53 Actors for Modeling The Classifier Diagrams information and transitions between them. Extracted from the mineral composition and this level’s diagram coordinates. SVG to polygons. Classifier: Locates the point’s region. Finer granularity Displays the point in the diagram for this level.

2/20/2004Building a KEPLER Extension Using Ptolemy II54 EML Data Ingestion Actor Ingests any data format described by EML metadata Converts raw data to Ptolemy format Data can then be operated on with other actors

2/20/2004Building a KEPLER Extension Using Ptolemy II55 Using EML ingestion actor

2/20/2004Building a KEPLER Extension Using Ptolemy II56 Third-Party Local Applications

2/20/2004Building a KEPLER Extension Using Ptolemy II57 What did we learn? Basic concepts of Ptolemy II Creating actors within the Ptolemy II framework The existing Kepler extensions What we didn’t learn: Extending the type system Need to define a class for the new type and add it to the type lattice Extending the user interface Effigy, Tableaux (Similar to Model-View-Controller architecture) Creating our own director Developing a director is harder! Need to inherit the Director class, and implement the semantics of how the actors will behave. (Semantics==Model of computation)

2/20/2004Building a KEPLER Extension Using Ptolemy II58 Ptolemy II Design Document Volume 1: User-Oriented Volume 2: Developer-Oriented Volume 3: Researcher-Oriented Adapted from the *.ppt slides by Edward A. Lee (See References)

2/20/2004Building a KEPLER Extension Using Ptolemy II59 Becoming a Kepler member? me: to get a read-only account. Will upgrade to become a contributing member depending in time.

2/20/2004Building a KEPLER Extension Using Ptolemy II60 References Ptolemy II Design Documentation and Source Code Kepler website: Overview of Ptolemy II (Powerpoint presentation), Edward A. Lee, October 9, 2003, presented in course EE249 - Design of Embedded Systems: Models, Validation and Synthesis. An Overview of the Ptolemy Project a nd Actor-Oriented Design (powerpoint presentation), Edward A. Lee, February 4, 2004, OMG Technical Meeting, Anaheim, CA. Presentations from the Ptolemy website: Kepler documentation, javadocs and this tutorial will be made available at the website soon.