© Donald F. Ferguson, All rights reserved. Topics in Computer Science: Modern Internet Service Oriented Application Development Lecture 6: Composite Applications Continued Dr. Donald F. Ferguson (Admin:

2 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration Contents

3 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration Contents Introduction –Tentative course agenda. –Assignment #3 concepts and objectives. Continuing with technology –A note on motivation. –Structural composition. –Clarifying connections, connectors, operations. –Activiti – more details. –Boomi – additional concepts. Some hypothetical implementation scenarios –Master CRM – store CRM synchronization. –Bulk store load. –Approve account open. –Approve new employee. –Update inventory/movie information.

4 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration Introduction

5 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration Tentative Course Agenda DateTopics/Comments 03-AprContinuation of composite applications Continuation of data/message integration and workflow/automation Discussion/ideas for project 3 10-AprI am out-of-town You can either have a project week or I will give a web lecture on Docker/containers, SaaS/milti-tenancy 17-AprRules engines Addition of rules engine to project 3 24-AprSome new databases Project 4 – adding database to projects MayEnd-to-end security 01-May to 15-May Complete projects Complete reviews Completeness and design of all projects determines final grade. The TAs and I will enumerate 10 or more criteria for each project. Grade will be on design document, demo and code review. You can resubmit to address feedback up to 15-May.

6 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration Assignment 3 Concepts Each person must implement a Boomi process and an Activiti process. –You can choose the scenario. I will provide examples. –Integration processes, by definition, “connect” to things. This projects focus is on integration. Having scaffold code/data in the place of real apps for integration is fine. Boomi process should contain –3 connections of at least two types. –3 maps. –1 use of document cache. –Be REST/WS-* callable. –1 simple Data Process. –1 each of branch and route logic nodes. The Activiti process should contain –Two user tasks with forms. –Two script tasks that manipulate process variables –Two service tasks implemented in Java that call a REST API, Web service or DB. –Two gateways –Throwing and catching an exception

7 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration Continuing with Technology

8 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration A Note on Motivation

9 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration Composition Diagrams The diagrams are always “nodes” and “arcs.” Structure: node A node B, where x is Requires/depends/calls/synchronizes with/… Control: –Execute A and then execute B, where A and B are tasks. –There is a shared set of “data objects” the tasks manipulate. Data flow: Do A to the message/document/… and then do B to A’s output Control FlowStructureData Flow

10 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration How do you Build the Composite Take off a palette Set properties Put on canvass Set control values Connect flow

11 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration Reminder of Concepts Editor Visual Notation Diagram “Program” “Engine”

12 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration Why Would You do This? Collaboration between developers and business professionals –Documents and specification can be imprecise, resulting in slightly off code. –Business professionals cannot read your code, and can only comment on the implementation, which results in “change after implementation.” –You can show them a picture, collaboratively modify and then “run a test.” Learning –Developers learning BPMN and other concepts often start with the tools –Learn concepts, assemble simple solutions, … –Run simple tests –And eventually drop into directly writing and editing the XML files. Agility – –Brings the HTML tag concept or JSP like programming to application logic. –I do not need to modify Java, compile, build Jar, put in classpath, restart, … –I can just drop the XML file into a directory and we can go. –Also enables just-in-time process creation for an unusual situation –I have the general account open process –Built want to modify for just this specific, unusual situation.

13 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration Structure Composition Diagrams

14 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration Service Component Architecture

15 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration Flexible Implementation – Some of which we will Cover

16 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration Eclipse SOA SCA Tool

17 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration OSGi Blueprint are Similar

18 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration OSGi Blueprints

19 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration TOSCA

20 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration TOSCA Same basic concepts –Conceptually a “visual” way of assembling SOA apps and infrastructure. –Visual notation for precisely specifying the components and composites. –Underlying XML file format for tool interoperability and some form of automation, which would be build/deployment in this cases versus “execution.

21 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration Comments Structural composition and models is a recurring concept, e.g. –OSGi –SCA –J2EE dependency references –Camel –UML Class, Module and Component Diagrams –TOSCA The concept waxes and wanes in popularity, and seems to be waning. –The explosion of micro-services and ad hoc applications. –Structure integrated into behavior composition –More dynamic, query based binding We will not spend more time on the topic.

22 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration Connectors

23 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration An Example – J2EE Connector Architecture

24 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration Connectors There are two categories of connector –Technology, e.g. RDB, JMS, HTTP, … … –How to “invoke” an operation on some technology. –The calling program needs to understand the details of –Infrastructure settings, e.g. security, message headers, … –Format of the data exchange, e.g. JSON, XML, serialized Java, … … –The specific application formats, e.g. JSON representation for Account. –Application, e.g. SAP, Salesforce, Jira, … … –An application connector has an embedded technology connector. –Has additional code that understand app data formats, infrastructure headers, … … For the purposes of our terminology –There are Connectors that can “connect” with something, e.g. JDBC. –There is a Connection that is configured for a specific instance, e.g. JDBC URL –There is a Connector Operation that is reusable across Connections of a given type, e.g. –A SQL statement for CRUD on a database that works for any MySql DB with the right schema. –An HTTP GET with the right application content types, accepts, receives, … …

25 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration Boomi Model Create a connection using one of the prebuilt connectors. Configure connection (type) specific properties, e.g. –SSL or no SSL –URL –Security model Use a previously built (or build a new) a connector operation with properties, e.g. –Application specific headers –HTTP operation –Data format

26 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration Connectors A lot of the value of integration middleware comes from the connectors. –Calling an application or infrastructure can be complex and tedious. –Develop productivity through –Separation of concerns and skills. –“Library/code” reuse. Including connectors in the visual builder –Enables “non-programmers to use connectors. –Simplifies development experience by avoiding –Jumping out of builder –Into code to write something –And then back into builder –And making sure you have the linkage

27 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration Example Boomi has a lot But almost all do, e.g. Mule WebSphere Weblogic etc.

28 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration Activiti

29 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration Overall Concepts Service A Service B P P

30 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration How to Think about BPM There are a set of “commands/verbs” –Call this API –Put X in this database –Compute this value –… … Wrap these with a plug-in adaptor, which in Activiti occurs through –Service task enablement by –Implementing org.activity.engine.delegate.JavaDelegate interface. –Interacts with the with the process/process engine by –Setting process variables; calling other APIs on DelegateExecution –Calling APIs on the process engine. –Assigning to a person/role that 1) Performs a task, 2) Fills out a form that updates process variables. Assemble the “verbs” into a sentence using BPMN Define the commands and verbs either “bottom-up” through inventory analysis or “top-down” from the process models and definitions.

31 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration Activiti Structure Business applications with user interfaces. Callable web service/REST service implementing control and form functions. A “default” application for defining processes, viewing/managing the execution of processes and assigning tasks/forms to people.

32 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration Some BPMN

33 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration Some BPMN

34 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration BPMN Task Types

35 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration BPMN Task Types

36 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration BPMN Task Types

37 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration Activiti Test Drive

38 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration Service Task

39 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration Service Tasks Callable POJOs with specific interface Get process variables Convert into JSON, XML, … In expected data model Call the thing by using the client API tools for REST, WS-*, SQS, … Get the response. Parse and map back into process variables.

40 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration With ESB Core application info integration/synchronization Reusable service APIs Common model Callable from other apps Web pages Mobile applications etc.

41 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration Boomi Continued

42 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration An Example

43 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration Walk Through Shapes and Docs

44 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration Backup

45 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration Activiti/BPMN Backup

46 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration BPMN BPMN 2.0 is an international standard for business process modeling. The BPMN specification describes how the elements of a process diagram have to look like (notation), how these can be combined with each other (meta model / syntax), what a diagram means (semantics) and how diagrams can be transferred from one tool to another (XML interchange format). Process models describe sequences of business activities from start to finish, e.g. Order-to-Cash, Account Open

47 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration BPMN and Activiti Some terms –BPMN 2.0 is a standard for defining/documenting workflows/orchestrations. –Activiti (and many others) are implementations of the standard –Runtime engine that can interpret and execute a BPMN process –A WYSIWYG editor –With prebuilt, configurable shapes palette that extend base BPMN shapes. Huh? BPMN has the concepts of a “Task,” which is one “step” in a process. But in a process model, this is just “prose” like “Order a Pizza” Implementations come with “common” types of tasks, e.g. Make an SQL call on a database. Call a REST API Put a document on a worklist etc. –A composition canvass for assembling shapes into processes. –Debugger, monitoring, reporting, versioning, etc. This is –The same concept as enterprise integration patterns and Boomi –But, –The clipart is different –And the graph mostly represents “control flow,” not data flow.

48 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration Simple BPMN Diagram

49 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration BPMN Symbols

50 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration BPMN Symbols

51 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration BPMN Symbols

52 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration BPMN Symbols

53 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration BPMN Symbols

54 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration BPMN Symbols Lots more go here. There are references, e.g.

55 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration BPMN Symbols

56 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration BPMN Symbols Seriously dude? There are references, e.g.

57 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration BPMN Symbols

58 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration And You can Define Processes

59 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration How do you Implement? Notify a person/role. Have them do something. Have the use an app that calls the engine to tell you they are done. Send an . POST a document etc. Decision table Call Drools etc. Insert into a DB Call a web service Run a Java app. etc. Run some JavaScript or Java or … Right at this point in the workflow/process And manipulate the data/documents/control.

60 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration User Task

61 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration Script Task

62 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration Service Task

63 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration

64 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration

65 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration

66 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration

67 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration

68 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration

69 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration

70 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration

71 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration Comments Three Core Aspects of BPMN and Actviti Human Centric Workflow –Define an organization model –Users, Groups –Properties for users and groups –Define workflow processes that have Human Tasks. For each Human Task –Define the form and data that enables a person to perform the task. –Define “rules” for assigning a task to people or groups. Automation/Orchestration –Implement a set of POJOs implementing the Delegate interface. Each POJO implements an automation action by … –Call a REST service –Sending a message on JMS or SQS –Etc. –Define a workflow with one or more Service Tasks. For each Service Task –Specify/bind to a Delegate. –Map process data variables to inputs and outputs of the Delegate.

72 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration Amazon Simple Workflow

73 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration AWS Simple Workflow

74 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration Amazon Simple Workflow

75 © Donald F. Ferguson, All rights reserved.Modern Internet Service Oriented Application Development – Lecture 7: Service Integration Application Structure Role of the Activities Worker: The activities worker performs the various tasks that the workflow must accomplish. It consists of: –The activities implementation, which includes of a set of activity methods that perform particular tasks for the workflow. –An activity worker, which uses HTTP long poll requests to poll Amazon SWF for activity tasks. When a task is available, Amazon SWF responds to the request by sending the information required to perform the task. The activity worker then calls the appropriate activity method, and returns the results to Amazon SWF. Role of the Workflow Worker: The workflow worker orchestrates the execution of the various activities, manages data flow, and handles failed activities. It consists of: –The workflow implementation, which includes the activity orchestration logic, handles failed activities, and so on. –An activities client, which serves as a proxy for the activities worker and enables the workflow worker to schedule activities to be executed asynchronously. –A workflow worker object, which uses HTTP long poll requests to poll Amazon SWF for decision tasks. If there are tasks on the workflow task list, Amazon SWF responds to the request by returning the information that is required to perform the task. The framework then executes the workflow to perform the task and returns the results to Amazon SWF.