1. Flexible Distributed Business Process Management Vinod Muthusamy University of Toronto Thesis Defense September 23, 2011

2. MIDDLEWARE SYSTEMS RESEARCH GROUP MSRG.ORG Composite applications Mashups Service-oriented architectures Cloud computing

3. MIDDLEWARE SYSTEMS RESEARCH GROUP MSRG.ORG Distributed business processes Vendor Sale Manufactory Finance Dispatch B Out-stock B Pick-up goods Packaging Marketing Design Out-stock B Target price Prototype OutTake Control Assign Confirm Determinate plan Check stock Raw materials Audit Raw Determinate plan Execute plan Process control Monitor Process Pay Check Signature Print receipt Warehouse Delivery FedEx Pick up Monitoring Statistic Chart Strategy Design Marketing Order Manufactory Payment Requirement collection Feature selection Goods selection Confirm features Material Make plan Feedback Check order Fill order Check dealerCheck credit Approval Validate Affirm order Sale prediction Sign Contract CCC administrate Goods delivery Fill dispatch bill Fill out-stock bill Credit card Case Study (Chinese Electronics Manufacturer): Global processes that compose departmental ones Department-level processes with 26 to 47 activities Thousands of concurrent instances Hundreds of collaborating partners Geographically distributed Administrative boundaries 3

4. MIDDLEWARE SYSTEMS RESEARCH GROUP MSRG.ORG Process execution architectures ABCDABCD ABCDABCD A A B B C C D D ABCDABCD ABCDABCD ABCDABCD ABCDABCD Process Centralized Clustered One execution engine May not scale Central point of failure Replicated engines Centralized control and data High bandwidth and latency Still may not scale Administrative limitations 4

5. MIDDLEWARE SYSTEMS RESEARCH GROUP MSRG.ORG Process execution architectures (2) A A B B C C D D Process Distributed D D C C A,B Distributed set of execution engines Flexible deployment Performance, security, etc. Process fragments can be modified Lower bandwidth and latency Fine-grained use of resources No single point of failure In-network processing Our relevant publications: ACM TWEB 2010, CASCON 2009, IEEE ICDCS 2009, ACM DEBS 2009, CASCON 2008, ACM DEBS 2008, CASCON 2008, ACM MobiCom 2005 5

6. MIDDLEWARE SYSTEMS RESEARCH GROUP MSRG.ORG Distributed process execution stack 6 Transactional movement Filter aggregation Distributed execution Activity placement Service discovery Service composition SLA Modelling

7. MIDDLEWARE SYSTEMS RESEARCH GROUP MSRG.ORG Input queue Output queue d2 Output queue d3 subscriptiondest Matching Engine Routing Table & Class = Loan d2 Service RTT > 1s d3 B B B B B Content-based Publish/Subscribe Subscriber Publications Subscriptions Notification Service RTT = 2s class = Loan Service RTT > 1s class = Loan, Service = credit check status = approved, 1. Advertise 2. Subscribe Content-based routing 3. Publish amount = $500K 7 Publisher

8. MIDDLEWARE SYSTEMS RESEARCH GROUP MSRG.ORG Agenda Transactional movement  Low overhead  Without interrupting process Distributed process execution  Process decomposition  Event-based coordination Event-based service discovery  Four discovery models  Exploit similarity 8

9. MIDDLEWARE SYSTEMS RESEARCH GROUP MSRG.ORG A movement operation relocates a client (including its state) Movement may fail because target broker rejects it, etc. During movement, there may be multiple copies of a client, but only one should be “running” at any time Client container helps coordinate the movement B1B1 B8B8 B2B2 B7B7 ∙∙∙ Client Container 1 Client A Client Container 7 Client A MOVE (to Broker 7) Client B 9

10. MIDDLEWARE SYSTEMS RESEARCH GROUP MSRG.ORG ACID-like transactional properties for various layers IsolationConsistencyAtomicity A movement should only modify those routing table entries associated with the moving client. Each broker should have the minimal set of routing table entries for a set of advs and subs. Either all or none of the set of routing table updates required for an operation (adv, sub, etc.) should all be applied. Routing layer The set of notifications delivered to stationary clients from a moving publisher should be independent of whether the publisher successfully moves. A moving client (whether successful or not) should receive the same set of notifications as one that did not move. Notifications are delivered exactly once to either the source or target client. Notifications layer Only the initial or final states of a moving client should be observable. There must be at most one running instance of each client. A moving client must be exclusively either at its source or target broker. Client layer Durability is omitted but can be achieved by persisting all state to stable storage Refer to ICDCS 2009 paper for formal definitions 10

11. MIDDLEWARE SYSTEMS RESEARCH GROUP MSRG.ORG Client and coordinator states at the source and target Coordinators are based on the three-phase commit protocol The failure and timeout transitions are omitted for brevity

12. MIDDLEWARE SYSTEMS RESEARCH GROUP MSRG.ORG The global reachable state graph can be used to prove some of the transactional properties E.g., in the commit state, the source client is clean, and the target client is started Refer to ICDCS 2009 paper for proofs

13. MIDDLEWARE SYSTEMS RESEARCH GROUP MSRG.ORG B2B2 B1B1 BjBj BiBi BlBl A′ A Movement can trigger burst of covered messages C B CB A A 13 P S P P S S S

14. MIDDLEWARE SYSTEMS RESEARCH GROUP MSRG.ORG Evaluations 14

15. MIDDLEWARE SYSTEMS RESEARCH GROUP MSRG.ORG The reconfiguration protocol is much faster than the covering protocol Movement of “root” subscriptions is more expensive in the covering protocol 15

16. MIDDLEWARE SYSTEMS RESEARCH GROUP MSRG.ORG The reconfiguration protocol scales with the number of moving clients The reconfiguration protocol achieves better movement latency despite more total messages because it is less bursty 16

17. MIDDLEWARE SYSTEMS RESEARCH GROUP MSRG.ORG Summary of transactional movement ACID-like transactional properties provide well-defined guarantees for the movement of clients  Properties are modularized to simplify reasoning and implementation Client layer movement and routing layer hop-by-hop reconfiguration protocols were developed Evaluations show proposed protocol is more efficient and stable with respect to various parameters  End-to-end movement using covering negatively affects performance 17

18. MIDDLEWARE SYSTEMS RESEARCH GROUP MSRG.ORG Distributed process execution stack 18

19. MIDDLEWARE SYSTEMS RESEARCH GROUP MSRG.ORG Agenda Transactional movement  Low overhead  Without interrupting process Distributed process execution  Process decomposition  Event-based coordination Event-based service discovery  Four discovery models  Exploit similarity 19

20. MIDDLEWARE SYSTEMS RESEARCH GROUP MSRG.ORG Coordinating process control flow – atomic subscription Process 1 activity1 flow1 activity5 activity3 activity6 activity8 activity2 activity4 activity7 activity2 publication: class = ACTIVITY_STATUS process = ‘Process1’ activityName = ‘activity2’ instanceId = ‘g001’ status = ‘SUCCESS’ activity2 publication: class = ACTIVITY_STATUS process = ‘Process1’ activityName = ‘activity2’ instanceId = ‘g001’ status = ‘SUCCESS’ activity3 subscription: class = ACTIVITY_STATUS process = ‘Process1’ activityName = ‘activity2’ instanceId = * status = ‘SUCCESS’ activity3 subscription: class = ACTIVITY_STATUS process = ‘Process1’ activityName = ‘activity2’ instanceId = * status = ‘SUCCESS’ 20

21. MIDDLEWARE SYSTEMS RESEARCH GROUP MSRG.ORG Coordinating process control flow – composite subscription Process 1 activity1 flow1 activity5 activity3 activity6 activity8 activity2 activity4 activity7 activity6 subscription: class = ACTIVITY_STATUS, process = ‘Process1’, activityName = ‘activity5’, instanceId = $X, status = ‘SUCCESS’ AND class = LINK_STATUS, process = ‘Process1’, activityName = ‘activity2’, instanceId = $X, status = * activity6 subscription: class = ACTIVITY_STATUS, process = ‘Process1’, activityName = ‘activity5’, instanceId = $X, status = ‘SUCCESS’ AND class = LINK_STATUS, process = ‘Process1’, activityName = ‘activity2’, instanceId = $X, status = * 21

22. MIDDLEWARE SYSTEMS RESEARCH GROUP MSRG.ORG BPEL transformation example Sub4: [class,eq,ACTIVITY_STATUS], [process,eq,’Process5’], [activityName,eq,’activity2’], [IID,eq,$X], [status,eq,’SUCCESS’] && [class,eq,LINK_STATUS], [process,eq,’Process5’], [activityName,eq,’activity2’], [IID,eq,$X], [status,isPresent,any] Sub1: [class,eq,ACTIVITY_STATUS], [process,eq,’Process5’], [activityName,eq,’activity1’], [IID,isPresent,any], [status,eq,’SUCCESS’] Pub1: [class, ACTIVITY_STATUS], [process,’Process5’], [activityName,’flow1’], [IID,’g001’], [status,’STARTED’] Sub2: [class,eq,ACTIVITY_STATUS], [process,eq,’Process5’], [activityName,eq,’flow1’], [IID,isPresent,any], [status,eq,’STARTED’] Pub2: [class, ACTIVITY_STATUS], [process,’Process5’], [activityName,’actitiy2’], [IID,’g001’], [status,’SUCCESS’] Pub3: [class,LINK_STATUS], [process,’Process5’], [activityName,’actitiy2’], [IID,’g001’], [status,’POSITIVE’] Pub4: [class, ACTIVITY_STATUS], [process,’Process5’], [activityName,’actitiy6’], [IID,’g001’], [status,’SUCCESS’] Sub5: [class,eq,ACTIVITY_STATUS], [process,eq,’Process5’], [activityName,eq,’activity4’], [IID,isPresent,any], [status,eq,’SUCCESS’] && [class,eq,ACTIVITY_STATUS], [process,eq,’Process5’], [activityName,eq,’activity7’], [IID,isPresent,any], [status,eq,”SUCCESS”] Pub5: [class, ACTIVITY_STATUS], [process,’Process5’], [activityName,’actitiy7’], [IID,’g001’], [status,’SUCCESS’] See ACM Trans. Web 2010 paper for full BPEL mapping Process 1 activity1 flow1 activity5 activity3 activity6 activity8 activity2 activity4 activity7 22

23. MIDDLEWARE SYSTEMS RESEARCH GROUP MSRG.ORG Distributed process execution Receive PADRES ESB 1 34 5 BPEL Process Receive Assign Flow Invoke Wait Reply Wait Reply Invoke 2 Assign WS Gateway Agent 6 END Web Service client Web Service http/soappub/sub

24. MIDDLEWARE SYSTEMS RESEARCH GROUP MSRG.ORG The distributed deployment scales better with request rates 24

25. MIDDLEWARE SYSTEMS RESEARCH GROUP MSRG.ORG Summary of distributed process execution engine Many large processes are inherently distributed  Multiple partners, many administrative domains, geographically dispersed Distributed architecture offers better scalability for large and highly concurrent processes Provides resource allocation flexibility 25

26. MIDDLEWARE SYSTEMS RESEARCH GROUP MSRG.ORG Distributed process execution stack 26

27. MIDDLEWARE SYSTEMS RESEARCH GROUP MSRG.ORG Agenda Transactional movement Low overhead  Without interrupting process Distributed process execution  Process decomposition  Event-based coordination Event-based service discovery  Four discovery models  Exploit similarity 27

28. MIDDLEWARE SYSTEMS RESEARCH GROUP MSRG.ORG 28 Supported models One-time discovery Continuous discovery Static resource Dynamic resource Static (e.g., find weather service) Dynamic (e.g., find micro- generation power) Static continuous (e.g., monitor real estate) Dynamic continuous (e.g., monitor grid resources) Resources Discoveries Event-based

29. MIDDLEWARE SYSTEMS RESEARCH GROUP MSRG.ORG Architecture 29 Resource providers act as publishers Discovery clients act as subscribers Advertise all attributes: system = linux memory <= 2000 disk <= 320 Publish updates of dynamic attributes: memory = 1500 disk = 80 Subscribe for resources: system = linux disk >= 200 B1 B4 B5 B2 B3 Distributed Content-Based Publish/Subscribe

30. MIDDLEWARE SYSTEMS RESEARCH GROUP MSRG.ORG Static model 30 Discovery is performed locally by any single broker. Advertisement: system = linux, memory = 2, disk = 320 Subscription: memory > 1Publication B1 B4 B5 B2 B3

31. MIDDLEWARE SYSTEMS RESEARCH GROUP MSRG.ORG Dynamic continuous model 31 Traditional pub/sub routing of messages. Discovery subscription is routed to and stored at matching resource host brokers. B5 B4 B3 B2 B1 Advertisement: system= linux, memory <= 2, disk < 320 Subscription: memory > 1Publication: memory = 1, disk = 200

32. MIDDLEWARE SYSTEMS RESEARCH GROUP MSRG.ORG Similarity forwarding 32 To retrieve old results: Send covered sub to the covering sub’s discovery host broker. To intercept new results: Store covered sub at the first broker with a covering sub. Adv: system= linux, memory <= 2, disk < 320 Sub2: memory > 2Pub: memory = 1, disk = 200 Sub1: memory > 1 B1 B5 B2 B3 B4

33. MIDDLEWARE SYSTEMS RESEARCH GROUP MSRG.ORG Discovery time Similarity forwarding optimization is faster Increased discovery similarity  Normal algorithm suffers More matching resources are found  Optimized algorithm benefits Reuse results Spatial clustering of resources  Normal algorithm benefits Smaller subscription propagation tree (more “multicast”)  Optimized algorithm benefits slightly Results are often retrieved from discovery host broker Spatial clustering of discoveries  Normal algorithm suffers Congestion of messages near discovery host brokers  Optimized algorithm suffers slightly Matching of cached results is relatively cheap Clustered spatial distribution of discoveries Balanced spatial distribution of discoveries 0.0 0.5 1.0 1.5 2.0 2.5 3.0 102030405060708090100 Discovery similarity (%) Avg discovery time (s) Normal(B) Similarity(B) Normal(U) Similarity(U) 0.0 2.0 2.5 3.0 0.5 1.0 1.5 102030405060708090100 Discovery similarity (%) Avg discovery time (s) Normal(B) Similarity(B) Normal(U) Similarity(U)

34. MIDDLEWARE SYSTEMS RESEARCH GROUP MSRG.ORG Summary of service discovery Distributed event-based resource discovery framework offers:  Parallel discovery of static resources  Efficient dissemination of dynamic resource attributes  Real-time discovery of new resources Similarity optimization reuses results of overlapping  Exploits publish/subscribe covering techniques  Benefits from more skewed spatial and interest distributions The distributed architecture achieves faster discovery at the expense of increased network traffic 34

35. MIDDLEWARE SYSTEMS RESEARCH GROUP MSRG.ORG 35 Conclusions

36. MIDDLEWARE SYSTEMS RESEARCH GROUP MSRG.ORG Distributed process execution stack 36 Transactional movement Filter aggregation Distributed execution Activity placement Service discovery Service composition SLA Modelling

37. MIDDLEWARE SYSTEMS RESEARCH GROUP MSRG.ORG Summary Dynamic publish/subscribe clients can move and change interests quickly and cheaply  Fast movement primitive with transactional properties  Congestion avoidance with incremental filter aggregation Distributed process execution offers more flexible deployment options  Distributed architecture with event-based coordination  Dynamic utility-aware process redeployment Service management can operate on distributed set of service registries  Distributed service discovery including continuous discovery model  Distributed service composition based on publish/subscribe matching 37

38. MIDDLEWARE SYSTEMS RESEARCH GROUP MSRG.ORG Future research directions Consider impact of contention among shared resources  New service impacts existing applications  New application impacts existing applications  Isolate application performance Develop a hybrid process execution architecture  Allow for both replication and distribution  Support process, activity, instance granularities Tightly integrate service discovery and composition  Support continuous composition based on continuous discovery results  Allow automatic service binding at runtime based on continuous discovery results 38

39. MIDDLEWARE SYSTEMS RESEARCH GROUP MSRG.ORG 39 Thank you