1. Managing Process Integrity (Chapter 8)
B. Ramamurthy
11/27/2018

2. Topics for discussion Review of last lectures:
Services layers diagram (Ch. 1-7)
Webservices (design and deployment)
Relate these two in the context of design, development, deployment cycle of an SOA.
Lets understand WSDL
Lets look at the components of WSDL and look at Amazon.com ECS as example
Review for midterm
Process Integrity
11/27/2018

3. Anatomy of a WSDL document
11/27/2018

4. Sample WSDL http://www.w3.org/2001/04/wsws-proceedings/uche/wsdl.html
11/27/2018

5. Significance of WSDL
WSDL for a web service (WS) when specified in an IDE (such as Netbeans) exposes the operations and the signature for the operations.
This can be used access the operations of the WS.
When designing business systems using BPEL, WSDL represents the requirements of a process in a BPEL.
Business BPEL WSLD  services (SOA)
Thus WSDL is the link that aligns a business process with an IT service.
11/27/2018

6. Data integrity Consistency Accuracy Correctness of data
Entity integrity: each row in relational table must be uniquely identified
Domain integrity: the values must fall within a specified value range
Referential integrity: specifies the validity of relationship between different data sets.
11/27/2018

7. Process integrity Business processes span multiple IT systems.
There are times when a process state may be inconsistent. For example, a cancelled order takes some time to register with order system.
Technical failures such as network failure: system-wide solution
Business exceptions must be handled at the process level: WSDL fault provisions
Special cases: overbooking of a flight (cannot be handled by domain integrity.) Must be handled at code level.
Process ownership is another problem. For example, consider a workflow application.
11/27/2018

8. Process integrity solutions
Logging and tracing
ACID transactions
Atomicity, consistency, isolation, durability
Transaction monitors
2PC: two-phase commit protocol is used to ensure ACID properties for transactions that span more than a single resource manager.
Phase 1: all participating resource managers acquire locks on shared resources
Depending on phase transaction coordinator informs participants whether to commit or rollback
Phase 2: participants either commit or rollback to previous state depending on instruction from coordinator
Issues: performance, lack of support for long-lived transactions, integration of legacy systems, not suited for discontinuous network, use of multi-level and nested transactions
11/27/2018

9. Process integrity solutions: transactional steps
Combination of transactional steps and persistent queue can improve reliability of complex processes.
Transactional steps are a set of closely related activities executed in the context of a single complex transaction.
Transactional steps facilitate the decomposition of complex, long-lived business processes into individual steps with short execution times and high transactional integrity.
Transactional steps dramatically increase the robustness and flexibility of distributed systems.
~ provide a great way of ensuring the integrity of individual process steps.
11/27/2018

10. Process integrity solutions transaction chain & compensation
ACID properties are too strong for complex transactions
Chained transactions with compensating transactions offer a viable means of dealing with process integrity.
Compensating transactions undo previously executed steps if a problem is encountered during the execution of a particular step in a transaction chain.
11/27/2018

11. Sagas
Sagas are formal workflow models that build on the concept of chained transactions (steps).
A Saga describes a workflow wherein each step associated with a compensating transaction.
How do you compensate “out of funds” failure?
11/27/2018

12. BPM and process integrity
BPM (Business Process Management) platforms provides features that enable business analysts and developers to design, execute and manage instances of complex business process.
Introduction of BPM will help the process integrity aspects of an enterprise application.
Explicit modeling separates technical integrity from process integrity
BPM engine provides mechanisms for monitoring processes at runtime
BPM supports compensating transactions which is key concept for managing the rollback of partially executed processes after a failure.
11/27/2018

13. Related WS-Standards See fig. 8-4
A number of different industry alliances and standardization bodies are working on WS-based transaction protocols.
OASIS Business Transaction Processing (BTP)
WS Flow language
WS-BPEL
BPEL4WS
XLANG
WS-Composite Application Framework
WS-coordination
WS-Transaction …
11/27/2018

14. Process Integrity Spectrum
2PC/TPM in
Heterogeneous env.
2PC/TPM in
homogeneous env.
Level of integrity
Sagas with complex
compensation
BPM engine with
Simple compensation
Transactional Steps
Custom-built log
infrastructure
DB log analyzer
Eyes closed &
Praying
Implementation cost
11/27/2018

15. Mid-term review Date: 11/29 Duration: 1 hour 30min
Topics: 1-7 Chapters
Closed book
Topics for questions:
Fundamental concepts: synchrony, loose coupling etc., interface vs. payload semantics
Service types
Design of an SOA for a application: layered architecture
11/27/2018

16. Process Integrity in SOA Case Study: Travel Itinerary Management
Analyze the problem
Choose concurrency control: optimistic vs pessimistic
Use idempotent update operations
Avoid distributed 2PC
Build transactional steps
Design simple and flexible compensations
Combine SOA, MOA and BPM for flexibility
11/27/2018

17. Problem Statement & Analysis (8.3.1 & Fig.8-5)
Typical sites enable you to book individual flights.
The new travel itinerary mgt. system should create complete itineraries for trips including car, hotel and flights.
Reuse existing IT subsystems esp. customer management and billing systems.
Two different front-ends: simple html front and a complex front-ends for call center (VB GUI) functionality
Three backend systems: customer management, billing, and other complex processes such as customer complaints etc.)
A number of partner systems need to be integrated; example: partner airlines, hotel and car reservation systems.
Key transactions are: confirm itinerary (typically irreversible) and create invoice; these transactions
cross organizational boundaries
cross several technical boundaries
involve different databases
involve different departments
11/27/2018

18. Architectural model 11/27/2018 Enterprise
Reservation systems (cars, hotels, other airlines)
Enterprise
Reservation systems (cars, hotels, other airlines)
Customer
Customer &
Itinerary DB
Web
front-end
IncidentManager
Process state DB
Billing
Callcenter
front-end
Invoice DB
11/27/2018

19. Concurrency Control (8.3.2, figs.8-6, 8-7)
This specifies how to deal with concurrent access to shared data in an SOA.
Optimistic and pessimistic
Pessimistic control gives exclusive access to data through acquisition of locks; locks typically are held for short duration.
Optimistic control: no locks are acquired during the transaction execution, only when data needs to be updated at the end of the transaction.
Optimistic control is the choice for long running transactions.
11/27/2018

20. Implementing Optimistic Concurrency Control
In order to determine write conflicts, optimistic concurrency model must, at the end of each transaction, determine whether a competing transaction has changed the data after it was initially checked out.
Different ways: timestamps, version counts, state comparisons
For the first two an extra column is added in the relational table
Version number or timestamp is added to the primary key used in the WHERE clause of UPDATE operation
State comparison is implemented in Microsoft’s ADO.NET; advantage: does not require extra column in the relational table
Version numbers are associated with a service at root level
If a client is a trusted party then version number is preferable solution; else state comparison
11/27/2018

21. Example Customer profile
Customer updates using web || call center updates using telephone
Customer and call center read profile version #645
Customer updates, saves, version #646
Call center looses its updates, needs redo due version conflict
Alternatively, we could allow merge operation; possible with state comparison
If nothing works resort to pessimistic control, such as in an incident management service
11/27/2018

22. Optimistic concurrency control
Read customer
profile
update customer
Version
conflict
Commit changes
Version inc. no
yes
Merge
possible
yes
Notify user
yes
merge changes
11/27/2018

23. Pessimistic concurrency control
Get a lock on
Incident object
Lock
Avail? no
Notify user
yes
Work on Incident
yes
Possibility for discard changes as well
as supervisor notification no
Commit changes
Release lock
11/27/2018

24. Make updates Idempotent
Effect of a N (N > 1) of the same requests is same as one request.
Use sequence numbers
interface Itinerary {
SQN getSeqeunceNumber();
void addBooking (in SQN s, in booking b); }
Can be used in error handling
11/27/2018

25. Idempotent Operation in error handling
while (retry limit not reached)
try {
// two idempotent operations in a block
itineraryManager.confirmItinerary();
invoiceManager.createInvoice(); }
catch (FatalError e) {
//manage retry limit counter
if (not successful) {
// handle error scenarios
11/27/2018

26. Avoid Distributed 2PC
First iteration: Client controlled transactions (fig. 8.9):
Has a transactions manager which implements the 2PC: begin commit & commit
Client should deal with distributed transactions
Lacks loose coupling
Second iteration: server controlled transactions (fig. 8.10):
Transactions spanning multiple databases combined on the server side
Tightly coupled, still 2PC, cost of sophisticated transaction monitor
11/27/2018

27. Building Transactional Steps
Split transactions and introduce queues
confirm_itinerary() operation stores a message in a “pending confirmations” queue
A background thread uses this queue as input and executes operation and stores in “confirmed itineraries” queue
System picks from this queue and informs customer of success or failure
Problem: a transaction monitor is needed for 2PC among queues and DB
Solution: introduce services at various layers
11/27/2018

28. Granularity Level BEING { pendingConfirmQ.get(); try {
itineraryMgr.confirm_itinerary(); }
catch (e1) { errorQ.put(e1); return; }
COMMIT;
invoiceMgr.create_invoice(r1);
catch (e2) { errorQ.put(e2); return;}
confirmedItineraryQ.put();
11/27/2018

29. Compensating Logic Use simple compensating logic
Confirm itinerary  Cancel itinerary
Create invoice cancel invoice
Compensating logic may not work all the time. In such cases treat them as special cases at the application level (see 8-16)
11/27/2018

30. SOA+MOA+BPM
Combine SOA (services) , MOA (messages and queues) and BPM (business processes, compensation and special handling) to increase flexibility
Read Ch.8 and review the figures explaining these concepts
11/27/2018