1. R Harikumar, TCS Nityan Gulati, TCS
ARCHITECTURE AND DESIGN FOR PERFORMANCE OF A LARGE EUROPEAN BANK PAYMENT SYSTEM
R Harikumar, TCS
Nityan Gulati, TCS

2. System Context and Architecture Design for performance - parameters
Outline
System Context and Architecture
Design for performance - parameters
Tuning Web Page response
Tuning Straight through process throughput
Batch process optimization
Tuning Database Parameters
Key Benefits Realized
Key Lessons Learnt

3. Performance Engineering: Typical Challenges
Large system characteristics – huge volumes, complex eco system, several unknowns
Understanding application user patterns
Varying Load conditions and increasing volumes
Understanding the business and its growth
Performance planned only as a testing phase !!

4. System Context and Architecture
It is a Corporate banking payment system
Huge daily volumes. Two million txs / day, peak of 300k per hour
2000 branch users across Europe
End of day completion under 2 hours
About 20 upstream systems

5. Key Design Considerations
Web page response
Search screen page response under 2-3 seconds
Blank Searches
Graceful stopping of long running queries
Handling network latency
STP throughout
Non uniformity in the sizes of file received (~1 to 100KB -> Small, 100 to 1000KB -> medium, 2 to 60 MB -> Large files)
Non uniformity in the type of transaction in a file
STP design
Batch jobs
Huge volume of transactions
Parallel processing
Managing Concurrency and hot spots

6. Web page response tuning – Key considerations
#1:
Gracefully stopping the long running SQLs that can impact the system stability.
Configure CPU_PER_CALL threshold post which the query is terminated with ORA error.
Configure this for a separate oracle user and exercise control on when it needs to be invoked
#2:
Network latency: Avoid HTTP 304 calls from the browser. It is a quick change and provides significant gains in browser performance. HTTPWATCH and DYNATRACE tools provide great insight into the page performance
Configuration required in HTTPD to prevent 304 calls
Configuration to reload static content during software upgrade

7. Web page response tuning – Key considerations
#3:
Control on the search performance on the most frequently used screens
Identify the most used search combinations along with business and tune them using indexes for best performance
For other non popular combinations, limit the data display to a weeks data by appending the date filter automatically
Blank Search requests: it means retrieving and sorting millions of rows and then presenting possibly the first few rows
Guide the query plan with a hint such that an index with column order matching the sort order is picked .
Additionally restrict the size of the result set with a specified period. E.g. one week of data
Use ROWNUM to restrict the size of the required result set

8. Straight Through Processing - scalable design
90% of the payments received are file based
No uniformity in the sizes of file received
Non homogeneity in the type of transaction in a file
~1 to 100 -> Small, 100 to 1000KB -> medium, 2 to 60 MB -> Large files
Payment systems are pre-dominantly file based
Design of the system has to be extremely flexible and robust
The component needs to be configurable for varying load conditions
The above design helped us to process 300,000 transactions per hour

9. Batch performance considerations
Batch processing Characteristics
Large volume of transactions to be processed at the EOD as batches leading to a surge of parallel activity
Large variation in Transaction to Account ratio resulting in deadlocks
#1:
Proper allocation of transaction to threads to avoid contention to ensure we have minimal contention amongst parallel paths of execution
Allocation of transaction to threads
Thread 1
( Account 1 > 4 Txns)
Thread 2
( Account 2  3 Txns)
Thread 3
( Account 3 2 Txns)
Txn 1 - Account 1
Txn 2 - Account 2
Txn 3 - Account 3
Txn 4 - Account 1
Txn 5 - Account 2
Txn 6 - Account 3
Txn 7 - Account 1
Txn 8 - Account 2
Txn 9 - Account 1
Distinct Accounts are processed by separate threads avoiding contention
file with payment txns
Modulus (Account)

10. Batch performance considerations
#2:
Group the components into logically related clusters and execute several groups in parallel
Parallelism in batch profile
Group the jobs logically into clusters and run them in parallel to squeeze the elapsed time
#3:
Using appropriate collection objects in the code. E.g. concurrent hash map reduces the possibility of transactions entering into dead lock situations.
#4:
Apply sequence caching. It reduces the overhead of ID generation and leads to improvement of performance .

11. Database considerations
The following are the key tuning considerations that helped to tune the database performance
Sno
Tuning parameter
Remarks 1
Reduce the quantum of Redo-logging and commit less frequently
Avoid repeated updates in the same commit cycle
Place the redo logs on faster storage and separating the log members / groups on different disks [PERF003] 2
Connection Pool library
Use specialized connection pool libraries e.g. BoneCP
Reduces application wait times for connection 3
Bulking of inserts / updates
Use Add batch and execute Batch JDBC methods
Saves on network round trips 4
Controlling the JDBC Fetch Size
The fetch size selected should be optimal to balance the network traffic and the available heap available on the client side 5
Table Partitioning
It reduces the hot spots such as index last block, where we have sequentially increasing keys for the index. 6
Using prepared statements having bind variables
Promotes statement caching, reduces memory footprint in the shared pool and reduces the parsing overhead.

12. Key Achievements
The picture below shows the response times of the most frequently used screens
SLA : under 3 seconds

13. Key Achievements
SLA was to process around 300k transactions under 60 minutes
The Graph below shows payment files and the time taken to process 300k transactions S no
Type
Time Taken
( minutes)
No of TX’s 1
Debit card Txs 59
54615 2
Accounting entries-1 44 3
Payments-1 17
25040 4
Payments-2 14 5
Payments-3 9 6
Trade settlements 58
59250 7
MT202
2400 8
MT103 37
3060
SEPA messages 60
5130 10
DD- remittances 12
35550 11
Accounting entries-2 22
10260
TOTAL
300,000

14. Lessons Learnt
Performance is a key component to be planned and worked upon right from the requirement stage till the performance testing stage. It needs to be monitored in production through appropriate KPIs
Keep the of SQLs fired to the minimum per unit of output. This should be controlled during the development phase
IE 8 / IE9 give significantly better results as compared to IE 7 on account of efficient rendering APIs in these versions
Eliminating HTTP 304 calls from the browser provides quick gains. It can be done through a simple configuration in web server
Application performance should always be measured considering the actual volume of historical data that resides in the database.
Caching features need to be exploited to the maximum across all layers in all high volume applications

15. Q & A

16. Appendix

17. HTTP 304 calls- TCS.COM
BACK

18. DYNATRACE TOOL
NETWORK PERFORMANCE
BACK

19. Batch Profile - parallelism
BACK
Timeline in minutes, SLA :120 minutes 10 20 15 15 15 10 5