Martin Pincott, Technical Director

unique requirements + special solutions

Unique Requirements 1. Explanation of system 2. Demo of Modules/Applications Special Solutions 3. Ramp it up, WebApp copes 4. Key points to remember WebApp to the Xtreme

Unique Requirements Why it’s a large website Organically grown over time First Application used in 2001 –Contained 429 specific data entry fields –Used for one purpose Ease-of-use benefited NHS –Provided an indication of the status of Estates & Facilities services in the NHS for the Department of Health.

What was solution for NHS improvement in monitoring targets “ Managing today's healthcare organisations poses unique challenges, and management need to be able to base their decisions on up-to-date and relevant information. NHS Estates' Performance Management Division is dedicated to supporting healthcare managers in this. Maintaining a vast range of data relevant to the project management of new healthcare premises and operating performance of NHS trusts.”

What was solution for Effective collection and monitoring of data Reporting and Analysing large amounts of data Providing a National overview using commonly defined data requirements

How was it constructed How was it constructed Originally developed as a WebApp 2.0 Upgraded into a WebApp 3.0 Split into separately developed programs All programs upgraded into WebApp 9.1 More programs added All programs upgraded into WebApp 10.0

Extreme Users Extreme number of users 4396 User Accounts 786 Organisations Sites 11 Modules

Extreme Operation Rapid development Extreme number of data entry points Servers are hosted on both the internet (WWW) and the NHS Net (NWW - Europe's largest private network)

Explanation of System Why it’s the first in its class Copes with peaks Designed for government environment Other web systems have been developed, but never succeeded due to the high demand of target deadlines.

WebApp achieves Identified Result: An extremely large application can be built successfully for high user demand on a large database with small resources.

Demonstration Please wait …..

Testing and Scaling up for High Volume Initial Problem Very high transaction volumes due to deadlines for data entry A high number of intensive users –3 week period up to deadline Without some control, sites generate Event 201 errors Max “safe” concurrent session count only 62 WebApp to the Xtreme

The Requirement How to set about testing more scientifically How to scale up to meet demand.

Testing Web Applications Testing Web Applications Microsoft Web Application Stress Tool Set up test server Create test script Run repeatedly –adjusting Stress test system threads count WebApp concurrent session count –monitoring NT Application event log Internal Dataflex (i.e. application) log files Memory and processor usage Stress Test system log

Test Goals Initially Session count limit –No event 201 errors Adjust concurrent session count setting –Show minimal Event 208 errors Adjust stress test Threads setting Applications running at full safe capacity Later Establish sensible limits –Memory –Performance

Initial Test Hardware Twin Pentium 1 Gig Processors 256 Meg ram WebApp and MS SQL Server both in this server Result Stress tool - 8 Threads WebApp concurrent session count for no errors - 67 Conclusion Limitation of concurrent session count protects against Event 201 errors

Separate MS SQL Server MS SQL Server deactivated, second (low spec) server with MS SQL substituted System Spec for MS SQL Server –Single Pentium 700 Processor –256 Meg Ram –Server also in use by development team Result Safe session setting of 67 could not be achieved. Doubling the WebApp timeouts allowed 67 connection count limit Actual setting limits were unreliable

Low Spec MS SQL Server Low Spec MS SQL Server Conclusion Event 201 errors are timeout related Anything that slows the response of the data-base down will cause 201 errors. Lengthening time outs is unproductive –Throughput declines

Low Spec MS SQL Server Low Spec MS SQL Server Recommendations Do not run MS SQL on a server that is also doing other things. One task, one server Do not use an old, under specified PC for MS SQL Server

High Performance Server High Performance Server New Server hardware Twin Xeon 3Gigahertz 2Gig Memory WebApp 9.1 WebApp and MS SQL in same server No Process pooling Results 32 Threads 277 Concurrent sessions Question Why could the (similar) live server not support these figures?

High Performance Server High Performance Server The live server was also supporting a small.NET application The.NET application was taking the equivalent of 1.75Gig of memory away from WebApp – for far less application complexity

High Performance Server High Performance Server Conclusions Moving to a higher spec server gives considerable bonus –256Meg ram supported 67 Session Count, –2Gig supports 277 Session Count Further 1 gig increased this to 324 Confirms relationship of memory availability to session count limit Recommendation Do not run.NET in the same server as WebApp

Separate WebApp Server and MS SQL WebApp and MS SQL server on separate servers Twin Xeon 3Gigahertz 3 Gigahertz processors 2Gig Memory WebApp 9.1 No Process pooling Results 32 Threads 277 Connection limit Peak sessions e-fm 236 Total Sessions 418 Peak sessions Eric 164 Total sessions 631 Slightly better throughput

Network speeds increased Network speeds increased Servers moved to separate, 1 gig Network Results No change in connection limit Throughput increased again

Network speeds increased Network speeds increased Conclusion Throughput is increased by –Separating MS SQL server from WebApp –Increasing Network speeds Session count is limited by available memory Recommendation Session count should be set such that memory does not start to page to disk – –in a 2 gig server, approx 1.5 gig memory usage

WebApp Server Enterprise WebApp Server Enterprise Master and one slave, separate MS SQL Server Twin Xeon 3 Gigahertz processors 2Gig Memory WebApp 10.0 No Process pooling Results 40 Threads 274 session count limit –3 less No significant change in peak/total sessions Master shows –<512 Meg memory in use, –Processor <10%

WebApp Server Enterprise WebApp Server Enterprise Conclusion Similar to single WebApp server Master/Slave configuration reduces safe connection count by three Master is very unstressed These results would scale up to multiple slave WebApp servers Observation Web App Enterprise master does not need a lot of memory.

Process Pooling Results 160 Threads (Previous 40) 1050 Session Count (Previous 274) Sessions e-fm peak 765 Total 1430 Sessions Eric peak 645 Total 2536 Shared between –e-fm < 80 processes –Eric max 81 processes Static memory load

Process Pooling Conclusion Huge increase in connections and throughput Session count and Process count decoupled Threefold increase in peak and total sessions Much increased throughput Much easier tuning

Process Pooling Other Observations No other effects detected when PP enabled Robust

Usability Initial 67 concurrent sessions Very slow working at peak times to Some complex reports unusable at peak times Now 1050 concurrent sessions Negligible delay when stress test running –Complex report now delivered in less than 10 seconds –Better than current live system

Scaleability Multiple Slave Servers now a matter of Good user performance Resilience and failover Spare capacity to deliver adequate performance in the event of any hardware failure

What did we learn? Load-testing is the key to tuning Do not guess – test! Your goal is a combination of –stability –performance –failover protection Understanding concurrent sessions is key. Memory usage is critical –Do not allow to page (go virtual) Process pooling is the key to controlling memory usage. Re-test will be necessary on adding new modules and applications

What did we learn? That all of this actually does work! Comments from Steve Meeley, Data Access Worldwide Keep.NET away from anything that is performance critical! Comment from ASCKEY!

Process Pooling Tips Process Pooling Tips Track the users through the system using a session table In your DD's Procedure mReset_DDO Set pSite_ID To "" End_Procedure In Your WO's Procedure mReset_WO Set pTrack_ID To "" Send mReset_DDO Of Site_DD Send mReset_DDO Of Block_DD End_Procedure

Thank you ! Any questions? WebApp to the Xtreme