1. iAssessment’s High Performance Gateway Presentation System 

2. Background Implementation Pitfalls Factors to Consider Questions and Answers Presentation Overview

3. Background / History Web/Portal History –Static – Dynamic – Application based - Portal iAssessment Evolution and Products –K-12 ASP Compliance and Reporting –Assessments and Survey –Teacher Resource Cataloging –Professional Development Grant/Policy Management –Communication Tools

4. uPortal – iAssessment Gateway Presentation System (GPS) –User management –Entity management –Individual client branding Controlled User Experience –User profile based –Role based applications –Client controlled not user controlled Integrated User Experience –Other disparate systems Background / Solution GPS K12 Smart Portal

5. Product Map

6. Implementation / Case Study GPS Case Study / CTAP 2 California Department of Education –>300,000 teachers & administrators –50,000 initial students for beta student system (with one district having ~2 million students) Use Model –Administrators –Teachers –User created accounts –Gradual adoption –Peaky use times (grant deadlines, etc.) –Account reuse

7. Implementation / Case Study Current CTAP 2 Statistics –~200,000 accounts –Peaks of 1,000 concurrent logins –Peak “season” average 5,000 logins per day –Tables with more than 20 million rows Architecture Requirements –Expandable at top level Load balancing Scaling Easy machine addition –Scalable software model Fixed session at login

8. Implementation / Case Study CTAP 2 Hardware –Web/Application Servers Sun 220R (2CPU/2GB) NFS Server Four X1’s (1CPU/1GB) –Database Server SunFire v880 (6CPU/12GB) T3 Disk Array redundant pair –Cisco content switch –Cisco Pix Firewall

9. Implementation / Case Study Current Hardware Capabilities –X1 can handle ~150 users each –220R ~400 users –Application front end current max load ~1,000 concurrent users –Database max load 3,000 users (keeping in mind the different load factor of iAssessment’s applications running in uPortal) Still could use optimization as well.

10. Implementation / Case Study Software –Solaris 8 & Linux (web/application servers) –Sybase 12.5 / Solaris 8 (database servers) –Tomcat 4.0 –Apache 1.3 mod_webapp –JDK 1.4.0 –uPortal 2.0

11. Non-performance-based uPortal changes Created generic servlet wrapper channel –Run with and without uPortal Layout management modifications –Layout per user type, school type, etc. Authentication modifications –First name, last name, id, password Database connection pool sharing

12. Things That Worked Planned uPortal extensibility generally worked well Relative URL space worked well –Load balancing (http://ctap2w{1,2,3,4,5}.iassessment.org/)http://ctap2w{1,2,3,4,5}.iassessment.org/ –Integration into existing URL namespace –https might be a problem

13. Problems & Fixes HTML production: XML “well-formed”- ness and XLST processing speed –jTidy didn’t work well –Fearful of XSLT overhead –FIX: avoided XSLT processing

14. Problems & Fixes High memory use (400MB-1GB/process) –Intrinsic needs of applications and ??? –Bugs (SmartCache, database connection pool) FIX: code changes –Caused java.lang.Runtime.exec failure (no vfork equivalent in Java) FIX: created “exec server”

15. Problems & Fixes Large database hit with large groups –Debug Sybase JDBC driver and ~20,000 UP_USER rows –uPortal 2.0 queries database for each member in set; long delay during account creation –FIX: used production JDBC driver; uPortal 2.0 code should be changed

16. Problems & Fixes Proxy servers and “User-Agent” HTTP header removal –FIX: wrapped original HttpServletRequest & protected BrowserInfo –Better FIX: all in one wrapper

17. Problems & Fixes Cisco content switch problems –Used IP mapping because cookies didn’t work –Load balancing from NAT/proxy didn’t work well with IP mapping –Single point of failure –No method of determining which machine user was on –FIX: using content switch for login redirection only

18. Problems & Fixes Caching of layouts –Performance gain, but required inter-JVM communication FIX: created small inter-JVM communication system –Aggregated Layouts implication (with multiple JVMs)? User-caused problems –Multiple submits of same request FIX: used a ‘please wait’ screen where needed

19. Performance Modifications Did layout caching –Per JVM –Reloaded on demand from inter-JVM message Bypassed channel-level XSLT transformations Didn’t use debug drivers in production Limited duplicate burdensome requests

20. Scalability Modifications Support for multiple web/application server machines Support for large JVM size

21. Lessons Learned Memory is limiting factor for GPS Well-formed HTML hard to achieve even when we control all the code Simple load balancing via publicly exposed web/application servers easiest and most effective

22. Lessons Learned Environment is hostile –Unfriendly proxy servers –“Mega proxies” (AOL) –Impatient users –Impatient firewalls

23. Things to Consider Architecture to support multiple web/application servers –Inter-JVM issues possibly –Per-server information (# of users, etc.)? Existing servlet code –To wrap or not? –Bypass XSLT?

24. Things to Consider Environment –Non-technical users –Unfriendly proxies, or firewalls? Application types –Large JVM size? –CPU/database intensive?

25. Questions and Answers “Why haven’t you checked in the code changes? “Can we get a copy of the code you did for xyz?” “How did you avoid XSLT processing?”