iAssessment’s High Performance Gateway Presentation System 

Background Implementation Pitfalls Factors to Consider Questions and Answers Presentation Overview

Background / History Web/Portal History –Static – Dynamic – Application based - Portal iAssessment Evolution and Products –K-12 ASP Compliance and Reporting –Assessments and Survey –Teacher Resource Cataloging –Professional Development Grant/Policy Management –Communication Tools

uPortal – iAssessment Gateway Presentation System (GPS) –User management –Entity management –Individual client branding Controlled User Experience –User profile based –Role based applications –Client controlled not user controlled Integrated User Experience –Other disparate systems Background / Solution GPS K12 Smart Portal

Product Map

Implementation / Case Study GPS Case Study / CTAP 2 California Department of Education –>300,000 teachers & administrators –50,000 initial students for beta student system (with one district having ~2 million students) Use Model –Administrators –Teachers –User created accounts –Gradual adoption –Peaky use times (grant deadlines, etc.) –Account reuse

Implementation / Case Study Current CTAP 2 Statistics –~200,000 accounts –Peaks of 1,000 concurrent logins –Peak “season” average 5,000 logins per day –Tables with more than 20 million rows Architecture Requirements –Expandable at top level Load balancing Scaling Easy machine addition –Scalable software model Fixed session at login

Implementation / Case Study CTAP 2 Hardware –Web/Application Servers Sun 220R (2CPU/2GB) NFS Server Four X1’s (1CPU/1GB) –Database Server SunFire v880 (6CPU/12GB) T3 Disk Array redundant pair –Cisco content switch –Cisco Pix Firewall

Implementation / Case Study Current Hardware Capabilities –X1 can handle ~150 users each –220R ~400 users –Application front end current max load ~1,000 concurrent users –Database max load 3,000 users (keeping in mind the different load factor of iAssessment’s applications running in uPortal) Still could use optimization as well.

Implementation / Case Study Software –Solaris 8 & Linux (web/application servers) –Sybase 12.5 / Solaris 8 (database servers) –Tomcat 4.0 –Apache 1.3 mod_webapp –JDK –uPortal 2.0

Non-performance-based uPortal changes Created generic servlet wrapper channel –Run with and without uPortal Layout management modifications –Layout per user type, school type, etc. Authentication modifications –First name, last name, id, password Database connection pool sharing

Things That Worked Planned uPortal extensibility generally worked well Relative URL space worked well –Load balancing ( –Integration into existing URL namespace –https might be a problem

Problems & Fixes HTML production: XML “well-formed”- ness and XLST processing speed –jTidy didn’t work well –Fearful of XSLT overhead –FIX: avoided XSLT processing

Problems & Fixes High memory use (400MB-1GB/process) –Intrinsic needs of applications and ??? –Bugs (SmartCache, database connection pool) FIX: code changes –Caused java.lang.Runtime.exec failure (no vfork equivalent in Java) FIX: created “exec server”

Problems & Fixes Large database hit with large groups –Debug Sybase JDBC driver and ~20,000 UP_USER rows –uPortal 2.0 queries database for each member in set; long delay during account creation –FIX: used production JDBC driver; uPortal 2.0 code should be changed

Problems & Fixes Proxy servers and “User-Agent” HTTP header removal –FIX: wrapped original HttpServletRequest & protected BrowserInfo –Better FIX: all in one wrapper

Problems & Fixes Cisco content switch problems –Used IP mapping because cookies didn’t work –Load balancing from NAT/proxy didn’t work well with IP mapping –Single point of failure –No method of determining which machine user was on –FIX: using content switch for login redirection only

Problems & Fixes Caching of layouts –Performance gain, but required inter-JVM communication FIX: created small inter-JVM communication system –Aggregated Layouts implication (with multiple JVMs)? User-caused problems –Multiple submits of same request FIX: used a ‘please wait’ screen where needed

Performance Modifications Did layout caching –Per JVM –Reloaded on demand from inter-JVM message Bypassed channel-level XSLT transformations Didn’t use debug drivers in production Limited duplicate burdensome requests

Scalability Modifications Support for multiple web/application server machines Support for large JVM size

Lessons Learned Memory is limiting factor for GPS Well-formed HTML hard to achieve even when we control all the code Simple load balancing via publicly exposed web/application servers easiest and most effective

Lessons Learned Environment is hostile –Unfriendly proxy servers –“Mega proxies” (AOL) –Impatient users –Impatient firewalls

Things to Consider Architecture to support multiple web/application servers –Inter-JVM issues possibly –Per-server information (# of users, etc.)? Existing servlet code –To wrap or not? –Bypass XSLT?

Things to Consider Environment –Non-technical users –Unfriendly proxies, or firewalls? Application types –Large JVM size? –CPU/database intensive?

Questions and Answers “Why haven’t you checked in the code changes? “Can we get a copy of the code you did for xyz?” “How did you avoid XSLT processing?”