Tuning Sakai for Large Production Systems: Hints and Tricks Raúl Mengod Universidad Politécnica de Valencia (Spain)

2 Overview Introduction: Universidad Politécnica de Valencia (UPV) Sakai in the UPV Servers Infrastructure Database Numbers Statistics of our first academic year. Risk Analysis Tuning the Database 24/7 Tuning Tomcat Tuning the application

3 Valencia, a place to visit

4 CAMPUS DE VERA

5 Introduction Universidad Politécnica Valencia (UPV) –Public academic institution since 1971 –Official and non-official studies –Graduate and Post–Graduate studies The UPV in numbers –4 Campuses – 15 Colleges –40 Research centers – 40 Official Degrees –20 post graduate studies – Students – Teachers –1.400 Staff

6 Sakai in the UPV Full production System in 2006/2007 –5 servers –Oracle Database v 9.2 –Integrated with SSO –Integrated with enrollment system –4.000 sites per year – students Based on the version –Customized to university requirements –A split branch from sakai 2.1.x maintenance branch –Many changes to the code, i18n, etc… –Almost impossible to merge patches/fixes

7 Sakai in the UPV

8 Server Infrastructure

9 Database Server –ULTRA SPARK RISK 1.4 GHz –16Gb RAM –Solaris v.10 Application Server –Xeon Multi-thread 3.4 GHz –4Gb RAM –Linux-Red Hat 3.0 Hw Balancer –CISCO CSS 11000

10 Database Numbers Sakai –# sites……………………………: –# students……………………….: –# enrollments…………………...: Resources –# resources items…………….…: –# resources folders……….….….: Melete –# sites using melete…..………..: 839 –# melete modules …….………..: –# melete sections……………….: Samigo –# sites using samigo…………....: 461 –# assessments………...………..: –# question pools………..……….: Other Tools –# assignments………………….: –# announcements………………: 5.896

11 Evolution Sept May 2007

12 Evolution Sept May 2007

13 Evolution Sept May 2007

14 Performance Problems Database Deadlocks Samigo-Gradebook integration High response time in Resources Tool High response time handling question pools Unacceptable response time in DropBox Tool Slow performance in Assignments Lists Unnecessary Database accesses of ResourceLoader General performance slow down with Site Stats Tool

15 Performance Risk Analyze What is the main risk for performance? –Developers Worried about ‘what to do it’ and not ‘How to do it’ Problems are generated with the design of the application –Test environments Developing environments are not good for performance testing QA is done on empty databases –Unknown future Unknown number of concurrent users

16 Tuning Sakai Database – uide+-+Database+Configuration+and+Tuninghttp://confluence.sakaiproject.org/confluence/display/DOC/Sakai+2.4+Admin+G uide+-+Database+Configuration+and+Tuning Sakai.properties – =false SAM_MEDIA_T.MEDIA BLOB in Oracle Enable caching queries on MySql Disk I/O issues with Linux –Heavy queries monitoring –Deadlock monitor Sakai integration problems with samigo-gradebook –Folllow the advices of your database provider –Trust your Database experts. Sakai is just another application

17 Indexes are our friends Access to table without an index is a possible future problem. –Developers sometimes are unaware of that problem –The problems doesn’t come suddenly Access to child table –Sometimes the PK is not used for all accesses –All access to an AK in a table should have an index

18 Logging SAKAI_SESSION and SAKAI_EVENT –Growing all the time –Never are deleted by Sakai –No indexes –Slows down some processes Our Database –SAKAI_SESSION… records –SAKAI_EVENTS…… records –Remove records to a Backup Table –Nightly process

19 Tuning Sakai Database SqlLAB –Test Database –Identify heavy queries (by time, by block) –Identify repeated queries –Is not free (Quest Software) Examples –Missing Indexes in SAMIGO –Multiple accesses to SAKAI_PREFERENCES Table from ResourceLoader –Child record recovered when not is necessary

20 Performance means 24 / 7 EMC Systems –Non stop for 24 hours (1 minute) –Make backup from Snapshots –Data files copy and Database Export

21 Tuning Sakai Servers Tuning –Have a good balancer IP balancing is simple but worked fine – Admin+Guide+-+JVM+Tuninghttp://confluence.sakaiproject.org/confluence/display/DOC/Sakai+2.4+ Admin+Guide+-+JVM+Tuning JAVA_OPTS Tuning the Garbage Collector (Sun) Java Tuning White Paper –UPV JAVA OPTS Maximum Heap 2Gb (Linux supports a maximum of 2.1 Gb) Minimum Heap 2Gb Perm Space 512 Mb No Out of Memory Working fine with users Stressed up to 700 users

22 Tuning Sakai Application/Sakai Tuning –Hibernate can help you or kill you Careful use of sets and subclasses Use of deferred queries when possible Be aware of what hibernate is doing –Tomcat logging Check the logging status INFO or WARN Working fine with INFO

23 Performance Testing Jmeter –Apache Foundation project –Sakai test suites available on reference directory Positive –Tests the whole system –Test max number of sessions working –Identifying bottlenecks Negative –Difficult to generate a real TestSuite –Not good for integration testing

24 Conclusions Follow the advices in the admin docs Try to anticipate to the performance Problems Have always ready an spare server Monitor systems Don’t trust anyone and Test, Test and Test.

Thank You