1. 12-Jun-2007 Sakai Metrics at Indiana University1 That Which is Monitored, Improves Dan McKee & Megan May

2. 12-Jun-2007Sakai Metrics at Indiana University2 At Indiana University we believe, That Which is Monitored, Improves How does Indiana do this? Extended Performance Metrics [EPM] Objective today: to present three such metric categories that are in use today Course site tool usage metrics Project site tool usage metrics Database performance metrics

3. 12-Jun-2007Sakai Metrics at Indiana University3 Sakai at Indiana University Background Sakai is called Oncourse CL at Indiana University Supports over a 100,00 users Measuring Migration It runs on a Virtualized IBM AIX (Unix) Server Its Database Management System is Oracle 10g Size is approximately 60 Gbs The typical number of CL Sessions at any one point in time is around 5,000

4. 12-Jun-2007Sakai Metrics at Indiana University4 EPM*: Sakai Tool Usage Metrics Background Designed to be Database Platform Independent Based upon data contained within the Sakai Application/Database Metrics are populated via PERL scripts Data Structures are: Designed to ease reporting and sql query construction Compatible with a variety of SQL Tools Aqua Data Studio, DBVisualizer, SQL Developer,SQL4X and others Tool Usage Metric DDL is available at: http://confluence.sakaiproject.org/confluence/x/cKc (see file epm_tool_proj_20061102_ddl.sql) SQL scripts and Perl scripts used to build and maintain EPM are available at the above link. (* Extended Performance Metrics)

5. 12-Jun-2007Sakai Metrics at Indiana University5 EPM: Tool Usage Metrics Fact Table

6. 12-Jun-2007Sakai Metrics at Indiana University6 EPM: Tool Usage Dimensions

7. 12-Jun-2007Sakai Metrics at Indiana University7 EPM:Tool Usage Metrics ER* Diagram * ER - Entity Relational

8. 12-Jun-2007Sakai Metrics at Indiana University8 EPM: SQL to Populate Tool Usage Example SQL used to extract Tool Usage SQL to Extract Tool Usage for Syllabus [SYL] select substr(site_id,1,2)TM, '20'||substr(site_id,3,2)YR, substr(site_id,6,2)CMP, count(distinct(Site_Id)) CNT from onc.sakai_site_property where site_id like 'FA06%’ and SITE_ID in (select CONTEXTID from onc.sakai_syllabus_item where ID in (select surrogatekey from onc.sakai_syllabus_data where not (title is null))) group by substr(site_id,1,2),substr(site_id,3,2),substr(site_id,6,2); File containing complete sql for extracting Tool usage is available at: http://confluence.sakaiproject.org/confluence/x/cKc (see file epm_tool_upd_qry.sql) http://confluence.sakaiproject.org/confluence/x/cKc Perl script used to extract and populate the Tool Usage Metrics is available at the above link (see file update_epm_tool_ft.pl)

9. 12-Jun-2007Sakai Metrics at Indiana University9 Tool Usage by Semester for All Course Sites FA05SP06SU06FA06 % Change FA05 to FA06 Announcement30143338886462253% Assignment9751002247172177% Calendar960897233139345% Chat126411522771213-4% Discussion37840233216-43% Drop Boxes77731911-86% E-Mail Archive12701328275191951% Gradebook18682100645309966% Message Center3639731820555608% PM3438026518435321% DF2052964682240% Podcasts---21N/A Post 'Em870152432938% Resources19391990557329370% Site Info Groups1565232441527% Syllabus21222549702394486% Test & Surveys CL290250230-100% Test & Survey (original)---385N/A # Active30143338886462253% Sample Report

10. 12-Jun-2007Sakai Metrics at Indiana University10 EPM: Usage Charts

11. 12-Jun-2007Sakai Metrics at Indiana University11 EPM: How is the migration progressing? Course Section Adoption % Using CMS% Using original Oncourse% using Oncourse CL FA0543%71%29% SP0636%60%40% SU0622%61%39% FA0637%47%53% SP0736%41%59%

12. 12-Jun-2007Sakai Metrics at Indiana University12 EPM: Tool Usage Reports Fall 06 by Campus BLCOEAINKOSBNWSE Announcement2393107761106194303214226 Assignment854564147955907567 Calendar700543136762854350 Chat503391240743805376 Discussion1041412628680 Drop Boxes40050020 E-Mail Archive14118736419512034 Gradebook16237630705121229145165 Message Center7121233973411113210990 PM5961213368410512510075 DF20513913632242719 Podcasts83060400 Post 'Em181052620602 Resources18725745756121174136123 Site Info Groups126326535711 Syllabus197213171904133283192255 Test & Surveys (CL)00000000 Test & Survey (original)10154141253815208 Wiki8976418762 # Active2393131761106194303214255

13. 12-Jun-2007Sakai Metrics at Indiana University13 EPM: Tool Usage Charts Fall 06 Campus Usage Breakdown BLCOEAINKONWSBSE % Using CMS29%56%34%46%36%30%31%38% % Using original28%49%70%64%36%40%53%61% % using CL72%51%30%36%64%60%47%39%

14. 12-Jun-2007Sakai Metrics at Indiana University14 EPM: Sakai Project Metrics Background Designed to be Database Platform Independent Based upon data contained within the Sakai Application/Database Metrics are populated via PERL scripts. Data Structures are: Designed to ease reporting and sql query construction Compatible with a variety of SQL Tools Aqua Data Studio, DB Visualizer, SQL Developer, SQL4X, and others Project Count Metric DDL is available at: http://confluence.sakaiproject.org/confluence/x/cKc (see file epm_tool_proj_20061102_ddl.sql ) http://confluence.sakaiproject.org/confluence/x/cKc SQL scripts and Perl scripts used to build and maintain EPM are available at the above link

15. 12-Jun-2007Sakai Metrics at Indiana University15 EPM: Project Metrics Fact Table

16. 12-Jun-2007Sakai Metrics at Indiana University16 EPM: Project Metrics Dimensions

17. 12-Jun-2007Sakai Metrics at Indiana University17 EPM:Project Metrics ER* Diagram * ER - Entity Relational

18. 12-Jun-2007Sakai Metrics at Indiana University18 EPM: SQL to Populate Project Metrics Example SQL used to extract Project Counts Project Count for Syllabus [SYL] select count (site_id) PRJ from onc.sakai_site where type = 'project' and SITE_ID in (select distinct (CONTEXTID) from onc.sakai_syllabus_item where ID in (select surrogatekey from onc.sakai_syllabus_data where not (title is null))); File containing complete sql is available at: http://confluence.sakaiproject.org/confluence/x/cKc (see file epm_proj_upd_qry.sql ) http://confluence.sakaiproject.org/confluence/x/cKc Perl script used to extract and populate the Project Count Metrics is available at the above link (see file update_epm_proj_ft.pl )

19. 12-Jun-2007Sakai Metrics at Indiana University19 EPM: Project Metric Report and Chart Project Sites Announcement479 Assignment159 Calendar297 Chat151 Discussion170 Drop Boxes50 E-Mail Archive189 Gradebook102 Message Center250 PM212 DF112 Podcasts13 Post 'Em17 Resources528 Site Info Groups75 Syllabus145 Test & Surveys116 Wiki113

20. 12-Jun-2007Sakai Metrics at Indiana University20 EPM: Oracle Database Performance Metrics Background Oracle Database Performance Metrics are NOT Database Platform Independent -- Oracle 10g Specific Based upon data contained within: Oracle Automatic Workload Repository [AWR] Other External Sources Database Extended Performance Metrics are populated via Perl. Data Structures are: NOT Specifically designed to ease reporting and sql query construction (these structures are supplied by Oracle not Sakai) compatible with a variety of SQL Tools Aqua Data Studio DB Visualizer SQL Developer SQL4X And others Oracle Database EPM DDL is available at: http://confluence.sakaiproject.org/confluence/x/cKc (see file epm_ora_db_ddl.sql) SQL scripts, Perl scripts and documentation used to build and maintain Database EPM are available at the above link.

21. 12-Jun-2007Sakai Metrics at Indiana University21 EPM: Oracle Database Metrics from Oracle’s AWR include: Buffer Cache Hit Ratio -- AWR CPU Usage Per Sec -- AWR CPU Usage Per Txn (server) Current Logons Count Cursor Cache Hit Ratio Database CPU Time Ratio Database Time Spent Waiting Host CPU Utilization (%) Library Cache Hit Ratio PGA Cache Hit % RAM USAGE Response Time Per Txn RowCache Hit Ratio SQL Response Time (%) Shared Pool Free % Sorts in Memory

22. 12-Jun-2007Sakai Metrics at Indiana University22 EPM: Oracle Database Metrics Whose Source is External to Oracle’s AWR include: (Metrics populated from external sources via Perl scripts) CL Sessions -- Sakai Database CL User Count -- Sakai Database Size of Database -- Sakai Database Percent Used -- Sakai Database Space Used -- Sakai Database CPU USAGE -- System Administration / AWR RAM USAGE -- System Administration / AWR

23. 12-Jun-2007Sakai Metrics at Indiana University23 EPM:Database Performance ER* Diagram * ER - Entity Relational

24. 12-Jun-2007Sakai Metrics at Indiana University24 EPM: SQL to Populate Database Performance Metrics Example SQL to Oracle’s Response Time per Transaction select metric_name, to_char(begin_time,'MM-DD HH24') HR, round((minval/10),1) "Min", round((maxval/10),1)"Max", round((average/10),1) "Avg” from sys.dba_hist_sysmetric_summary where begin_time >= sysdate - 2 and begin_time <= sysdate and metric_id = 2109 order by begin_time; Perl script used to extract and populate the Tool Usage Metrics is available at: http://confluence.sakaiproject.org/confluence/x/cKc (see files extended_perf_metric_update.pl and extended_perf_metric_usercnt.pl ) http://confluence.sakaiproject.org/confluence/x/cKc

25. 12-Jun-2007Sakai Metrics at Indiana University25 EPM: Database Performance Reports Example Response Time Report MetricHourMinMaxAvg ------------------------- ----------- ----- ----- ----- Response Per Txn11-06 040.210.41.8 Response Per Txn11-06 050.616.21.8 Response Per Txn11-06 060.612.71.9 Response Per Txn11-06 070.211.92.3 Response Per Txn11-06 080.230.52.5 Response Per Txn11-06 090.182.64.4

26. 12-Jun-2007Sakai Metrics at Indiana University26 Example Response Time Chart EPM: Database Performance Charts

27. 12-Jun-2007Sakai Metrics at Indiana University27 Available to Oracle Enterprise Manager [Grid] Via Database Links Views from the Sakai Database into your OEM Repository Contact us if you have such an interest: damckee@indiana.edu 812-855-7197 EPM: Database Performance Charts

28. 12-Jun-2007Sakai Metrics at Indiana University28 That Which is Monitored, Improves Questions?