Course Information Andy Wang CIS 5930 Computer Systems Performance Analysis

2 Contact Information Andy Wang Office: 269 Love Building Office hours: after class (also by appointments) Class website:

Objectives Become exposed to common issues involved in evaluating software systems Learn statistical tools and experimental design techniques Gain experience in performance evaluation 3

4 What Will This Course Teach You? Proper methods for designing and performing experiments on systems Proper methods for analyzing and presenting data gathered from such experiments Proper methods for critiquing experiments and data gathered

5 What Won’t This Course Teach You? Basic systems software principles Systems software modeling Queueing theory Simulation techniques for systems software

6 Who Should Take This Course Well, everyone, but especially - Software developers Software researchers Software purchasers Software evaluators

Course Materials Lecture notes and papers (posted on the class website) Textbook –The Art of Computer Systems Performance Analysis, by Raj Jain

8 Grading Up to 5 assignments (take home exams) - 10% each Peer evaluation of projects by other groups - 10% Project – remaining grade

Project You need to develop a project in teams of two or three Goal: –Publishable results

10 Suitable Subjects for Projects OSes or their components (file systems, I/O subsystems, process handling, etc.) Compilers Databases Real-time applications Application packages Networks/networking systems The Web

Possible Projects Measure/simulate/compare systems Develop and validate a simulator Characterize workloads of a system Analyze the effects of various factors on a system Develop a software monitor to observe the performance of a system 11

Project Proposal Due on the 2 nd week –All team members are required to participate 2-page written proposal –System background and evaluation goals –Metrics –Parameters, factors, levels –Evaluation technique –Workloads –Division of labors

Experimental Design Due on the 4 th week –All team members are required to participate 2-page written design –Brief background –Experimental design –Data analysis techniques –Anticipated graphs

Project Presentation During the last week of the course 15-page (max) written paper due by the last lecture (double column, single- space, 10-pt font)

15 Grading of Projects Several criteria will be used: Proper design of the experiment Care and thoroughness of its execution Completeness of analysis Quality of data presentation Insight gained from experiment Insight is the most important

16 Evaluation of Projects by Other Groups Submitted by each student individually Two 1-page critiques on other group’s proposed experiment –Due by 5 th week Two 1-page critiques on other group’s results –Due by final’s week Graded on basis of insight into strengths and flaws of each project