Outcomes Assessment in Computer Science Robert Lingard Assessment Coordinator August 27,

Outline  The Importance of Assessment  Problems with Assessment  Goals of an Assessment Process  An Approach to Assessment  Applying the Software Engineering Paradigm  The Annual Assessment Process at CSUN  Conclusions 2

Importance of Assessment  Assessment is required by accrediting boards (e.g., ABET).  Those with vested interests (financial and otherwise) in education are demanding accountability.  Without assessment it is difficult to know what changes to make to improve learning. 3

Problems with Assessment  Pressure for assessment can cause a “rush to assessment,” and produce meaningless results.  Pressure to “close the loop” can cause premature decisions regarding program changes.  There is a tendency to only assess high performance areas in order to “showcase” a program.  Results are often not validated.  The effects of changes are often not assessed. 4

Goals of an Assessment Process  To facilitate the direct assessment of student learning  To measure student retention and the ability to apply what they have learned  To provide mechanisms to ensure the continuity of the process  To be an efficient and natural extension of normal operations  To satisfy the requirements of ABET and the University for assessment 5

Approach to Assessment  “The first and only goal [in education is to] teach for long-term retention and transfer.” – Diane Halpern  The best assessments of student learning are based on direct measures of achievement.  How well students have learned is best assessed by looking at their ability to apply what they learned earlier to new situations. 6

Requirements for a Complete Assessment Process  It must be comprehensive – must cover the full range of learning outcomes  It must include multiple judgments – multiple sources of evidence must be used  It must include multiple dimensions – different facets of student performance must be included  It must collect direct evidence – direct measures of student attainment must be used 7

The Software Engineering Paradigm  Understand the problem  Plan a solution  Carry out the plan  Make sure the solution is correct 8

Understanding the Problem  Before beginning assessment, the first step is to decide what to assess – i.e., pick the most important things to assess.  Collecting information from faculty, students, alumni, and employers can give hints as to where learning problems exist.  Surveys, faculty meetings, and student interviews are ways of collecting this information. 9

Planning a Solution  Once a learning outcome is selected, a plan for conducting the assessment must be developed.  It must be determined how the assessment will be done – i.e., embedded questions in exams, a standard instrument, etc.  How the results will be evaluated must also be determined – e.g., rubrics must be developed.  It must also be determined who will do the various tasks required and when they will be done. 10

Carrying Out the Plan  The steps of the plan must be carried out, and the plan must be monitored to ensure successful completion of the assessment.  Someone must be designated as the lead, and this person has the responsibility for monitoring the plan. 11

Making Sure the Solution is Correct  Assessment results obtained must be validated – this step is often omitted.  One way to validate results is to make comparisons among several independent assessments using different approaches.  Failure to validate the results can result in program changes that may not be appropriate.  After program changes are made the effects of the changes must be assessed. 12

13 Reliability and Validity Reliable but not valid Valid but not reliable Reliable & valid

Questions to Ask about an Assessment  Was the instrument we used reliable?  Did we measure what we intended to measure?  If we used a sample, was the sample size large enough to be confident in the results?  If we are comparing the results of two assessments, are the differences significant? 14

Iteration  Like software engineering this process is iterative.  Once program changes have been made, they must be assessed to determine whether the desired result has been accomplished.  Analysis of this reassessment might indicate the need for further changes. 15

16

The Annual Assessment Process at CSUN 1. Department forms Assessment Committee The program is divided into seven areas and a coordinator is chosen for each area. These coordinators constitute the Assessment Committee. 2. Assessment Committee recommends outcomes/objectives to be assessed The Assessment considers results from previous assessments, informal assessments, and the length of time since particular outcomes have been assessed to determine the current set to be assessed. 3. Department approves assessment goals and assessment plans are prepared After department approval the Assessment Committee decides on an assessment approach and develops a schedule of activities. 17

The Assessment Committee Consists of the Program Area Coordinators PROGRAM AREA COURSESPROGRAM OUTCOMES Fundamental Concepts Rick Covington  Intro. to Algorithms & Prog.  Data Structures & Prog. Design  Advanced Data Structures  Plus Electives  Demonstrate an understanding of algorithms and data structures.  Demonstrate proficiency in using a high-level computer language.  Demonstrate a problem solving ability. Systems Robert McIlhenny  Computer Architecture  Computer Organization  Operating Systems & Sys. Arch.  Plus Electives  Demonstrate an understanding of computer organization and architecture. Language/Theory Gloria Melara  Concepts of Prog. Languages  Automata, Languages & Comp.  Discrete Mathematics  Combinatorial Algorithms  Symbolic Logic  Plus Electives  Demonstrate an understanding of programming language concepts and knowledge of a variety of programming language paradigms.  Demonstrate an ability to apply mathematical skills appropriate to the computer science discipline. Software Engineering George Wang  Intro. to Software Engineering  Plus Electives  Demonstrate proficiency in collecting, analyzing, and interpreting data.  Demonstrate an understanding of emerging technologies and a working knowledge of currently available software tools.  Demonstrate an understanding of the principles and practices for software design and development.  Be able to apply the principles and practices for software design and development to real problems. Societal Issues Peter Gabrovsky  Societal Issues in Computing  Plus Electives  Demonstrate an awareness of the evolution and dynamic nature of the foundational core of computer science.  Demonstrate knowledge of the social impact of computing.  Demonstrate an understanding of the professional and ethical considerations of computing. Communications Diane Schwartz  Intro. to Software Engineering  Societal Issues in Computing  Plus Electives  Be able to effectively communicate orally.  Be able to effectively communicate in written form.  Be able to work effectively on a team. Lifelong Learning Jack Alanen  Societal Issues in Computing  Plus Electives  Demonstrate the knowledge and capabilities necessary for pursuing a professional career or graduate studies  Demonstrate the recognition of the need for, and ability for, continuing professional development. 18

The Annual Assessment Process at CSUN 1. Department forms Assessment Committee The program is divided into seven areas and a coordinator is chosen for each area. These coordinators constitute the Assessment Committee. 2. Assessment Committee recommends outcomes/objectives to be assessed The Assessment considers results from previous assessments, informal assessments, and the length of time since particular outcomes have been assessed to determine the current set to be assessed. 3. Department approves assessment goals and assessment plans are prepared After department approval the Assessment Committee decides on an assessment approach and develops a schedule of activities. 19

Student Learning Outcomes 1.Demonstrate an understanding of algorithms and data structures. 2.Demonstrate an understanding of computer organization and architecture. 3.Demonstrate an understanding of programming language concepts and knowledge of a variety of programming language paradigms. 4.Demonstrate proficiency in using a high-level computer language. 5.Demonstrate an ability to apply mathematical skills appropriate to the computer science discipline. 6.Demonstrate an awareness of the evolution and dynamic nature of the foundational core of computer science. 7.Demonstrate proficiency in collecting, analyzing, and interpreting data and information. 8.Demonstrate a problem solving ability. 9.Demonstrate an understanding of emerging technologies and a working knowledge of currently available software tools. 10.Demonstrate an understanding of the principles and practices for software design and development. 11.Be able to apply the principles and practices for software design and development to real world problems. 12.Be able to effectively communicate orally. 13.Be able to effectively communicate in written form. 14.Be able to work effectively on a team. 15.Demonstrate knowledge of the social impact of computing. 16.Demonstrate an understanding of the professional and ethical considerations of computing. 17.Demonstrate the knowledge and capabilities necessary for pursuing a professional career or graduate studies. 18.Recognize the need for, and show an ability for, continuing professional development. 20

Program Educational Objectives A few years after graduation, graduates of the computer science program will: 1.Be able to apply the principles of computer science, mathematics, and scientific investigation to solve real world problems appropriate to the discipline 2.Be able to apply current industry accepted computing practices and new and emerging technologies to analyze, design, implement, and verify high quality computer-based solutions to real world problems. 3.Exhibit teamwork and effective communication skills. 4.Be able to positively and appropriately apply knowledge of societal impacts of computing technologies in the course of career related activities. 5.Be successfully employed or accepted into a graduate program, and demonstrate a pursuit of lifelong learning. 21

The Annual Assessment Process at CSUN 1. Department forms Assessment Committee The program is divided into seven areas and a coordinator is chosen for each area. These coordinators constitute the Assessment Committee. 2. Assessment Committee recommends outcomes/objectives to be assessed The Assessment considers results from previous assessments, informal assessments, and the length of time since particular outcomes have been assessed to determine the current set to be assessed. 3. Department approves assessment goals and assessment plans are prepared After department approval the Assessment Committee decides on an assessment approach and develops a schedule of activities. 22

Sample Assessment Plan Learning Outcome: Be able to effectively communicate in written form Tool(s) to be Used in the Assessment: 1.The Writing Proficiency Exam (WPE) 2.Term Papers written in Comp 450 Assessment Activities to be Performed, When and by Whom: 1.Establish minimum and average WPE score values considered to be acceptable for each version of the WPE. (E.g., 3% better than the average scores over the past 5 years.) – May 2005, by the Software Engineering Course Group. 2.Collect the WPE scores for all Computer Science students over the last 5 years (noting which version was taken). – September 2005, obtained from Institutional Research by the Software Engineering Course Group. 3.Statistically analyze the WPE scores relative to past scores of similar students. – October 2005, by the Software Engineering Course Group. 4.Create a set of criteria (rubric) for assessing the quality of student written work, including the setting of standards for acceptability. – November 2005, by the Software Engineering Course Group. 5.Gather the term papers written by all students currently taking Comp 450. – December 2005, obtained from the Comp 450 instructors by the Software Engineering Course Group. 6.Select a sample of the term papers of graduating students and assess them according to the established criteria. – January 2006, by a committee selected by the department for the assessment of written communication. 7.Prepare an assessment summary report for the department including recommendations for program improvement. – February 2006, by the Assessment Committee. 23

The Annual Assessment Process at CSUN 4. Department approves assessment plans and assessments are conducted Program Area coordinators ensure that the planned assessment activities are completed and that a final report is prepared 5. Assessment are analyzed and program changes are recommended The Assessment Committee analyzes the results of the completed assessments and makes recommendations for program changes 6. Department reviews recommendations and makes program changes that are determined appropriate At a department meeting the recommendations of the Assessment Committee are discussed and proposed program changes are made only if there is approval from the department as a whole. 24

Sample Rubric Learning Outcome: Be able to effectively communicate in written form Assessment Rubric for Written CommunicationStrong 2 Acceptable 1 Weak 0 EFFECTIVENESS OF THE THESIS: Term papers written in an academic context are expected to contain a thoughtful and insightful thesis, main idea, position, or claim that is sustained throughout the paper. The thesis is clear, insightful and thought-provoking. It is sustained consistently throughout the paper. The thesis is clear and plausible. It is sustained consistently throughout the paper. The thesis is weak or absent. It is not sustained throughout the paper. RESPONSE TO ASSIGNMENT: Papers written in an academic context are expected to address the topic and issues set forth in the assignment and address all aspects of the writing task. Usually requires some discussion and refutation of an opposing view point. The paper responds to the assignment and addresses the topic and issues. Discussion of a counter- argument is included when appropriate. The paper responds to the assignment and addresses the topic and issues. Some discussion of a counter-argument is included when appropriate. The paper does not respond to the assignment or treats the assignment in a superficial, simplistic, or disjointed manner. Little or no discussion of a counter-argument in included. SUPPORT: Papers written in an academic context are expected to provide support for main points with reasons, explanations, and examples that are appropriate for intended audience. The thesis is fully and convincingly developed, supported with good reasons, explanations and examples. The thesis is adequately developed, supported with reasons, explanations, and examples. The thesis is inadequately developed, unsupported with reasons, explanations, and examples. ORGANIZATION: Papers written in an academic context are expected to be well-organized, in both overall structure & paragraphs. The paper is well-structured; its form contributes to its purpose. Paragraphs are well-organized and carefully linked to the thesis. The paper is generally well- structured, with only a few flaws in overall organization. Paragraphs are adequately organized and generally linked to the thesis. The paper is poorly structured; organizational flaws undermine its effectiveness. Paragraphs are not well organized; nor are they linked to the thesis. STYLE; Papers written in an academic context are expected to be stylistically effective – that is, to contain well- structured sentences, well-chosen words, and an appropriate tone, as a means of achieving its purpose. The sentence structure, word choice, fluency, and tone of the paper enhance its effectiveness and reinforce its purpose. The sentence structure, word choice, fluency, and tone of the paper contribute to its effectiveness and adequately support its purpose. The sentence structure, word choice, fluency, and tone of the paper detract from its effectiveness or are inappropriate to its purpose. GRAMMAR AND MECHANICS: Papers written in an academic context are expected to maintain sentence level correctness in terms of syntax, grammar, spelling, punctuation, and format. The paper is correct in terms of its syntax, grammar, spelling, punctuation, and format. Sentence level errors do not seriously detract from the paper’s effectiveness. Sentence level errors are so frequent and disruptive that they detract from the paper’s effectiveness. 25

The Annual Assessment Process at CSUN 4. Department approves assessment plans and assessments are conducted Program Area coordinators ensure that the planned assessment activities are completed and that a final report is prepared 5. Assessment are analyzed and program changes are recommended The Assessment Committee analyzes the results of the completed assessments and makes recommendations for program changes 6. Department reviews recommendations and makes program changes that are determined appropriate At a department meeting the recommendations of the Assessment Committee are discussed and proposed program changes are made only if there is approval from the department as a whole. 26

Sample Assessment Results Learning Outcome: Demonstrate an understanding of the principles and practices for software design and development 27

The Annual Assessment Process at CSUN 4. Department approves assessment plans and assessments are conducted Program Area coordinators ensure that the planned assessment activities are completed and that a final report is prepared 5. Assessment are analyzed and program changes are recommended The Assessment Committee analyzes the results of the completed assessments and makes recommendations for program changes 6. Department reviews recommendations and makes program changes that are determined appropriate At a department meeting the recommendations of the Assessment Committee are discussed and proposed program changes are made only if there is approval from the department as a whole. 28

The following recommendations for program improvement were made based on the results of the assessment: 1.Modify the course objectives for introductory computer science courses to include an introduction to software engineering concepts. STATUS: This was approved by the department and the members of the Software Engineering Course Group will provide information to the Fundamental Concepts Course Group to assist them in appropriately revising the course objectives for the introductory courses. 2.Modify the course objectives for elective courses with software engineering projects to include the reinforcement of software engineering concepts. STATUS: This was approved by the department and the members of the Software Engineering Course Group will provide information to those teaching elective courses to assist them in appropriately revising the course objectives to reinforce software engineering concepts. 3.Add a senior software engineering design project as a require for graduation. STATUS: After extensive discussions by the department regarding ways of implementing such a requirement, a committee was formed to develop a recommended approach and report back to the department for approval. Sample Program Improvements Learning Outcome: Demonstrate an understanding of the principles and practices for software design and development 29

Conclusions  The process has been used successfully for program assessment.  It has been embedded into the normal operations of the department and become accepted by faculty.  Since all faculty are involved, no one has become overburdened.  It has provided an effective means for directly assessing student learning with a focus on retention and transfer.  More attention is needed in the areas of validating results and ensuring the reliability of the assessment instruments used. 30