The Development of a Comprehensive Assessment Plan: One Campus’ Experience Bruce White ISECON 2007

Feedback / accountability 1. “For society to work […] we must be accountable for what we do and what we say.” 2. “No person can succeed unless he or she is held accountable” 3. “Feedback is the breakfast of champions” 4. “You need a culture of assessment, not a climate” 1 -Betty Dowdell 2 – Grant Wiggins 3 – Ken Blanchard 4 – Gloria Rogers

Overview Are we teaching what we say we are? Are students learning? How can we be more effective in our instruction? SO … What do we want students to learn? Why do we want them to learn it? How can we help them to learn it? How do we know what they have learned?

Furthermore...  Stakeholders want to see if we are accomplishing our goals of education.  Possible stakeholders:  Students  Parents  Employers  Board of Regents / State Agencies  Accrediting groups (AACSB / ABET / etc.)  Faculty  Alumni

Our campus program  The Information Systems Management program at Quinnipiac University in Hamden Connecticut started our journey towards a comprehensive assessment program in  Prior ‘assessment’ was informal:  So ISM faculty – how do you think we are doing?  So Advisory Board – what advise do you have for us?  So IS education community – what should we teach (like IS2002 curriculum)  So Employers – what do our students need to know (or know better)  Etc.

Our desired outcomes:  Analysis and design of information systems which meet enterprise needs.  Use and experience with multiple design methodologies.  Experience in the use of multiple programming languages.  Development of hardware, software and networking skills.  Understanding of data management.  Understand the role of IS in Organizations.

Possible assessment methods: Direct Assessment Methods :  Simulations  Behavioral Observations  Performance Appraisals  Locally Developed Exams  External Examiner  Portfolios / E-portfolios  Oral exams  Standardized Exams (Source: Gloria Rogers- ABET Community Matters 8-06)

Indirect Assessment Methods  Exit and other interviews  Archival data  Focus groups  Written or electronic surveys / questionnaires  Senior exit surveys  Alumni surveys  Employer surveys  Other factors:  IS model curriculum  Advisory board  Alumni Surveys

Foundation of Our Assessment Program We got interested in the CCER IS Assessment test early It is a direct assessment test based on the IS2002 model curriculum It has been thoroughly tested and analyzed It has been shown to be valid and reliable Test scores reported on 37 different areas – and relevant to our learning outcomes The test questions are written at higher levels of Bloom’s taxonomy – with scenarios

More on our assessment process  We also use a senior exit survey (indirect measure)  An advisory board gives input  Informal controls:  Campus decisions (such as number of credits allowed, changes in general education)  Model Curriculum Changes  Employer input  Conferences / technologies

Specific Learning Skills Skill Set 3.0 Strategic Org. Systems Develop Avg 3.1 Organizational Systems Development Strategic Utilization of Information Technology IS Planning IT and Org. Systems Information Systems Analysis & Design Decision Making Systems Concepts, Use of IT, Customer Service Systems Theory and Quality Concepts

3.2 Project Management Avg Team Leading, Project Goal Setting Monitor and Direct Resources and Activities Coordinate Life Cycle Scheduling and Planning Apply concepts of continuous improvement Project Schedule and Tracking Continued

Overall Analysis Area Avg Hardware and Software Modern Programming Language Data Management Networking and Telecommunications Analysis and Design Role of IS in Organizations Number taking test Overall

Senior Exit Survey Learning Objective 2006 scores 2007 scores Systems Analysis (including project management Alternative Design Methodologies Programming Languages Hardware and Software Networking Data management 4.2 IS in Organizations Ethics in IS / IT Global aspects of IS / IT

Next Step – setting metrics  So … we have a solid direct measurement  And … we have a good indirect measure  Now … what  We are working on setting metrics (especially for our direct measurement – the CCER IS Assessment test)  From ABET:  “Every student does not have to achieve the desired outcomes, but targets must be defined.”

Setting expectations  The faculty have considered the goals of the program and feel our program emphasizes systems analysis, the role of IS in organizations and data management ISECON Accreditation16 AreaExpectation Systems Analysis80% will score 50 or higher Programming60% will score 45 or higher Data Mgmt70% will score 50 or higher Role of IS in Org.80% will score 50 or higher Networking70% will score 45 or higher Hardware / Software70% will score 45 or higher

Now what  Next year as students take the CCER IS Assessment test and as we get feedback from our senior survey, we will analyze the data to see if our outcomes have been reached.  If they have:  If they haven’t:  We analyze why not – was it poor instruction? Poor students? Poor textbook? Overly optimistic expectations?  We change in an effort to ‘constantly and continually improve’ our program!!