2004 Southern Business Administration Association Meeting, Myrtle Beach, South Carolina The UCF College of Business Approach to Assessing Student Learning Taylor Ellis, Ph.D.Patrick Moskal, Ph.D. Associate Dean Director, Testing & Evaluation University of Central Florida

Presentation Outline 1.UCF background. 2.Overview of the assessment process in the College of Business Administration. 3.Development of student learning outcomes. 4.Description and role of our Testing Lab in the assessment process. The UCF College of Business Approach to Assessing Student Learning

UCF Background  Large metropolitan university with over 44,000 students.  College of Business Administration (CBA) offers 7 undergraduate, 10 Masters, and 5 Doctoral degree programs.  ~140 full-time and 60 part-time faculty in CBA.  ~8,000 students enrolled in CBA each year. The UCF College of Business Approach to Assessing Student Learning

Issues Faced in UCF CBA  Number of students.  Required courses in core.  Pre-requisites in majors.  Computer Testing Lab. The UCF College of Business Approach to Assessing Student Learning

CBA Assessment Process The UCF College of Business Approach to Assessing Student Learning

CBA Assessment Process (continued) The UCF College of Business Approach to Assessing Student Learning

Assessment Process CBA Departmental Review Committee (DRC)  DRC consists of CBA Department Chairs.  DRC Chair participates on the UAC and reports CBA assessment results.  Supported by Director of Testing and Evaluation. The UCF College of Business Approach to Assessing Student Learning

Assessment Process CBA Teaching and Learning Committees  Selected for each degree program area.  Establish student learning outcomes (SLOs) and ensure data collection.  Participate in an annual review of assessment results, and plans for upcoming year. The UCF College of Business Approach to Assessing Student Learning

Assessment Process  Each degree program will develop 4-10 general learning objectives (expectations).  Each objective will be evaluated by multiple outcome measures (exam questions or assessment types).  Summative scales will be developed to measure most learning outcomes.  Allows for evaluation in multiple classes.  Allows for pinpointing specific weak areas. The UCF College of Business Approach to Assessing Student Learning

Assessment Process Degree Program Learning Expectations Expectation 1 Class 1 Affective 1 Questions & Beliefs 1-? Collaboration score Cognitive 1 Questions 1-? Projects Behavior 1 Questions & Projects 1-? Class 2Class 3, etc. Expectation 2 Expectation 3, etc. The UCF College of Business Approach to Assessing Student Learning

Assessment Goals  Establish student learning outcomes for each degree program, based on the CBA Mission.  Acquire data that provide direction for program improvement and meet both AACSB and regional accreditation requirements.  Develop comprehensive data collection and reporting system.  Collect multiple measures over time (data triangulation).  Utilize existing data where possible.  Cause minimal intrusion on normal faculty workload. The UCF College of Business Approach to Assessing Student Learning

Why Do We Need Student Learning Outcomes?  Provide feedback to students about their progress.  Communicate instructional goals to stakeholders (e.g., trustees, community, university administration, faculty, accrediting agencies, parents).  Provide the means for assessing student learning. The UCF College of Business Approach to Assessing Student Learning

Learning Objectives Begin by developing a list of goals and objectives, such as the following “essential competencies” for all UCF Business graduates. BE2010 Competencies:  Teamwork  Communication  Creative Thinking  Adapting to change  Diversity  Ethics The UCF College of Business Approach to Assessing Student Learning

Learning Objectives From our goals, we develop specific learning objectives. An objective is defined as a quantifiable and/or observable achievement accomplished under specifiable conditions.  How do we develop objectives? Start by answering basic questions like:  What do we want our students to know when they finish our course or program?  What should our students be able to do when they finish our course or program? The UCF College of Business Approach to Assessing Student Learning

Objectives Vs. Outcomes  In general, an objective is what we hope that we will see, and an outcome is what we did see.  Learning objectives define the outcomes of instruction in clear terms.  Outcomes must be both observable and measurable.  All objectives/outcomes should include three primary components: the action or behavior to be measured, conditions of learning, and standards of performance. The UCF College of Business Approach to Assessing Student Learning

You Want SMART Objectives  Specific (Clear and definable terms describing the knowledge, skills, and attitudes)  Measurable (Reliable data that can be assessed in multiple ways)  Attainable (Feasible to be accurate; need not be perfect)  Realistic (Expected standards must be practical)  Time-bound (When will results will be observed?) The UCF College of Business Approach to Assessing Student Learning

Types and levels of student learning outcomes  Cognitive: intellectual skills.  Knowledge, comprehension, application, analysis, synthesis, evaluation (from Bloom, 1956).  Affective: what a student believes (values).  Behavioral: psychomotor skills. The UCF College of Business Approach to Assessing Student Learning

General Knowledge and Skills Examples  Communication  Organizational  Quantitative  Critical Thinking  Interpersonal Skills and Awareness  Group Dynamics  Motivational and Leadership  Ability to Learn and Train Others  Ability to Manage Information Technology  Global Awareness  Understanding Issues of Diversity  Ethical Awareness The UCF College of Business Approach to Assessing Student Learning

CBA Assessment Types  Course embedded assessments of individual and group knowledge and skills (e.g., exams, projects).  Program level exams (e.g., doctoral candidacy exams).  Certification and standardized exams.  Direct observation.  Teamwork.  Student portfolios.  Internships and service learning.  Student records.  College, AACSB, and employer surveys.  Placement/employer data. The UCF College of Business Approach to Assessing Student Learning

CBA Accreditation Database Development Plan

CBA Assessment Database  Maintain results for each student.  Prefer automated input from electronic testing.  Allow input from additional sources.  Allow data analyses.  Create reports specific to each degree program and for diverse stakeholders. The UCF College of Business Approach to Assessing Student Learning

CBA Testing Laboratory  140 networked PCs  Positioned to maintain exam security.  Netsupport School Pro software to manage PCs (e.g., monitor, block unapproved access).  WebCT and InQsit 1 packages available for exams.  Online student exam scheduling. 1 Developed at Ball State University. The UCF College of Business Approach to Assessing Student Learning

CBA Testing Laboratory The UCF College of Business Approach to Assessing Student Learning

CBA Testing Laboratory  Over 7,500 exams given in fall  Over 12,500 exams given in spring  Goal is to automatically extract specified program evaluation and accreditation data from course exams. The UCF College of Business Approach to Assessing Student Learning

Testing Software Features  InQsit: Easy to use and generate tests (Word, text files, Publisher’s test banks).  WebCT: Complete course management system.  Test Question Types: T/F, multiple choice, multiple response, short answer, matching, identification of objects on chart/photo (InQsit), limited essay grading (InQsit), ungraded essay, survey. The UCF College of Business Approach to Assessing Student Learning

Testing Software Features (continued)  Random selection of questions from a question pool.  Randomization of questions and answer choices.  Adaptive scoring.  Use of audio and video in tests.  Start and stop times may be set by date and time.  Analysis of Results: automatic student grading and item analysis. The UCF College of Business Approach to Assessing Student Learning

Testing Lab Components Under Development  Lab entrance will have student ID reader. Students swipe card or enter PID for computer verification.  Proctor software will allow the faculty/proctor PC to display:  Student name and photo.  Test the student is authorized to take.  Computer assigned for exam.  Seat reservation duration. The UCF College of Business Approach to Assessing Student Learning

Questions? The UCF College of Business Approach to Assessing Student Learning

For more information, contact: Patrick J. Moskal, Ph.D. University of Central Florida College of Business Administration Undergraduate Student Services, BA-II, Room 101G Orlando, Florida The UCF College of Business Approach to Assessing Student Learning