Project Funded by: E-Outcomes Assessment Project: Technology Linking Assessment & Learning Luvai Motiwalla, Steven Tello & Kathryn Carter College of Management UNIVERSITY OF MASSACHUSETTS LOWELL Presentation at The NERCOMP Conference Mar 20 th 2007, Worcester, MA.

Project Funded by: Agenda Assessment Challenge Project Overview Program-Level Assessment Process Systems Integration Progress to Date Future Directions

Project Funded by: The Assessment Challenge Increased calls for accountability in higher education  The Spelling Report, Secretary of Education calls for “evidence-based” assessment measures  Regional and Professional Accreditation agencies Widespread interest in higher-education institutions to define and assess student learning objectives No resolution on how to define, measure, and achieve quality learning Different approaches for outcomes assessment  student learning within courses and academic programs  student learning within specific fields or majors  Pre and Post learning assessments, surveys, etc.

Project Funded by: The Assessment Challenge What is Assessment? Many different perspectives  Administrator, Faculty, Student, Agency “Assessment is the systematic collection, review, and use of information about educational programs undertaken for the purpose of improving student learning and development.” (Palomba & Banta, 1999)

Project Funded by: The Assessment Challenge Problems with Assessment Tedious Costly Untimely Painful Useless A labor intensive, paper-based process driven by accreditation agency visits & timelines

Project Funded by: A Possible Solution… Design and develop an electronic outcomes assessment process which captures the course or student activities (e.g., assignments, discussions, or exams) within a learning management systems (LMS) categorize it relative to the college or university’s stated learning goals or outcomes.

Project Funded by: Project Overview Two-year, grant-funded project UMass Lowell & Davis Educational Foundation Use LMS to organize and report student progress toward program-level learning outcomes Interdisciplinary focus 6 Business Core Courses, 6 General Education Two operational foci Faculty Development Systems Development

Project Funded by: Project Overview - Goals Improve the learning environment for students by increasing student access to content, instructors and peers. Improve content and assessment consistency across multiple disciplines. Facilitate student development of academic e-portfolios. Reduce overall administrative costs through online paperless assessment, reporting, tracking.

Project Funded by: Assessment Process - Learning Outcomes University General Education Outcomes COM Learning Outcomes Breadth of Knowledge Critical Thinking Clear Communication Diversity Ethics Self-Direction/ Collaboration Information Literacy IIIIIIIVVVIVIIVIII Quantitative & Qualitative Skills Decision- making& Problem solving Communication Skills Diversity Impact Ethics & Values Team Leadership & Membership Information Communication Technologies Global Impact

Project Funded by: Assessment Process – Self Reporting Current Process Manual, Paper-based, Self-report

Project Funded by: GLOBAL IMPACTS DIVERSITY IMPACT ETHICS IT USAGE TEAM PARTICIPATION QUANTITATIVE SKILLS COMMUNICATION SKILLS Course A Course B Course C Course N Discussion Topic A Assignment 1 Exam 1 & 3 Assignment 2 Group Tool Usage Group Project Excel Assignment Case Assignment Assessment Process: New Approach Proposed Process Automated, Menu-driven, Drill down

Project Funded by: Assessment Process: Future Reporting Possibilities

Project Funded by: Systems Integration Use LMS (WebCT Vista) as Assessment Collection Tool  Organize & present course assessment activities  Present outcomes to students  Collect assessment artifacts System limitations require external database  To map course activities to program-level outcomes  To organize & report progress toward outcomes

Project Funded by: Assessment Tool Syllabus Tool Discussion Tool Assignment Tool Grade Book Course Content Files Groups Chat Tool Systems Integration Use the activity tools in WebCT Vista Train faculty in specific use of these tools

Project Funded by: Systems Integration Use PowerSight to report off Grade book Train faculty to use grade book

Project Funded by: Systems Integration EOutcomes Database  My SQL, Perl, Web Interface  Export CSV from WebCT, Import to EOutcomes Faculty map outcomes to specific course activities in EOutcomes Database Administrators then report on progress toward outcomes through EOutcomes

Project Funded by: EOutcomes System V1 Systems Integration

Project Funded by: Learning Outcomes Menu Systems Integration

Project Funded by: Progress to Date – Year 1 Conducted Summer Training Workshop Seven courses offered in Fall 2006 / Spring 2007 Developed a prototype EOutcomes database and application

Project Funded by: Goal 1: Improve learning environment by increasing student access… 7 Courses offered Fall2006, 385 students Each Web-Enhanced to varying degrees.

Project Funded by: Goal 1: Improve learning environment by increasing student access…

Project Funded by: Goal 2: Improve content and assessment consistency across multiple disciplines EOutcomes Training Institute, June 2006 (June, 2007 as well)  Learning Outcomes  Assessment Matrix in Excel  Assessment Tools in WebCT  Reporting Tools in WebCT  Respondus Exam Tool  EOutcomes Database  Tablet PC Demo Training Materials online   EOutcomes WebCT Course

Project Funded by: Goal 3: Facilitate student development of web- based academic portfolios. ePortfolios provide ability to link sample student work to activity to learning outcomes  Very limited success due to WebCT issues and manual nature of effort  ePortfolios often focused on student need to build a portfolio, rather than administrative need to measure achievement  Continue to review possible tools within WebCT and external  Possibly EOutcomes Database

Project Funded by: Goal 4: Reduce overall administrative costs Possible savings include:  Supply savings – less scantron sheets and paper for exams  Proctor savings – less time spent scanning, reviewing and reporting exam grades (less room for error)  Faculty time savings when: grading exams, many are auto-scored grading quizzes, many are auto-scored filling out assessment spreadsheets collecting student work samples making copies  Savings are balanced by initial development effort on part of faculty, and maintenance

Project Funded by: Emerging Research Questions Does the use of a LMS in a web-enhanced course increase student awareness of program-level learning outcomes? Do students perceive a benefit in the use of a LMS for course-level assessment activities? Do faculty perceive a benefit in the use of a LMS for course-level assessment activities? Does the use of a LMS facilitate the organization and reporting of program-level assessment data for accreditation purposes? Does the LMS in a web-enhanced course have any positive impact on students’ involvement in a course or satisfaction with the course or their learning?

Project Funded by: Next Steps Develop EOutcomes Reporting System tools Incorporate six more courses in academic year Conduct training institutes in Summer 2007 Measure Project Achievements Integrate Research Agenda

Project Funded by: Thank you Prof. Luvai Motiwalla Prof. Steven Tello EOutcomes Website