The use of a computerized automated feedback system Trevor Barker Dept. Computer Science.

Slides:



Advertisements
Similar presentations
Eli Collins-Brown, Ed.D. Illinois State University July 12, 2006 Aspects of Online Courses That Are More Effective and Successful than Traditional, Face-to-Face.
Advertisements

Blended Learning Curriculum Design – Transforming Culture Peter Bullen, Director of the Blended Learning Unit Mark Russell, Deputy Director of the Blended.
Centre for the Enhancement of Learning and Teaching Supporting & Enhancing Online Teaching & Learning by Catherine Ogilvie Centre for the Enhancement of.
Presentation to Faculty Board Valerie Cooper. Background §Students need to practice skill of movement analysis §No feedback available as each movement.
Successful online courses have…... Seven Principles for Good Practice Chickering, A. W. and Gamson, Z. F. (1987). "Seven Principles for Good Practice.
Directorate of Human Resources Successful online courses have…..
An Introduction to Computer- assisted Assessment Joanna Bull and Ian Hesketh CAA Centre Teaching and Learning Directorate.
Online Course Development Techniques and Best Practices Online Academic Services W.P. Carey School of Business.
You can use this presentation to: Gain an overall understanding of the purpose of the revised tool Learn about the changes that have been made Find advice.
Pre-registration training website The pre-registration portfolio is completed in an online capacity. This e- presentation will guide you though a series.
E-Portfolio July2014 Managing Multi-source Feedback.
Assessment matters: What guides might we use as individuals, teams and institutions to help our assessment endeavours? A presentation to Wolverhampton.
Aim to provide key guidance on assessment practice and translate this into writing assignments.
Camtasia Screen Capture Software Version 6 Dominic Bygate
The impact of case studies formatively and summatively assessed on students’ examination performance. Geeta Hitch (Senior Lecturer, Dept of Pharmacy) Janet.
ANY A HIGGINS Senior Lecturer Sport Studies USING FLIP CAMERA IN ASSESSMENT.
Four Core Technologies To Support Learning and Teaching Dominic Bygate Javiera Atenas Adriano Marinelli
ON-SCREEN TESTING on THE ISLE OF WIGHT The Power of Digital Assessment June 15 th 2005 – Extending the classroom walls.
How can you improve the quality of the feedback that you give to students and also save time in the process? *Colin Pitts, # Roger McNally, # Stephen Aylmer,
An Approach to Creating and Facilitating Workshops Library Instructor College Fall 2010 Charles Dershimer Erping Zhu Center For Research on Learning and.
© AJC /18 Extended Matching Sets Questions for Numeracy Assessments: A Case Study Alan J. Cann Department of Microbiology & Immunology University.
Integrating Educational Technology into Teaching Integrating Instructional Software into Teaching & Learning.
1 Abu Dhabi Women’s College Certified Designer Course.
Pre-registration training website The pre-registration e-portfolio will continue to be completed in an online capacity for the pre-registration.
Computerized Adaptive Testing: What is it and How Does it Work?
Electronic EDI e-EDI. The EDI has been in use since 1999 using a paper-based system and computerized spreadsheets to collect and manage EDI data. Over.
Evaluations and recommendations for a user support toolkit Christine Cahoon George Munroe.
SECTION ONE: PEDAGOGICAL ISSUES Let us start our discussion here by posing two related questions as follows. First, does the literature tell us that a.
ACE TESOL Diploma Program – London Language Institute OBJECTIVES You will understand: 1. The challenges of assessing student speaking ability. 2. Various.
Copyright Facilitate.com Beyond Web Conferencing: How To Create Value With Web Facilitation
Tami Eggleston Chickering, A., & Gamson, Z. (1987). Seven principles of good practice in undergraduate education. AAHE Bulletin, 39, 3-7.
ULTSEC Innovation Fund Ruth Valentine School of Dental Sciences Simon Cotterill Learning and Teaching Support Unit, Faculty of Medical Sciences.
By Richard Schutt For Colorado Christian University Management of Web Based Classes EDU543 Professor Andrew Roob.
Designing and Planning Technology Enhanced Instruction
Damien Raftery Lecturer & eLearning Development Officer Teaching & Learning Centre Institute of Technology Carlow Using Google Docs to support Project-based.
Internships: Benefits Through Distance Education Thomas A. Bodnovich Karen L. Duda.
Directorate of Human Resources Successful online courses have…..
E- ASSESSMENT Why use it.. whatever it is? Phil Davies School of Computing University of Glamorgan.
etools.massey.ac.nz Tools to help lecturers mark assignments John Milne Eva Heinrich.
Learner-Centered Teaching In Engineering/Technology Classrooms Steven A. Freeman Iowa State University Center for Excellence in Learning and Teaching.
Cooperative Language Learning (CLL) Collaborative Learning (CL)
THE GENERATION OF AUTOMATED STUDENT FEEDBACK FOR A COMPUTER-ADAPTIVE TEST University of Hertfordshire School of Computer Science Mariana Lilley Dr. Trevor.
Embedding information literacy in an undergraduate Management module: reflecting on students’ performance and attitudes over two academic years Clive Cochrane.
Using a VLE for Efficient and Effective Feedback Nick Lund Manchester Metropolitan University.
Blended Learning in Higher Education Workshop 3
A review of peer assessment tools. The benefits of peer assessment Peer assessment is a powerful teaching technique that provides benefits to learners,
August 2003 At A Glance VMOC-CE is an application framework that facilitates real- time, remote cooperative work among geographically dispersed mission.
SYSTEMS ANALYSIS AND DESIGN LIFE CYCLE
Based on Common Core.  South Carolina will discontinue PASS testing in the year This will be a bridge year for common core and state standards.
Instructional Design Strategies for Online Course Dr. Alisa Cooper Instructional Technologist & English Faculty Glendale Community College Online Course.
1 Using Feedback as a Teaching Tool in the Online Classroom.
Educator’s view of the assessment tool. Contents Getting started Getting around – creating assessments – assigning assessments – marking assessments Interpreting.
Mtivity Client Support System Quick start guide. Mtivity Client Support System We are very pleased to announce the launch of a new Client Support System.
1 Techniques for Online Retention Dr. Andrea Henne, Dean, Online and Distributed Learning.
State University of New York An Emerging Model for Online Learning MERLOT International Conference – August A Systemic Approach to Online Learning.
Programme design and student assessment David Baume 1.
Program Assessment User Session Experts (PAUSE) Information Sessions: RSS & Subscription Services October , 2006.
Oman College of Management and Technology Course – MM Topic 7 Production and Distribution of Multimedia Titles CS/MIS Department.
Self-assessing with adaptive exercises Chye-Foong Yong & Colin A Higgins University of Nottingham.
Keep Them Talking:Designing Effective Discussions Ferlin McGaskey, Ph.D., Postdoctoral Fellow, UT-Austin Center for Teaching and Learning
M253 Students Study Guide Mrs. Fatheya Al Mubarak – AOU Dammam.
Assessing Student Learning: Moving Beyond Objective Tests Susan Polich & Zachary Goodell.
Chapter 1 Assuming the Role of the Systems Analyst.
Best Practices in F2F Teaching Renee F. Aitken, Ph.D.
Online Quality Course Design vs. Quality Teaching:
Your mission, should you choose to accept it, is to provide as
Student Attitudes to Audio Feedback
Data Collection in MTM Choosing the right method for survey data collection.
Best Practices in Online Teaching
Assessing the value of rubrics in online assessment
Presentation transcript:

The use of a computerized automated feedback system Trevor Barker Dept. Computer Science

Contents Feedback considerations Approaches to feedback Automated feedback – Previous research Examples Discussion

Chickering and Gamson’s Seven Principles: Good practice in higher education… 1.Encourages contact between students and lecturers 2.Develops reciprocity and cooperation among students 3.Encourages active learning 4.Gives prompt feedback 5.Emphasises time on task 6.Communicates high expectations 7.Respects diverse talents and ways of learning

Gives prompt feedback Feedback must be prompt but it must also be good i.e. – Appropriate – Useful – Accurate – Individual – Fast – Facilitate feed forward

5 Reasons for automated approaches to testing and learning Vast investment in infrastructure Availability of MLE systems such as UH Studynet Changes in nature of Higher Education Online and distance education Increase in student numbers (SSR) Increasing pressures on time and cost

6 Previous research Computer-Adaptive Test Based on Item Response Theory (IRT) If a student answers a question correctly, the estimate of his/her ability is raised and a more difficult question is presented If a student answers a question incorrectly, the estimate of his/her ability is lowered and an easier question follows

7 Previous research Computer Adaptive Testing Computer-Based Tests (CBTs) mimic aspects of a paper-and-pencil test – Accuracy and speed of marking – Predefined set of questions presented to all participants and thus questions are not tailored for each individual student Computer-Adaptive Tests (CATs) mimic aspects of an oral interview – Accuracy and speed of marking – Questions are dynamically selected according to student performance

8 Benefits of the adaptive approach Questions that are too easy or too difficult are likely to – Be de-motivating – Provide little or no valuable information about student knowledge The CAT level identifies a unique boundary between what the student knows and what he or she does not know

9 Providing individual feedback based on CAT. An application of the CAT approach is in the provision of automated individual feedback This approach has been in operation for several years at the University of Hertfordshire in two BSc. Computer Science modules Recently this model has been extended to make it easier to use on other modules

10 About the Feedback Learners received feedback on: – Overall proficiency level; – Performance in each topic; – Recommended topics for revision – Cognitive level (Bloom) Feedback on assessment performance was initially made available to learners via a web- based application

11 Bloom’s taxonomy

Example questions 12

13

14 Performance Summary

15 Points for Revision

16 Results: tutors’ opinions Tutors consider that the fast feedback provided by a CAT is as good as or better than that currently provided in many cases. The link to Bloom’s levels was positive The approach was considered to be efficient, possibly freeing time for other activities CAT considered to be best as a formative tool, rather than for summative assessment Some tutors were concerned that the approach was ‘impersonal’ There is a need for a monitoring role for tutors, for practical and ethical reasons

Recent research The CAT automated feedback system has been extended from objective testing to include written and practical tests Testing and evaluation of the new system with approximately – 350 first yearBSc (1 final practical test), – 120 second year BSc(2 written and practical tests) and – 80 final year BSc (2 final practical tests) – 70 MSc students ( 2 written tests)

First prototype

Detailed marking scheme for one question showing feedback

Converted manually into simple database file

Output from system -

Later prototype

Final summary screen

Added features Markers able to comment on the completeness of the hand-in – In this version, the hand-in information is presented to the marker who may then make additional comments on the completeness or nature of the hand-in. Feedback was determined by the system based on the mark awarded in each section of a question, reading it from the database file for the assignment. After all the question sections had been marked, the system presented a final summary screen so that the marker could check that the marks had been awarded accurately. The marker can add additional feedback at the end

Results Student attitude to feedback was good irrespective of score on test – Useful – Fair – Convenient – Quantity – Quality Internal moderator happy with feedback Suggestions from moderator were included in the next prototype

Latest version

Modifications Easy to set up feedback database automatically Tutors can modify and add to feedback for each question Additions to feedback saved for re-use later

33 In summary Larger class sizes, greater use of online and distance assessment ensures that feedback is often too slow and too general to be of any real use to learners. Personalised automated feedback is likely to become increasingly important in the future. It is being used in four modules currently at UH. Learners and tutors accept the need for automated feedback and most appreciate the benefits of such systems. The system is being further developed to make it simpler for general use