Automated Essay Scoring The IntelliMetric® Way Past, Present, Future The IntelliMetric® Way Paul Edelblut Vantage Learning Newtown, PA USA Copyright 2005-2008 Vantage Technologies Knowledge Assessment, L.L.C.

Brief History of Essay Scoring… AI Based Time Human level scoring 2000 Machine Learning NLP 1990 1980 Surface Features LSA 1970 1960 Complex Simple

NCLB – The Lightning Rod For Change Marketing Team

Why is IntelliMetric Important? Makes it feasible to use authentic open ended assessment Supports the basic premise that if you want to students to learn to write, have them write Provides immediate feedback Is more accurate than human scoring, particularly compared to local and single scorer models Eliminates reporting lag time Reduces assessment costs

Background of IntelliMetric™ Essay Scoring IntelliMetric has been used successfully for over 8 years to accurately score essays and, more recently, short-answer items as well Capable of marking papers written in English as well as many other languages Uses Artificial Intelligence to learn how to score essays Trained with a set of known papers Creates a unique solution for each prompt

IntelliMetric Approach Blend of proprietary Artificial Intelligence Techniques Learning Engine CogniSearch™;Quantum Reasoning™ Internalizing the pooled wisdom of many expert scorers Evaluates more than 450 Syntactic, semantic and discourse level features

Text Parser (Syntax analysis, Feature Extraction) Essay Files Prior Knowledge Base (16 million word Concept Net, 500,000 word vocabulary) Computational Analysis Text Preprocessor (Prepare text for processing) Judge 2 Judge 1 IntelliMetric ™ Final Score Judge N

Sample results of IntelliMetric™ scoring in global operations English Chinese (Mandarin) Bahasa Malay Hebrew Holistic D1 D2 D3 D4 Exact 86% 51% 46% 40% 44% 50% Adjacent 100% 43% 90% 92% 96% Discrepant 0% 6% 10% 8% 4% 14% Pearson .96 .65 .83 .81 .84 .86 .72

Conclusions and Discussion The multi-judge approach of IntelliMetric™ applies more accurate scoring than the Vantage Bayesian Statistical models “In part because individual judgment is not accurate enough or consistent enough, cognitive diversity is essential to good decision making.”—James Surowiecki, in The Wisdom of the Masses As found with expert human scoring, multiple people are more accurate than one Future research will include a comparison of IntelliMetric with other types of AES models

What Can’t IntelliMetric do? Score poetry, creative writing, humor, music Limited utility in a paper-based exam Replace humans entirely Create World Peace

Closing Thoughts What if the computerized scoring does not match the human scoring? Three hypotheses: a) the computer is wrong, b) the human is wrong or, c) they are both wrong. From our informal research reviewing mismatching papers, we have found that there is about an equal likelihood that the computer is wrong as the human is wrong, with a slight edge going in the computer’s favor.

“I write because I don’t know what I think until I read what I wrote.” Closing thought for the day “I write because I don’t know what I think until I read what I wrote.” --Flannery O’Connor American Essayist and Author

ã 2005 Vantage Technologies The information provided by Vantage Learning in this document/presentation is proprietary and confidential. This document or other information discussed in this meeting shall not be duplicated, used, disclosed or distributed in whole or in part to any third parties.