1. Automated Essay Scoring The IntelliMetric® Way
Past, Present, Future
The IntelliMetric® Way
Paul Edelblut
Vantage Learning
Newtown, PA
USA
Copyright Vantage Technologies Knowledge Assessment, L.L.C.

2. Brief History of Essay Scoring…AI
Based
Time
Human level scoring
2000
Machine
Learning
NLP
1990
1980
Surface
Features
LSA
1970
1960
Complex
Simple

3. NCLB – The Lightning Rod For Change
Marketing Team

4. Why is IntelliMetric Important?
Makes it feasible to use authentic open ended assessment
Supports the basic premise that if you want to students to learn to write, have them write
Provides immediate feedback
Is more accurate than human scoring, particularly compared to local and single scorer models
Eliminates reporting lag time
Reduces assessment costs

5. Background of IntelliMetric™ Essay Scoring
IntelliMetric has been used successfully for over 8 years to accurately score essays and, more recently, short-answer items as well
Capable of marking papers written in English as well as many other languages
Uses Artificial Intelligence to learn how to score essays
Trained with a set of known papers
Creates a unique solution for each prompt

6. IntelliMetric Approach
Blend of proprietary Artificial Intelligence Techniques
Learning Engine
CogniSearch™;Quantum Reasoning™
Internalizing the pooled wisdom of many expert scorers
Evaluates more than 450 Syntactic, semantic and discourse level features

7. Text Parser
(Syntax analysis,
Feature Extraction)
Essay
Files
Prior Knowledge
Base
(16 million word Concept
Net, 500,000 word
vocabulary)
Computational
Analysis
Text
Preprocessor
(Prepare text for
processing)
Judge 2
Judge 1
IntelliMetric ™
Final
Score
Judge N

8. Sample results of IntelliMetric™ scoring in global operations
English
Chinese (Mandarin)
Bahasa Malay
Hebrew
Holistic D1 D2 D3 D4
Exact
86%
51%
46%
40%
44%
50%
Adjacent
100%
43%
90%
92%
96%
Discrepant 0% 6%
10% 8% 4%
14%
Pearson
.96
.65
.83
.81
.84
.86
.72

9. Conclusions and Discussion
The multi-judge approach of IntelliMetric™ applies more accurate scoring than the Vantage Bayesian Statistical models
“In part because individual judgment is not accurate enough or consistent enough, cognitive diversity is essential to good decision making.”—James Surowiecki, in The Wisdom of the Masses
As found with expert human scoring, multiple people are more accurate than one
Future research will include a comparison of IntelliMetric with other types of AES models

10. What Can’t IntelliMetric do?
Score poetry, creative writing, humor, music
Limited utility in a paper-based exam
Replace humans entirely
Create World Peace

11. Closing Thoughts
What if the computerized scoring does not match the human scoring?
Three hypotheses: a) the computer is wrong, b) the human is wrong or, c) they are both wrong.
From our informal research reviewing mismatching papers, we have found that there is about an equal likelihood that the computer is wrong as the human is wrong, with a slight edge going in the computer’s favor.

12. “I write because I don’t know what I think until I read what I wrote.”
Closing thought for the day
“I write because I don’t know what I think until I read what I wrote.”
--Flannery O’Connor
American Essayist and Author

13. ã 2005 Vantage Technologies The information provided by Vantage Learning in this document/presentation is proprietary and confidential. This document or other information discussed in this meeting shall not be duplicated, used, disclosed or distributed in whole or in part to any third parties.