Automatic Writing Evaluation

AWE Why might it be impossible? Why might it be possible?

AWE Why is it needed Use for testing and for feedback Shift from product to process approaches to writing Use of editing software (Word) and hence multiple drafts rather than typewriters (one or two drafts) Writing as a way to demonstrate proficiency Use for testing and for feedback

Warschauer & Page

Folz, Lochbaum & Rosenstein (Pearson)

What kinds of features potentially distinguish writing proficiency levels Ideas from previous classes and readings. ??

Samples Levels 1 to 5 Can you assess them correctly Are there features that can be used in an automatic system

New tools We will probably need to make use of a variety of tools and types of analysis Basic methodology create a large database of manually evaluated writing samples categorized by relevant features type of writing set prompts L1 (possibly) Discipline Concept – patterns can be extracted from large corpora (patterns may not be identifiable by humans)

General resources for language analysis Taggers (POS) Parsers/Tree taggers – (last week) Semantic analysis Wordnet Framenet Semantic tagging

Wordnet

Wordnet

Hypernym

Another search

Wordnet Here we checked the webpages Computational systems access Wordnet directly Use levels and “distances” to assess different kinds of semantic relatedness

Framenet

Search for give

Semantic Tags Different ways of attempting semantic tags Give “sense” definitions of a word - run a company run [10] – sense 10 in COBUILD dictionary Provide semantic type information

Semantic tags Similar to a thesaurus organisation

Semantic tagging

Semantic tag list

Semantic tags Examples doubted_A7- UNLIKELY vague_A4.2- GENERAL best_A5.1+++ policy_X7+ EVALUATION GOOD WANTED truth_A5.2+ EVALUATION TRUE lies_A5.2- EVALUATION FALSE

AWE Features Different kinds of features and methods are used in AWE Surface features: word structure, lexical items, POS, sentence structure (latter two require tagging and parsing and so not overtly “surface”) Typically scores are higher with more words longer words subordination passives nominalizations prepositions

More sophisticated analyses Cohesion analysis Latent Semantic Analysis (LSA) Move analysis

AWE features Cohesive features Crossley and McNamara 2012. J. of Research in Reading 35, 2, 115-135 Lexical diversity – more word “types” Word overlap – to increase cohesion Synonyms Connectives

Coh-Metrix

Coh-Metrix

Crossley and McNamara 2012

Latent Semantic Analysis Large corpora are used to assess meaning similarity among words or passages The meaning of a word relates to all the passages in which the word appears, which can be represented as a matrix matrix (vector) for bus will more similar to car than biscuit

Latent Semantic Analysis The meaning of a passage relates to all the words that appear in the passage These vectors are the input to the processing that is something like a factor analysis Number of dimensions are reduced to give a more abstract and deeper analysis of word-context relations (maybe similar to human cognition)

Latent Semantic Analysis Initial analysis -- LSA space created by instructional materials or something similar plus student essays Different methods Some essays are graded by instructors and test essays are compared using LSA with these Exemplary essay is written and closeness of the test essays are judged Distance between each sentence of a standard text and each sentence of the student essay. A cumulative score is given All the student essays are compared and then ranked

Latent Semantic Analysis Depth is based on word choice and content Not syntax etc Students choose from a pre-existing set of prompts Typically requires a set of texts against which the test essays are judged

Move analysis Elena Cotos project at Iowa State Database of articles in different disciplines Examined “moves” in different sections of papers Created Intelligent Academic Discourse Evaluator (IADE) which is being used across the university

What is Move? Move is a term used by Swales, referring to the function of each rhetorical unit (Swales 1990).

Using CorpusTools Annotate or tag a text using an annotation scheme you devise: Examples For Errors For Evaluative terms For Moves

Corpus and layers Layer Corpus

Create a tagging scheme

Corpus and layers

Assign Tags to each unit Move: describe characteristics of cases/informants

Search Instances Tag Instances

AWE For testing For feedback what are the aims of an AWE system what kind of feedback should be given

Warschauer & Page

E-rater Identifies discourse elements: background, thesis, main ideas, supporting ideas, and conclusion – leads to a “development” score and a length score (number of words) Lexical complexity type/token ratio vocabulary level average word length

E-rater Prompt-specific vocabulary – based on content analysis (vector analysis comparing the words of the essay with the reference corpus) Essay length (number of words)

Weighting features All the assessment procedures are based on a sets of features – average word length, pronoun use, passive, … Statistical analyses will allow a weighting of each feature to produce a system that gives the best results – the most accurate scoring of the essays Features for feedback purposes are likely to be different – (Write more!)

Intelligent Essay Assessor (Pearson)

Pearson System

Holt Online Essay Scoring

Move feedback IADE – Elena Cotos

Move-annotated corpus: 20 articles × 50 disciplines = 1000 IADE is based on a move-annotated corpus.

Main Functions of IADE Look for DEFINITIONS of each step Check STEP STATISTICS Search ANNOTATED CORPUS Get REVISION TIPS

DEFINITIONS of each step: Examples (left), definition (right)

STEP STATISTICS

ANNOTATED CORPUS

REVISION TIPS

Criterion A commercial AWE system from ETS http://www.youtube.com/watch?v=62mft8hulDQ

AWE What is writing? What is the relation of AWE and learning?