You can play/download the audio file from here. here Please, play the audio track at the same time you view the next slide in the Slide Show for optimal synchronization.

Technology and Writing Instruction Effect of New technologies on the nature of writing itself, the writing process, and the norms and standards for written texts: Effects of personal computers and the World Wide Web on writing instruction through: ONLINE CHAT ROOMS DISCUSSION FORUMS TEXTING & MICROBLOGGING WORD PROCESSORS ONLINE DICTIONARIES BLOGSWIKIS

Effective Feedback EFFECTIVE FEEDBACK is Timely Fair Detailed Motivating Frequent

Challenges of Providing Effective Feedback Time-consuming Teacher overburden Impossible in large classes Teacher-Centered Feedback Forms Only Possible for Skilled, Well-Trained Teachers Raters bias

Automated Writing Evaluation (AWE) DEFINITIONDEFINITION The use of computer programs to evaluate the quality of an essay by providing a single score, a detailed evaluation of essay features, or both. THE FIRST AWE SYSTEM Project Essay Grade™ (PEG), developed in the US (1966) to be used in large-scale high school essay assessments THE BIGGEST SHORTCOMING OF EARLY AWE SYSTEMS Focus on the surface features of the essay and ignoring content (a threat to the construct validity of AWE assessment)

Automated Writing Evaluation (AWE) CURRENT AWE SYSTEMS - They are based on Natural Language Processing (NLP). - NLP uses Latent Semantic Analysis (LSA) to evaluate the semantic content of words used in essays, beside statistical analysis of surface variables. Today, the major AWE systems come with commercial products that provide automated feedback to writing. COMMERCIAL AWE PROGRAMS My Access!® Criterion® Holt Essay Scoring Developed by Vantage Learning Developed by Educational Testing Service (ETS) Developed by Pearson Knowledge Technologies

My Access!® : the AWE Program Used Provision of immediate feedback, along with holistic and analytical scoring Offering over 700 unique prompts for different levels and in various genres Offering instructional writing tools: progress- tracking, pre-writing activities, a spell-checker, and a word bank Separate evaluation of 5 traits of writing: Content, Organization, Mechanics, Focus, and Style $ 99 per year

My Access!® Screenshots MY EDITOR highlights the errors, explains them, and suggests how to correct them The Writing Environment

My Access!® Screenshots MY TUTOR Provides The Generic Feedback on Each Trait The Performance Report

The Advantages of AWE Immediate, individualized feedback Frequent scoring of draftsPromotion of process writing Promotion of formative assessment Promotion of autonomous learning Freeing up the teachers’ time

The Disadvantages of AWE Elimination of human interaction in writing assessment Discrimination against students who are less familiar with using technology The risk of students’ assuming their writing being unimportant as their audience is replaced by a machine Limiting teacher’s range of topic choice and stifling teacher’s creativity Limited capability in evaluating the content of essays Insensitivity to the aesthetics of the written text and creative writing

Research on AWE Independent Research Developer- Funded Research Psychometric Research on Native Speakers Computer- Human Score Agreement Construct Validity Cmpt-Cmpt & Human-Human Score Agreement Classroom Research Research on L2 Learners Independent Classroom Research on EFL learners has resulted in conflicting findings.

Research Questions ? Does the employment of AWE and human scoring of essays produce significantly different effects on the improvement of EFL learners’ writing skill? What are the student-writers’ attitude toward using an AWE program as the scorer of their essays? 1 2

Participants 22 NATIONALITY Iranian AGE/SEX/OCCUPATION 20 to 37/M&F/College Students PROFICIENCY LEVEL Advanced COMPUTER SKILLS Average SETTING/DURATION Essay Writing Classes /10 Ws SELECTION Intact Classes They were assigned to TWO groups THE EXPERIMENTAL GROUP Using My Access!® as the scorer of their essays and the main source of feedback 11 THE CONTROL GROUP Using a teacher scored for similar purposes.

Instrumentation An Essay Writing Pretest An Essay Writing Posttest A Delayed, Timed Writing Posttest A 20-Item AWE Attitude Questionnaire Vantage 6-point Independent Writing Rubric

Materials 5 Seven Essay Writing Prompts A Teacher Who Taught Both Groups and Scored the Control Group’s Essays The AWE Program My Access! ® A My Access! ® User Guide, along with Individual Training Sessions The Text Book “College Writing Skills” by John Langan (2004)

Procedure Providing the students with an illustrated user guide Holding individual training sessions (30 min) STEP 1 Administrating the Pre-test STEP 2 An essay writing pre-test on a topic chosen from My Access!® Topic Bank STEP 3 The Treatment: the Classroom Phase Same instruction, including the same teacher, materials, teaching approach, and classroom routine was used for both groups. Each week, all the students were given a topic to write an essay on.

Procedure (cont.) STEP 3 The Treatment: the My Access!® Phase Provision of My Access login information on a weekly basis Allocating 90 minutes for essay submission and revision Administering the Post-test An essay writing post-test similar to the pre-test STEP 4 18 Likert-scale items Administering the AWE-Attitude Questionnaire STEP 5 2 open-ended items A paper-pencil timed essay writing post-test administered over a three-week interval Administering the Delayed, Timed Post-test STEP 6

The Tests’ Results p= 0.86 PRETEST Independent-Samples t-test t(20)=.179, p=0.86 (two-tailed) POSTTEST Independent-Samples t-test t(20)=4.64, p=.000 (two-tailed) * p=. 000 p= 0.40 POSTTEST’S LENGTHS Independent-Samples t-testt (15) =.866, p=0.40 (two-tailed) DELAYED, TIMED POSTTEST Independent-Samples t-test t(22)=.04, p=.96 (two-tailed) Inter-rater Reliability 0.79 p= 0.96 p= 0.86 PRETEST Independent-Samples t-test t(20)=.179, p=0.86 (two-tailed) POSTTEST Independent-Samples t-test t(20)=4.64, p=.000 (two-tailed) * p=. 000 p= 0.40 POSTTEST’S LENGTHS Independent-Samples t-testt (15) =.866, p=0.40 (two-tailed) DELAYED, TIMED POSTTEST Independent-Samples t-test t(22)=.04, p=.96 (two-tailed) Inter-rater Reliability 0.79 p= 0.96

AWE Questionnaire’s Results PRACTICALITY Ninety one % Ninety one % considered AWE quite practical. NUMERICAL GRADING 64% 64% could not decide if the scores were fair. EFFECT ON PROGRESS 64% 64% believed AWE boosted their confidence and improved their writing. FEEDBACK SATISFACTION 73% 73% thought AWE feedback had good suggestions. FEEDBACK VALUE 73% 73% said they read AWE feedback, but could not decide if it was valuable. REVISION BEHAVIOUR 73% 73% said they adopted the feedback for revision. 55% 55% said they did more revisions when using AWE.

Open-Ended Items’ Results RANK RESPONSEFREQ. THE BEST 1 Immediate scoring and feedback 82% 2 The possibility to submit multiple drafts of an essay to receive new scores 45% 3 Boosting self-confidence in EFL writing 27% THE WORST 1 Lack of a personal account to spend more time using the program at desired times 27% Misdetection of errors by My Editor 27% Requiring high-speed Internet connection 27% 2 Being stressful in the initial weeks 9% Unclear rule explanations by My Editor 9%

What We Found L2 learners who used My Access!® outperformed those who receive scores and feedback from a teacher. 1 1 The first finding was not repeated in a delayed paper-pencil test situation. 2 2 This implies that AWE’s environment is central to the students’ improved performance and casts doubt on AWE’s long-term effects. My Access!® users revised their drafts more frequently in response to the program’s immediate editorial feedback. 3 3

What We Found However, their revisions were partial and limited to the sentence-level superficial features. more writingrevising the content and organization Thus, it never led to more writing or revising the content and organization. 4 4 The most positive effects of AWE were: 1) boosting the students’ writing confidence, 2) involving them in the writing process, 3) motivating them to submit more drafts, 4) helping struggling writers to reduce their mechanical errors. 5 5

What We Found 6 6 AWE did not save teacher time. As the teacher still had to read every essay and give oral feedback to the whole class, as well as provide them with accounts every week. 7 7 Some of the factors discouraging the students from using AWE were: 1) low-speed connection, 2) frequent error misdetections, 3) lack of personal accounts, 4) lack of sufficient experience with the program.

Pedagogical Implications AWE for Encouraging more revisions and reducing mechanical errors of EFL writers AWE for prep. courses for computerized proficiency tests AWE for motivating struggling students by allowing them to learn at their own pace 4 AWE as a supplement, not a replacement AWE systems be enhanced to provide comprehensive, essay-specific feedback 6 Administrators should be informed about the types of AWE, and ensure the availability of their infrastructural requirements

The Bottom Line blessingtorture So, is the use of AWE to score, and provide feedback to, EFL writers’ essays a blessing or a torture? AWE is helpful in boosting learners’ writing confidence and motivation. AWE is effective in reducing mechanical and sentence-level errors. o Contrary to its developers’ claim, AWE did not lead to longer essays and did not save teacher’s time and energy. o AWE’s effects on writing development in test situations and outside its environment are questionable. o AWE might not safely replace the teacher.