Standardized Assessment Presentation TOEFL Assessment Analysis By: Raneen Elbakry FLT 808 Assessment in Foreign Language Teaching Spring 2016

What is a standardized test? A standardized test is any form of test that: (1) Requires all test takers to answer the same questions, or a selection of questions from common bank of questions, in the same way. (2) Is scored in a “standard” or consistent manner, which makes it possible to compare the relative performance of individual.

What is the TOEFL? It is a standardized test that evaluate the potential success of an individual whose native language is not English, to use and understand standard American English at a college level.  No matter where in the world you want to study, the TOEFL test can help get you there. More than 9,000 colleges, universities (including the top colleges and universities) in the U.S., Canada, U.K., Australia, New Zealand as well as licensing agencies in 130 countries accept TOEFL scores TOEFL is produced and administrated by ETS ( Educational Testing Services) that is based in NJ, United States.

What does it measure ? The TOEFL test measures the ability of non- native English speakers to use and understand the English language as it is heard, spoken, read and written in the university classroom. TOEFL is not tied to any curricula or text book and tests all four skills; reading, writing, speaking and listening.

What it is the purpose of TOEFL ? To quote the original TOEFL® framework document (Jamieson, Jones, Kirsch, Mosenthal, & Taylor, 2000, pp.10–11): “The purpose of the … test will be to measure the communicative language ability of people whose first language is not English … The test will measure examinees’ English-language proficiency in situations and tasks reflective of university life … ” where instruction is conducted in English. The purpose of the TOEFL can be: Proficiency or Admissions. It is the most highly respected around the world and it’s the most widely accepted as well. More than 9000 colleges, agencies and other institutions in over 130 countries accepted TOEFL scores.

How TOEFL is conducted? Discrete Point: refers to testing one language element at a time ( Writing , reading , listening or speaking ). such as T/F, MC and fill in the blank. Integrate Questions: refers to testing that requires the test takers to combine more than one language element / skill to complete cretin task such as writing and speaking tasks.  

TOEFL iBT® Test Structure / Sections Time Limit Question Tasks Reading 60-80 m 36 – 56 Read 3 or 4 passages from academic texts and answer questions. Listening 60-90 m 34 – 51 Listen to lectures, classroom discussions and conversations, then answer questions Break 10 m ----- Speaking 20 m 6 tasks Express an opinion on a familiar topic; speak based on reading and listening tasks. Writing 50 m 2 tasks Write essay responses based on reading and listening tasks; support an opinion in writing. https://www.ets.org/Media/Tests/TOEIC/pdf/TOEIC_LR_sample_tests.pdf https://www.ets.org/Media/Tests/TOEFL/pdf/SampleQuestions.pdf

TOEFL iBT® Test Sections Listening Length: 60–90 minutes 4–6 lectures, each 3–5 minutes long with six questions a piece , 2–3 conversations, each 3 minutes long with five questions apiece Tip: Introduces more than one native English accent Scoring: 0–30 points Speaking Length: 20 minutes 2 independent tasks: speak about a familiar topic 4 integrated tasks: speak based on what you read and hear Tip: You have 30 seconds to prepare and one minute to respond Scoring: 0–4 points, converted to a 0–30 score scale

TOEFL iBT® Test Sections [continued] Reading Length: 60–100 minutes 3–5 passages from academic texts, each about 700 words long 12–14 questions per passage Tip: Includes a glossary to define key words Scoring: 0–30 points Writing Length: 50 minutes 1 integrated task: write based on what is read and heard 1 independent task: support an opinion Tip: Typing is required Scoring: 0–5 points, converted to a 0–30 score scale

TOEFL iBT Test Content Reading: This section measures test takers’ ability to understand university-level academic texts. TOEFL test takers read 3-5 passages of approximately 700 words each and answer 12-14 questions about each passage. The passage have all the information needed to answer the questions. There is no requirement for special background. The questions are mainly intended to assess the test takers’ ability to comprehend factual information , infer information from passages, understand vocabulary in context and understand author purpose. These questions are multiple choice. Listening : This section measure test takers' ability to understand spoken English in an academic setting. The questions intend to assess the ability to understand main ideas , import details, recognize a speaker’s attitude or function. It’s mostly multiple choice questions.

TOEFL iBT Test Content [continued] Speaking : This sections measures test takers’ ability to speak English effectively in educational environments, both inside and outside of the classroom. Tow of these tasks are independent where test takers do not receive any oral or written materials. The other four tasks assess integrated skills such as listening and writing. Writing: This section measures test takers’ ability to write in an academic environment and includes two tasks – one independent and one integrated. For the independent task test takers receive no written stimulus materials; instead, they are required to respond to a relatively general question that allows them to tap their own knowledge and experience. For the integrated part test takers read a passage; then they listen to a lecture that takes a position that is somehow different from the position presented in the reading passage. Test takers must then, in connected English prose, write a summary of the important point in the lecture.

The Way the TOEFL is Scored ETS uses both human raters and automated scoring methods to offer a complete and accurate picture of a test taker's ability. While automated scoring models have advantages, they do not measure the effectiveness of the language response and the appropriateness of its content. Human raters are needed to attend to a wider variety of features, such as the quality of ideas and content as well as form. Additionally, studies have shown that prompts designed for fully automated scoring have been more vulnerable to prompt-specific preparation and memorized responses. The TOEFL test uses automated scoring to complement human scoring for the two tasks in the Writing section. Combining human judgment for content and meaning, and automated scoring for linguistic features, ensures consistent, quality scores.

Methods of scoring Objective scoring: Which is any scoring system in which a response will receive the same score, no matter who does the scoring. No judgment is required to apply the scoring rule such as MC , T/F and some fill in the blank. Subjective scoring: which is any scoring system that requires judgment on the part of the scorer. With subjective scoring, different scorers could possibly assign different scores to the same response such as writing tasks , open ended questions and speaking responds. Norm-referenced: Test takers are scored relative to one another.

How ETS ensure TOEFL scoring quality ? ETS raters are trained extensively, pass a certification test and are calibrated daily. The calibration includes task familiarization, guidance on scoring the task, and practice on a range of responses. Raters are continuously monitored for accuracy by ETS scoring leaders and checked each time they score a new test question. Where Tests get rated? To ensure the security and integrity of scores, it is critical that scoring not take place at test sites, but rather through a centralized scoring network that implements and ensures consistent scoring standards. The TOEFL test is scored by a network of raters, carefully controlled from a secure central location. ETS uses a highly diverse pool of raters rather than those exclusive to an applicant's country of origin, and ETS raters score responses anonymously for truly objective scoring. Multiple raters' judgments contribute to each test taker's Speaking and Writing scores in order to minimize rater bias.

Your TOEFL iBT® (Internet-Based TOEFL) scores: Your TOEFL iBT® (Internet-Based TOEFL) scores will provide accurate information about your ability to participate and succeed in academic studies in an English-speaking environment. The TOEFL iBT will test English language skills in four areas, and your TOEFL score report will contain five scores: one total score on a scale of 0 to 120, and four skill scores, each on a scale of 0 to 30. Listening (0 to 30 points) Reading (0 to 30 points) Speaking (0 to 30 points) Writing (0 to 30 points) Total Score (0 to 120 points) Scores will be available online 15 business days after you take the TOEFL test Scores will be valid for two years after the date you take your TOEFL test.

Essential elements of language assessment Validity : Does the test measure accurately what it expects to measure? ( different types of validity to be discussed). Reliability :Administer and score assessments that are maximally reliable Fairness: using scores accurately and fairly Practicality: Keeping constraints in mind: budget, staffing, time, etc

Validity , It is all about the quality of the test Construct validity is: the degree to which a test measures what it claims or purports / constructs to be measured. “ It is not enough to assert that a test has construct validity; empirical evidence is needed. Such evidence may take several forms, including the subordinate forms of validity, content validity and criterion-related validity” Said Hughes, Testing for Language Teachers 2003 . Is TOEFL Valid ? Flash video …. https://www.ets.org/toefl/research/topics/validity

Content Validity “ A test is said to have content validity if its content constitutes a representative sample pf lanugage skills, structure.” said Hugues, Testing for language teachers,2003. TOEFL measures test takers abilities to successfully participate in academic setting using the English language. Hence, all the test content include either lectures, passages from textbooks , discussion in a classroom or outside the classroom but still academic related. Additionally, the questions on these materials require the test takers to present their ability in using the English language the way they will be required inside a classroom sitting either in learning institution or universities.

Content Validity [continued] Another supporting element is; when the students prepare to take TOEFL , they practice how to comprehend factual information , infer information from passages, understand vocabulary in context and understand author purpose. Also, they listen to life or recorded lectures to be able to practice summarizing the lecture and answering questions based on the topic discussed. Hugues, in Testing for language teachers,2003, related the greater content validity to the accuracy for measuring what the test is suppose to measure. It can be concluded that the TOEFL shows a great content validity based on the consistence of its materials and questions in matching the purpose and the construct of the test.

Criterion- related validity “This type of validity related to the degree to which results on the test agree with those provided by some independent and highly dependable assessment of the candidate ability”, Hugues, Testing for language teachers,2003. There are two types of Criterion- related validity: Concurrent Validity : That happen when the test ( the TOEFL in this case ) and the criterion ( which should have a similar construct)are administrated at about the same time. When the TOEFL was created around the 60s , it was compared to other universities’ test and its coloration was .90 Predictive Validity: That present the capability of the test to predict the future performance to the test takers. There is no data to proof that TOFEL scores can predict such a capability on the long term. It’s evidence that high score can help student reach their goals to attend perspective schools but their academic achievements later one could not be predicted. However , it fair to say that TOEFL high scores is a great tool for test takers and can open lots of door of opportunity through their career.

Face Validity Face validity is measuring if the takers and stake holders can subjectively view the test to cover the concept it purport to measure. One of the indication if a test has a have validity is the degree of acceptance from educations authorities and institutions. TOEFL is accepted by almost 9000 different colleges around the world and in 110 different countries. Students and teachers go through a lot of preparation and practice in order to achieve their goals in acquiring high scores that can polish their qualifications.

Reliability Mainly is related to the consistency of having the same results. In the TOEFL iBT test, the reliability estimation for the Reading and Listening sections that contain selected response questions is carried out using a method based on item response theory (IRT) (Lord, 1980). For the Speaking and Writing sections that contain constructed response tasks, generalizability theory (G- theory) is used (Brennan, 1983) The above-mentioned reliability and generalizability analyses are conducted for every test form. Table 1 presents the average section and total score reliability estimates and standard errors of measurement based on operational data from 2007.

Table 1. Reliabilities and Standard Errors of Measurement (SEM) Score Scale Reliability Estimate SEM Reading 0 – 30 0.85 3.35 Listening 3.20 Speaking 0.88 1.62 Writing 0.74 2.76 Total 0 - 120 0.94 5.64 https://www.ets.org/s/toefl/pdf/toefl_ibt_research_s1v3.pdf

TOEFL Reliability The reliability estimates for the Reading, Listening, Speaking, and Total scores are high, while the reliability of the Writing score is somewhat lower. This is a typical result for writing measures composed of only two tasks (Breland, Bridgeman, & Fowles, 1999) and reflects one well-documented limitation of performance testing—reliability estimates for measures composed of a small number of time-consuming tasks are often lower than estimates for measures composed of many shorter, less time-consuming tasks. However, the construct of academic writing as defined for the TOEFL iBT test required the production of extended writing samples (Cumming, Kantor, Powers, Santos, & Taylor, 2000). One implication of these results is that, for making high-stakes decisions such as admissions to college or graduate school, the Total score provides the best information, both because it reflects all four language skills and because it is the most reliable. Nevertheless, there are circumstances under which decision makers may want to examine the profile of scores for test takers, such as the demands of the curriculum or a need for additional language training.

TOEFL Reliability - [continued] Also note that ETS encourages score users to consider a number of other factors, when making admissions decisions, including grade point average, scores on other admissions exams, teacher recommendations, and interviews with individuals. GREAT RECOMMENDATION

Authenticity TOEFL test materials match real life situation as far as academic settings Test questions use natural academic language. Contextualize passages and question around the academic setting. In listening section, materials and questions adhere to the authenticity of the English accents.

Practicality Validity TOEFL testing on computer and via the internet makes test administration more convenient for test takers and test sponsors. Test takers who wish to take the test can sign in and pay the test fee online and receive a confirmation document that has all the information about the test location and time for their reference. TOEFL is administrated in a test centers with administration staff who are highly trained to guide the examinees and help if any technical problem accrue. TOEFL test takes approximately four hours to complete including checking in including one break in the middle of the test.

Washback Washback is the influence that a test has on teaching and learning in a particular educational context . While there were great amounts of data and research findings that TOEFL affects how ESL teachers teach & how learners learn when preparing for TOEFL. I did not encounter research yet that can state firmly and evidently that TOEFL preparation materials cause increase test scores. As was seen in J. Chales Adnderson and Liz Hamp –Lyons article on TOEFL preparation courses : A study of washback https://www.researchgate.net/publication/240738514_TOEFL_preparation_courses_A_study_of_washback

TOEFL Test Preparation ... My personal experience It looked as if it is a requirement rather than an option to take the TOEFL test in order to peruse my master degree. My first attempt is to take test samples to check my true ability in academic English language and identify my strength and weakness areas and plan my study accordingly. These attempts brought poor results that I did not anticipate. My challenge was mostly in the reading sections. I spent a good portion of my assigned time reading the articles and then go to read the multiple questions ( including all the distractors) , then I would try to find the answers within the text. That took too long.

TOEFL Test Preparation ... My personal experience Then I was advised by an ESL teacher to buy test prep books such as BARRON’S to learn more about TOEFL as a standardized and its unique structure, I was able to learn the right strategies in using my time wisely when it comes to the reading parts. I also learned when to skim and when to scan. My preparation reflects on the TOELF washback and how it affects the learners and how they learn and prepare for the test. Without learning about how to prepare for the TOEFL , I would not be see noticeable improvement during my prep time. Although I ended up not taking the test, I learned a lot from my study preparing to take the TOEFL exam. I can certainly believe in the TOEFL positive washback.

References 1-Testing for Language Teachers/ Arther Hughes. Cambridge Language Teaching Library, 2003. 2- http://ltj.sagepub.com Language Testing J. Charles Alderson and Liz Hamp-Lyons TOEFL preparation courses: A study of washback DOI: 10.1177/026553229601300304 1996; 13; 280 http://ltj.sagepub.com/cgi/content/abstract/13/3/280 https://www.researchgate.net/publication/240738514_TOEFL_preparation_courses_A_study_of_was hback 3- www.ETS.org ETS TOEFL Official site www.ets.org/research/research_videos/ https://www.ets.org/s/toefl/pdf/toefl_ibt_research_insight.pdf https://www.ets.org/Media/Research/pdf/RR-03-18.pdf https://www.ets.org/s/toefl/pdf/toefl_ibt_insight_s1v4.pdf https://www.ets.org/Media/Tests/TOEFL/pdf/SampleQuestions.pdf https://www.ets.org/s/toefl/pdf/toefl_ibt_research_s1v3.pdf