Language testing: Current trends and future needs 24 – 25 October, 2014 Centro de Lenguas Modernas, Granada University PaedDr. Miroslava Jurenková Mgr. Alžbeta Palacková Mgr. Michaela Ujházyová, PhD. NATIONAL INSTITUTE FOR CERTIFIED EDUCATIONAL MEASUREMENTS Modern education for the knowledge society/The project is co-financed by the European Union

Was established in 2008 and is responsible for: external part of the Maturita examination, testing of 5 th and 9 th grade pupils, international projects (e.g. PISA, PIRLS, TIMSS), national projects co-financed by ESF (e.g. "E-testing"). Increasing the quality of education at primary and secondary schools with the use of electronic testing (2013 – 2015) NÚCEM

What will we be talking about? Some information about the Slovak national project: "E -testing ". E-testing: new ways of test and item development Answer keys: how to prepare answer keys for paper-based (printed) tests and e-tests.

Activity 1.1. Development of tests and items: for school testing (School item bank), for national testing (NUCEM item bank). The aim is to develop about 30,000 new items & 130 tests in cooperation with teachers and experts – item writers, local supervisors, reviewers... (approximately 600). to assess students’ knowledge and skills in various subjects. Training of teachers in development of e-tests and e-items.

Activity 1.2. Implementation of electronic testing. Cooperation with IBM: developing a software for items and tests development, e-testing and item banking. Training of the school’s IT administrators of e-testing. Providing research analyses and reports based on schools’ measurable outputs. Evaluation and comparison of the quality of schools, monitoring the value added indicators.

Paper-based testing vs. E-testing What are the differences between them? We have paper, we have computers. It looks simple... But… using e-testing tools can transform the entire test development life cycle.

E-testing and Item banking Advantages: E-testing offers more response options  new types of test items (drag and drop, interactive ones...). Item content can include multi-media content (audio, video, animations, rich graphics, etc.). Item banks can be easily updated. Tests can be easily created by rearranging items. Testing listening and reading comprehension (receptive skills)can easily be implemented and delivered via e-testing. Most question types can be scored instantly  students are more motivated if they get their scores immediately after taking a test (immediate feedback).

E-testing and Item banking Advantages: Some programs permit only multiple-choice items and some open-ended items(completion, essay, fill-in). The program developed within the Slovak national project support 12 types of items and some other are in the process of development. By developing the system for computer-based testing, it later enables us to use adaptive tests which will be tailored to the individual ability of the test-taker (better estimates of ability with shorter tests).

E-testing and Item banking Challenges: Transforming some types of items used in a traditional paper testing to e-form is sometimes difficult. Many item writers involved in the process - variations in test item style, format and difficulty  a guide with item development standards had to be prepared. Preparing answer keys for open-ended questions and scoring them is for now quite difficult (human/computer?). More time consuming for those who develop the tests. Computer or Internet based exams represent some risk, especially with regard to exam security (in March 2015 – the national online testing: Maturita, T9).

E-testing and Item banking Challenges: Additional skills are required for test writers and test administrators (training is needed). To carry out tests in a computer requires some IT skills: typing, mouse navigation, key combinations  some test takers may not work with computers therefore taking electronic exams may be difficult for them. Reading from a screen is more exhausting, especially long passages with scrolling  the amount of material that can be presented on a computer screen is limited. This limitation could be a problem, for testing languages if we want to develop a reading test based on relatively long passages.

Types of items o Multiple choice (with one response) o Multiple choice(with multiple response) o Fill (one blank) o Fill (multiple blanks) – Custom Fill o Ordering o Single choice matrix o Multiple choice matrix o Marking words in the text o Drag & Drop o True/false o Hotspot o File upload

Multiple choice (with one response)

Multiple choice (with multiple response)

Fill (multiple blanks)

Ordering

Multiple Matrix

Single Matrix

Marking text

Drag and drop

True – False

Hotspot

File Upload

The answer key for open-ended items 1. Before the pilot testing: The test writer develops the 1. version of the answer key. Test coordinators check the key, discuss it with professionals - next possible answers are added. 2. After the pilot testing: Student´s responses are collected and judged by reviewers. New possible responses are added based on student´s answers in pilot testing. If it is needed instructions are edited to make the task more comprehensible.

Scoring open-ended questions/items Human factor (paper testing) The teachers scoring the responses do not respect the answer key: 1. Teachers mark the response as correct instead of incorrect: 2. Teachers mark the response as incorrect instead of correct: 3. Teachers mark the response as incorrect/correct instead of no answer: without knowing the rules and instructions for marking, accidentally, intentionally.

PC factor (e-testing) The system scoring the responses sticks to the key (no exceptions). 1. The system marks the response as incorrect instead of correct: Automatic computer-based marking of open-ended responses operates at basic levels of character or rule recognition. The pre- defined key cannot predicts all specificities in responses of test takers (double space, wrong apostrophe, typing errors, capital letters, quotation marks, more words etc.). For this reason, we are still dependent on human marking. Because the answer key does not include all of the possible answers. 2. The system marks the response as incorrect instead of no answer: Because it can not distinguish incorrect from irrelevant response. Scoring open-ended questions/items

Paper testing vs. e-testing Paper testingE-testing more subjective opportunities to cheat (students & evaluators) How to avoid problems giving feedback to the National school inspection, establishing test centers coordinated by NUCEM, relying on e-system. more subjective opportunities to cheat (students & evaluators) How to avoid problems giving feedback to the National school inspection, establishing test centers coordinated by NUCEM, relying on e-system. more objective and fast reduced opportunities to cheat How to avoid problems Developing new types of items, with instructions adapted to e- system, Utilizing wide data form pilot testing (responses) for making answer keys more precise, relying on people. more objective and fast reduced opportunities to cheat How to avoid problems Developing new types of items, with instructions adapted to e- system, Utilizing wide data form pilot testing (responses) for making answer keys more precise, relying on people.

1.typing errors + double spaces 2.more words than required 3.irrelevant response 4.more errors in one response Sample types of incorrect responses

Answer key for open-ended items in paper testing: English language Teachers who score the responses are trained to know how to work with instructions provided in the answer key. Item type: Custom Fill.

"To key" or "not to key"? E-test form of the answer key would be:

Answer key for open-ended items in paper testing: Slovak language and Slovak literature Teachers who score the responses are trained to know how to work with instructions provided in the answer key. Type of items: Fill.

To key or not to key?  Item nr. 55: ako, ako/ako ako/ako – ako/ako/ako tie, ako tie/ako tie ako tie/ako tie – ako tie/a, a /a a /a – a /ako tys’ pekná, krajina moja, ako mladistvosť milá mi tvoja/ako tie Božie plamene, ako tie kvety na chladnej zemi/ako tie Božie plamene, ako tie kvety na chladnej zemi, ako tie drahé kamene/ako tie kvety na chladnej zemi, ako tie drahé kamene/a pekný život tie kvety žili, a diamant v hrudi nezhnije/...55  Item nr. 55: ako, ako/ako ako/ako – ako/ako/ako tie, ako tie/ako tie ako tie/ako tie – ako tie/a, a /a a /a – a /ako tys’ pekná, krajina moja, ako mladistvosť milá mi tvoja/ako tie Božie plamene, ako tie kvety na chladnej zemi/ako tie Božie plamene, ako tie kvety na chladnej zemi, ako tie drahé kamene/ako tie kvety na chladnej zemi, ako tie drahé kamene/a pekný život tie kvety žili, a diamant v hrudi nezhnije/...55  Item nr. 54: tys/tys´/tys‘54  Item nr. 54: tys/tys´/tys‘54  Item nr. 32 Tak, že, za/tak, že, za/Tak že za/tak že za/Tak že za/za, že, tak/za že tak/za, tak, že/za tak že/že, za, tak/že za tak/že, tak, za/že tak za/...32  Item nr. 32 Tak, že, za/tak, že, za/Tak že za/tak že za/Tak že za/za, že, tak/za že tak/za, tak, že/za tak že/že, za, tak/že za tak/že, tak, za/že tak za/...32 E-test form of the answer key would be:

Changing the type of an item: item 54 Type of item: Fill Type of item: Marking text The item was prepared for paper-based testing ; in e-testing environment we want to avoid problems with many possible responses.

Changing the type of an item: item 32 Type of item: Fill Type of item: Marking text The item was prepared for paper-based testing ; in e-testing environment we want to avoid problems with many possible responses.

Granada: IATEFL conference E-testing presentation