Using Verbal Reports for Data Collection and Analysis Maja Gjurovikj STANAG 6001 testing team Ministry of Defence Republic of Macedonia

Why trialling? Collect data to be analyzed: Do items test what they were meant to test? Do they distinguish between masters or non- masters? Are the ambiguous? Inform about test administration procedures: time instructions answer keys Bank items to generate new tests: determine characteristics of items keep/revise/reject

Test item characteristics CODE DATA FROM AS NO. GROUP A B C D ? FV DI IMMIGRATION Pretesting Feb. 8 2011, Perugia 32 T 4 13 26% -0,36 M 7 10 12 2 % 36% 24% 13% 0% CODE DATA FROM AS NO. GROUP A B C D ? FV DI CAR ALARMS 2 PretestingOct. 2011, BiH 40 T 3 17 4 5 2 36% 0,30 M 8 10 6 7 % 14% 16% 21% 12%

Psychometric approaches vs. mentalistic approaches empirical validity theoretical validity concurrent validity predictive validity QUANTITATIVE APPROACHES EXAMINE PSYCHOMETRIC PROPERTIES OF THE TEST face validity content validity construct validity QUALITATIVE APPROACHES PROVIDE MORE INFORMATION ON TEST CONTENT, TEST TASK PROPERTIES AND TEST- TAKING PROCESSES

What is verbal report analysis? a special label used to describe the verbal data gathered a technique that draws inferences about the cognitive processes that produced the verbalization comprised of ‘protocols’ - utterances made by an individual invaluable at the trialling and pretesting stage supplement other quantitative data

Variations on verbal report procedure Form of Report TALK ALOUD THINK ALOUD Temporal Concurrent Retrospective Concurrent Retrospective Variations Procedural Mediated Non-med. Mediated Non-med. Mediated Non-med. Mediated Non-med.

Stages in collecting and analyzing verbal reports DATA COLLECTION STAGES Task identification Task analysis Selecting an appropriate procedure Selecting subjects Training subjects Collecting verbal reports Collecting supplementary data Transcribing verbal reports DATA ANALYSIS STAGES Developing an encoding scheme Segmenting protocols Encoding protocols Calculating encoder reliability Analyzing data Source: Green, A. (1998). Verbal Protocol Analysis in Language Testing Research: A Handbook. Cambridge: Cambridge University Press and University of Cambridge Local Examinations Syndicate

Examples of verbal report methodology Cohen (1994) five test-takers a two-part test concurrent reports observe test-takers during the test-taking process give test-takers a questionnaire conduct in the test-takers’ L1 Buck (1994) six test-takers 54-item test based on a single listening text 13 short sections, one at a time attend a post-test interview take the test items again

Advantages vs. disadvantages reveals test validity provides direct information has wide application commitment in time disruption of behaviour individual differences

Some qualitative data analysis softwares Coding Analysis Toolkit (CAT;  https://cat.texifter.com) Weft QDA (http://www.pressure.to/qda) Text Analysis Markup System (TAMS; http://tamsys.sourceforge.net) Transana (https://www.transana.com) QDA Miner Lite (https://provalisresearch.com/products/qualitative- data-analysis-software)

Points to note reveals the dynamics of the test taking processes uncovers students’ unobservable behavior resourceful feedback concerning the mental processes or test-taking strategies complements the analysis generated from quantitative investigation a developing and maturing method

References: (https://www.youtube.com/watch?v=BwpPIiBK0cA) Green, A. (1998). Verbal Protocol Analysis in Language Testing Research: A Handbook. Cambridge: Cambridge University Press and University of Cambridge Local Examinations Syndicate. Grotjahn, R. (1986). Test validation and cognitive psychology: Some methodological considerations. Language Testing 3: 159–185. Zheng, Y. (2009) Protocol Analysis in the Validation of Language Tests: Potential of the Method, State of the Evidence. International Journal of Pedagogies and Learning 5: 124–137 Council of Europe (2009). Relating Language examinations to the Common European Framework of Reference for Languages: learning, teaching, assessment Cambridge: Cambridge University Press. (https://www.youtube.com/watch?v=BwpPIiBK0cA)