1. Sample judges questionnaire

2. Semi-structured intview schedule
How well were you prepared for your role as judge?
Did you manage to follow my instructions? Why not?
Can you recap the judgement process? What exactly were you asked to do? Why can't you, do you think?
How would you evaluate the judgement process? What are your reasons for saying that?
Here is the Lancaster model for the judgements. Do you think we were following it? What exactly was missing?
Were my instructions in agreement with it?
What do you think of the model?
Which year's grammar and listening papers were better written, do you think? How do they compare for difficulty? Is this your own observation or other staff said so too?
In your opinion, how do different years -- 'cohorts' – of students compare for their English?
Have you observed any of the 3 years of students to have had special strengths or weaknesses? If so, what were these?
Look at my assessment of the test data. Any comment?
How do you think we could make improvements so that there is no such gap between the judges' estimates and the test performance of borderline students?
What would you have to do in order to improve judgements next year?
This study is conducted rather late. Do you think this can affect the quality/validity of the findings?
Having had this talk, how would you assess your understanding of the judgement procedure now, as opposed to the beginning of the judgement exercise?
Anything else to add?
Photocopies of the schedule may have to be provided.
Source: Dávid (1995)
A continuum of interview types
Unstructured through semi- structured to completely structured
Question types: closed and open
© 2015 Dávid & Ryan