THE CERTIFICATE OF DUTCH AS A FOREIGN LANGUAGE (CNaVT) TBLT Honolulu – September 2007 Katholieke Universiteit Leuven Belgium Universiteit van Amsterdam The Netherlands Language testing and culture

Do you congratulate someone with or on their birthday? Question …

Neither! It’s a trick question. We don’t use ‘to congratulate’ or ‘congratulations’ in connection with birthdays in English, except in very special cases (18, 21, 100, etc.): we normally just say ‘Happy Birthday’ or ‘Many happy returns’, instead. And we don’t shake hands as we do so. However, we can congratulate people on getting a job, winning an award, or getting married, for instance. And then a handshake is often part of the ritual. ‘With’ is never used with ‘to congratulate’ or ‘congratulations’ – it’s always ‘to congratulate (someone) on’… and ‘congratulations on’.. Answer …

The Certificate of Dutch as a Foreign Language -CNaVT (Certificaat Nederlands als Vreemde Taal) -administration in more than 40 countries around the world: from Madrid to Jakarta and from Warsaw to Rio de Janeiro candidates each year

Assess as direct as possible -construct on the basis of needs analysis -authenticity: 6 exams represent 6 different TLU domains -task-based: exam tasks represent real life language use situations

Valid assessment Uiterwijk & Vallen (Language Testing 2005, 22) (…) the question is: does the test measure a particular construct for the different groups in a valid way? In other words, does it measure, for various subgroups, what is claims to measure?

The effect of culture Marquardt & Gillam (Language Testing 1999, 16) The most frequent problem related to misinterpretation of assessment results is related to cultural (…) differences between test items, examiners and their (…) candidates.

Fairness to candidates McNamara & Roever (Language Testing. The Social Dimension) An interpretative argument is only as strong as its weakest link. It is the social responsibility of testers to report efforts to estimate the impact of these factors and to control them.

Item bias analysis -statistical procedures to find out if an item is shows DIF -research the factors responsible for the DIF -determine if the cause of DIF is construct (ir)relevant, leading to bias

CNaVT item bias research -PTIT exam items, spread over 2 versions, 9 tasks per version -Facet-analysis: facet ‘country’ -Mantel-Haentzel analysis

CNaVT item bias research Facets -empirically 1324 bias terms item * country -68 interactions showed DIF -56 disadvantageous to subgroup -12 advantageous to subgroup

CNaVT item bias research

Example 1 Country: Indonesia Item: Question in form ‘What sort of object have you lost?’

CNaVT item bias research Example 2 Country: Russia Item: Question from friend in letter (…) ‘But I often think back with nostalgia about my holiday in your country. (…) And you? What did you do during your holidays? Did you travel? Where to? Or did you stay home and what did you do?’ (…)

Our issues concerning DIF analysis -Categorisation: on the basis of nationality, region? No strict boundaries. Or: what is ‘culture’? -Determination that culture is the causing factor of variance: so many intervening factors (for example: socio-economical status) -Determination that DIF is part of the construct > tension

What we do -awareness of our construct: what (concerning culture) is relevant, what is irrelevant -generate input on possible cultural bias from different sources in different stadia -In case of suspicion of irrelevance: DIF analysis

To conclude More research Further define our construct Communication to stakeholders Fairness to candidates!!!

THE CERTIFICATE OF DUTCH AS A FOREIGN LANGUAGE (CNaVT) TBLT Honolulu – September 2007 Katholieke Universiteit Leuven Universiteit van Amsterdam