Maurice Grinberg evaluation.nbu.bg ALTE meeting, Lisbon, November 12-15

 Modern private university  Year 1991: students in 30 programs  Year 2008: students in more than 100 programs  Bachelor, Masters, and PhD programs  Distance learning  For the first time in Bulgaria: General Education Test as entrance exam four times a year  Assessment Center ALTE meeting, Lisbon, November 12-15

 Since 1995: General Education Test as admission exam (more than candidates per year)  Since 1998 for all NBU students: L2 exams at the end of a course of study, levels from A2 to B2 6 languages: English, German, Italian, Russian, Spanish, and French L1 exam for writing skills in Bulgarian language Computer skills exam  Since 2002: placement tests ALTE meeting, Lisbon, November 12-15

 Staff: Experts in testing, psychometrics, cognitive science, languages  Advisory board: Department of Psychology and Cognitive Science, and department of Foreign Languages NBU QA department External consultant: prof. D. Dimitrov, George Mason University  Projects and research Leonardo da Vinci program (WELKOM) Erasmus program (NETTLE) Assessment in e-learning Cognitive assessment, small group assessment ALTE meeting, Lisbon, November 12-15

 Monitoring and development of all examination procedures in NBU  Development of exams for NBU students  Analysis and interpretation of test results: test score reports to test takers and university administration  Item-banking and storage of data  Training for item-writers and standardized workshops for examiners ALTE meeting, Lisbon, November 12-15

 Fairness and equity Standardized administration Monitoring during examination Special accommodation for candidates with disabilities Information about the content and format of the examination: question types, rubrics and appropriate test-taking strategies, test samples ALTE meeting, Lisbon, November 12-15

 B2 is a required level in NBU for all students  Goals: use the language to express and understand opinions, to build up and understand arguments, to communicate with students from other countries in English, Spanish, Italian, French, German or Russian.  Certification decisions and graduation ALTE meeting, Lisbon, November 12-15

 Test takers: Bulgarian native speakers with different topical knowledge and second language experience  Tasks: both from the instructional and real-life domain  Scores are reported to candidates in grades  Tested abilities: speaking, listening, reading, and writing ALTE meeting, Lisbon, November 12-15

 Psychometrics: IRT used for evaluation of test item properties Many-facet Rasch analysis in assessing essay writing task (facets: topic, raters, and assessment criteria) Estimation of candidates’ abilities, raters’ severity, relative difficulty of tasks, and relative difficulty of assessment criteria Bias/interaction analysis ALTE meeting, Lisbon, November 12-15

 Meet all ALTE standards for quality in language tests: Revise existing foreign language exams Achieve a consensus about test qualities Develop a new test for Bulgarian as a foreign language  Exchange of best practices: Investigation and assurance of test reliability Improvement of validation argumentation Elaboration of procedures to involve all stakeholders in the examination processes ALTE meeting, Lisbon, November 12-15

 Development of tests for Bulgarian as a foreign language Situational analysis: need for a A1 test for foreign university students in Europe and Asia and related textbooks Planning phase: users and test takers identified, resource balance performed ALTE meeting, Lisbon, November 12-15

 Development of tests for Bulgarian as a foreign language (cont.) Design phase: theoretical background of the test Approaches considered:  Bachman model of CLA will underlie each task as a needed executive resource  CEFR for the domain, communicative goal and language function  Weir’s Frameworks for Developing and Validating tests ALTE meeting, Lisbon, November 12-15

 NBU:  Assessment center: (in construction!) ALTE meeting, Lisbon, November 12-15

It consists of two parts. The first one is a task for monological discourse. The candidates have to express their own opinion and support it with arguments. The length of expected response is about 1,5 min. Task stimuli are: verbal prompts arranged under six topics, graphics, diagrams or pictures, real-life situations. The candidate interacts with the examiner, who acts as an interlocutor. The second one is a task for spoken interaction (conversation with interlocutor) Each performance is rated by two independent trained raters. The rater uses a 5-point scale to range the candidate’s performance. The assessment criteria are: accuracy, vocabulary, fluency, cohesion and task fulfillment. ALTE meeting, Lisbon, November 12-15

It consists of three parts: listening, reading and writing. Listening tasks test the ability to understand main points, gist, opinion, specific information, attitude and tone. The texts are excerpts from news bulletins, interviews, dialogues, etc. of about 3-4 min in length. All texts are heard twice. Response format: multiple choice. Reading tasks test careful reading and expeditious reading on both of global and local level. Texts are authentic (with slight adaptation where appropriate), usually from magazines. Reading tasks involve the processing of two texts with different discourse characteristics (about 800 words overall). Response formats are: multiple choice, matching, short answer questions. ALTE meeting, Lisbon, November 12-15

Language use tasks test grammar, vocabulary and syntax. Response formats: gap-filling, multiple choice, matching. Texts: usually two to three texts of about words each Writing task (essay or letter) tests the student’s ability to produce and sustain in writing a coherent, appropriate and purposeful text. The length of the expected answer is about words. The candidates has a choice of two topics. The rubric is formulated in two to four sentences, which provide just enough information to stimulate the examinee to generate his/her own ideas on the topic. Two raters assess each essay/letter, using a 4 point scale. The assessment criteria are: grammar and vocabulary, organization and cohesion, relevance of content. Test instructions are in Bulgarian. Test rubrics are in the target language. ALTE meeting, Lisbon, November 12-15

The test result claims to be based on Bachman’s model of CLA. The construct involves: Language knowledge ORGANIZATIONAL KNOWLEDGE GRAMMATICAL KNOWLEDGE  Knowledge of vocabulary  Knowledge of syntax TEXTUAL KNOWLEDGE  Knowledge of cohesion  Knowledge of rhetorical or conversational organization PRAGMATICAL KNOWLEDGE FUNCTIONAL KNOWLEDGE  Knowledge of ideational function SOCIOLINGUISTIC KNOWLEDGE  Knowledge of register Strategic competence  GOAL SETTING  ASSESSMENT  PLANNING Problems: we know what want to measure, but we haven’t yet accumulated construct-related, content-related and criterion-related evidence Decisions: start the process of validation ALTE meeting, Lisbon, November 12-15

Problems: need for the statistical procedure to become routine; need to build an item bank Decisions: customizing developed software, using IRT, Rasch based FACETS model and Least Squared Distance Model (LSDM) ALTE meeting, Lisbon, November 12-15

About Impact: Our strive to achieve positive impact involves Providing public information and advisory services for students Seminars to familiarize university teachers with our research on test qualities Training for item writers and examiners Reports to the University Board Cooperative work with the Department of Foreign Languages to improve continuous assessment in language classes Problems: we have not investigated the educational impact of our examinations Decisions: start collecting evidence about “face validity” About Practicality: The Assessment Center develops and implements the appropriate procedures for managing all aspects of the examination process. Problems: need to increase research studies in all aspects of test production Decisions: identification of suitable experts ALTE meeting, Lisbon, November 12-15