Overview of ICCS field trial National Research Coordinators Meeting Windsor, June 2008

NRC Meeting Windsor June 2008 Content of presentation Participation in field trial General outcomes of field trial Feedback from national centres –Summary in NRC(June08)1.pdf Problems identified in field trial Consequences of field trial

NRC Meeting Windsor June 2008 Field trial outcomes Field trial data analysis –revealed that generally the instruments worked well Cognitive test well targeted to ability & reliable Questionnaire scales reliable and fit structure –informed the selection of item material for main survey –confirmed general feasibility of on-line surveys for teacher and schools Review of implementation procedures –Generally procedures worked well –Confirmed the feasibility of using regional instruments –Some possibilities for improvement & streamlining

NRC Meeting Windsor June 2008 Field trial participation 32 countries undertook field trial One country to field trial in June data sets included in international field trial analysis report Approximately –19,000 students –9,000 teachers –700 schools

NRC Meeting Windsor June 2008 School response rates 21 countries had school response rates of at least 85 percent BEFORE replacement 4 countries had school responses rates of at least 85 percent AFTER replacement Only 7 countries had school response rates below 85 percent Period for field trial close to the end of the year contributed to problems with school participation

NRC Meeting Windsor June 2008 Student response rates Overall, about 90 percent of students participated within schools Response rates (slightly) below 85 percent were found in only two countries

NRC Meeting Windsor June 2008 Teacher response rates Across countries, there was a median teacher response rate within schools of 90 percent When combined with school response rates the overall response rate for the teacher survey was 89 percent (median across countries) In 10 out of 31 countries the overall response rate was below 75 percent

NRC Meeting Windsor June 2008 Results of NRC Survey Overall satisfaction with manuals Main problems encountered: –Convincing schools to participate –Explaining sampling procedures –Implementing some test administration procedures –Reaching a high student and teacher participation within schools

NRC Meeting Windsor June 2008 Issues raised regarding survey instruments Length of student test and questionnaires (7 countries) Questions in instruments seen as too difficult for students of this age (4 countries) General directions too long and repetitive (5 countries)

NRC Meeting Windsor June 2008 Issues raised regarding the teacher sampling design Listing of all teacher teaching in target grade seen as a problem Two national centres saw the procedures as a major obstacle for school participation Data about teachers’ month/year of birth seen as inappropriate

NRC Meeting Windsor June 2008 Issues raised regarding teacher participation Teachers not teaching CCE saw questionnaire as irrelevant to them Two national centres viewed questionnaire as too long Teacher participation more difficult to obtain than student participation

NRC Meeting Windsor June 2008 Issues raised regarding regional modules Problems with layout of European instrument (part 2 being upside-down and back-to-front) Problems with administration and length of general directions

NRC Meeting Windsor June 2008 Issues raised regarding manuals and software More than 80% found the manuals “very helpful” but suggestions to: –Simplify and shorten manuals –Refine software tools for sampling and data entry Need to improve the scoring guides for coding of open-ended questions

NRC Meeting Windsor June 2008 Issues identified by the ISC Need for improvement of adaptation notes Provision for countries to record their national adaptations to general directions Inclusion of checklist in manual for submission for verification

NRC Meeting Windsor June 2008 Looking ahead Reduction in assessment time –Testing time now 45 minutes –Student questionnaire time now about 35 minutes –Regional instruments shorter –Some reduction in teacher and school questionnaires Reduction in teacher sample size –Cluster size of 15 for main survey –On average, no more than 15 teachers in target population at schools

NRC Meeting Windsor June 2008 Expert advice Project Advisory Committee Meeting 7-8 June here in Windsor Discussion of outcomes and proposals Some changes to the proposed instruments following PAC advice –PAC suggestions will be included in presentations

NRC Meeting Windsor June 2008 Questions or comments?