The CSES Questionnaire Design Process David Howell University of Michigan dahowell@umich.edu Jessica Fortin GESIS – Leibniz Institute for the Social Sciences jessica.fortin@gesis.org European Survey Research Association (ESRA) Lausanne, Switzerland July, 2011

CSES Questionnaire Over 60 countries participate Common module of survey questions new theme every five years some questions the same some different according to new theme to be run “as is” Administrative, demographic and voting variables to national standards

Questionnaire Design Process

Questionnaire Design Process A lengthy development process Over a three year period, 2008-2011 Iterative As transparent as possible Collaborative, with broad input

Planning Committee Formation A Planning Committee, comprised of, selected by, and informed by collaborators, designs and oversees each Module. Public call for nominations Nominating Committee recommends members Plenary Session confirms list and suggests additional members Result = 20 members from 15 countries

Theme Development Every five years a new module is developed to specifically address a big question in science. Multiple public calls for theme proposals Task Force collects proposals Task Force presents and evaluates proposals for Plenary Session and Planning Committee Theme(s) tentatively decided upon before questionnaire design begins

Questionnaire Development After a theme is arrived at, attention turns to developing the questionnaire for it. Subcommittees formed to pursue the new theme(s) and improvements to past questions Subcommittees return to proposers for revisions to their original proposals Draft questionnaire produced Planning Committee discussion, revision Repeat as necessary

Pretesting The draft questionnaire is pretested cross-nationally in different contexts. Partial pretests in Brazil, Germany, UK Full pretests in Canada, Ireland Reports received from pretest countries

Questionnaire Finalization After having received extensive feedback, the questionnaire can now be finalized. Subcommittees formed to address sections outside of the new theme(s) A Plenary Session provides final feedback Review of pretest results, subcommittees Planning Committee makes final changes Final questionnaire disseminated

Challenges

Design Challenges 10-15 minutes is not much questionnaire time … but, pressure to reduce collaborator burden competing with other content on surveys competing with other comparative studies Existing, validated questions preferred Priority to cross-national questions must be applicable in most all countries must work well across most all countries must be feasible in a variety of study designs

Information Challenges A lot of people are involved Managing and tracking lots of input Reconciling many opinions and approaches Consensus not always possible Making decisions with imperfect information More testing in more contexts would always be better, but relies on volunteers who... Have an election coming up (optimally) Have a funded survey with available space Can produce reports in a timely fashion Eventually decisions must be finalized

Collective Challenges A lot of countries are involved Cross-national and cross-cultural applicability and equivalence Accommodating the variety of institutional arrangements in the many countries Difficult to predict or account for all situations, even with broad cross-national participation in the process

Where to go from here

Where to go from here A formal post-process evaluation We’ve done this a number of times now What worked well, what didn’t Consider improvements Formalize and document process for next time Some possibilities: More extensive pretesting Allow more time for pretesting Test in more cultures and contexts Possible use of focus groups

Where to go from here More focus on cross-national/cultural issues ... concept equivalence ... collaborator/interview instructions ... translation issues More methodological working groups Learning from the approaches of other cross-national studies

Thank you for your time! To learn more about CSES, or to download data: www.cses.org ...or email your questions to: cses@umich.edu