Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 1 Chapter 22: Evaluating a Research Report Objectives For each of the major sections and subsections of a research report, identify the criteria used in determining its adequacy. For each type of research, identify criteria used to determine the adequacy of a study representing that type.

Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 2 Evaluating a Research Report Introduction Problem Is there a statement of the problem? Does the problem indicate a particular focus of study? Is background information on the problem presented? When necessary, are variables directly or operationally defined? Does the problem statement indicate the variables of interest and the specific relations among the variables that were investigated?

Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 3 Evaluating a Research Report Introduction Review of related literature Is the review comprehensive? Is the review well-organized? Does it flow? Is the review more than a series of abstracts or annotations? Are all cited references relevant to the problem under investigation? Is the relevance of each reference explained? Are most of the sources primary? Are references cited completely and accurately?

Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 4 Evaluating a Research Report Introduction Hypotheses Are specific research questions listed or specific hypotheses stated? Is each hypothesis testable? Does each hypothesis state an expected relation or difference? If necessary, are variables directly or operationally defined?

Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 5 Evaluating a Research Report Method Participants Are the size and major characteristics of the population described? Is the sample of adequate size for the study that was conducted? Are the accessible and target populations described? Is the sampling method clearly described? Does there appear to be any sampling bias?

Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 6 Evaluating a Research Report Method Instruments Do instruments and their administration comply with IRB standards? Were permissions obtained? Are the instruments appropriate for measuring the intended variables? Is a rationale provided for the selection of the instruments used?

Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 7 Evaluating a Research Report Method Instruments Are the instruments and any sub-scales fully described? If an instrument was developed, are procedures that establish reliability and validity shared? Does the researcher have the needed skills or experience to construct or administer the instrument?

Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 8 Evaluating a Research Report Method Design and Procedure Are the design and procedures appropriate for examining the research question or testing the hypotheses of the study? Were the instruments and procedures applied correctly? Do the procedures logically relate to one another?

Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 9 Evaluating a Research Report Method Design and Procedure If a pilot study was conducted, is it clearly explained? Are control procedures described? Does the researcher discuss and account for confounding variables that were not controlled?

Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 10 Evaluating a Research Report Results Are appropriate descriptive statistics presented? Are the tests of significance appropriate, given the hypotheses and design of the study? Is every hypothesis addressed? Are assumptions of analyses met by the data?

Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 11 Evaluating a Research Report Results Was probability level set? Are the results clearly described? Are the tables and figures easy to understand and are they organized? Are they described in the text? Was the inductive logic used to produce results in the qualitative study explicit?

Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 12 Evaluating a Research Report Discussion (Conclusions & Recommendations) Is each result discussed in terms of the original hypothesis or topic? Is each result discussed in terms of how it related to existing theory and previous research?

Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 13 Evaluating a Research Report Discussion (Conclusions & Recommendations) Are generalizations consistent with results? Are theoretical and practical implications of the findings discussed? Are possible effects of uncontrolled variables discussed?

Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 14 Evaluating a Research Report Abstract or summary Is the problem stated? Are the number and type of participants and instruments described? Is the design identified? Are the procedures described? Are the major results and conclusions stated?

Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 15 Evaluating Types of Reports Survey Research Are questionnaire validation procedures described? Was the questionnaire pilot tested and if so, are the pilot test procedures adequately described? Are directions to questionnaire respondents clear? Does each item in the questionnaire address a single concept and relate to an objective of the study?

Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 16 Evaluating Types of Reports Survey Research Does the cover letter explain the purpose and importance of the study, and does it give the potential respondent a good reason for cooperating? If appropriate, is confidentiality or anonymity of responses assured in the cover letter?

Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 17 Evaluating Types of Reports Survey Research What is the percentage of returns, and how does it affect the study results? If the response rate was low, was any attempt made to determine any major differences between respondents and nonrespondents? Are follow-up activities to increase returns described? Are data analyzed in groups or clusters rather than in a series of many single-variable analyses?

Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 18 Evaluating Types of Reports Correlational Research Relationship studies Were variables carefully selected (i.e., was a shotgun approach avoided)? Is the rationale for variable selection described? Are conclusions and recommendations based on values of correlation coefficients corrected for attenuation or restriction in range? Do the conclusions avoid suggesting causal relations among the variables investigated?

Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 19 Evaluating Types of Reports Correlational Research Prediction studies Is a rationale given for selection of predictor variables? Is the criterion variable well defined? Was the resulting prediction equation validated with at least one other group?

Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 20 Evaluating Types of Reports Causal-Comparative Research Are the characteristics or experiences that differentiate the groups (i.e., the grouping variable) clearly defined or described? Are critical extraneous variables identified? Were any control procedures applied to equate the groups on extraneous variables? Are causal relations discussed with due caution? Are plausible alternative hypotheses discussed?

Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 21 Evaluating Types of Reports Experimental Research Was an appropriate experimental design selected? Is a rationale for design selection given? Are threats to validity associated with the design identified and discussed? Is the method of group formation described? Was the experimental group formed in the same way as the control group? Were groups randomly formed and the use of existing groups avoided?

Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 22 Evaluating Types of Reports Experimental Research Were treatments randomly assigned to groups? Were critical extraneous variables identified? Were any control procedures applied to equate groups on extraneous variables? Were possible reactive arrangements (e.g., the Hawthorne effect) controlled for? Are the results generalized to the appropriate group?

Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 23 Evaluating Types of Reports Single-Subject Research Are the data time constrained? Was a baseline established before moving into the intervention phase? Was condition or phase length sufficient to represent the behavior within the phase? Is the design appropriate to the question under study?

Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 24 Evaluating Types of Reports Single-Subject Research If a multiple baseline design was used, were conditions met to move across baselines? If a withdrawal design was used, are limitations to this design addressed? Did the researcher manipulate only one variable at a time? Is the study replicable?

Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 25 Evaluating Types of Reports Qualitative Research (In General) Does the researcher give a general sense of the focus of study? Does the researcher state a guiding hypothesis for the investigation? Is the application of the qualitative method described in detail? Is the context of the qualitative study described in detail?

Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 26 Evaluating Types of Reports Qualitative Research (In General) Is the purposive sampling procedure described and related to the study focus? Is each data collection strategy described? Is the researcher’s role stated (e.g., nonparticipant observer, participant observer, interviewer, etc.)? Are the research site and the researcher’s entry into it described?

Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 27 Evaluating Types of Reports Qualitative Research (In General) Were the data collection strategies used appropriately, given the purpose of the study? Were strategies used to strengthen the validity and reliability of the data (e.g., triangulation)? Is there a description of how any unexpected ethical issues were handled? Are strategies used to minimize observer bias and observer effect described?

Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 28 Evaluating Types of Reports Qualitative Research (In General) Are the researcher’s reactions and notes differentiated from descriptive field notes? Are data coding strategies described and examples of coded data given? Is the inductive logic applied to the data to produce results stated in detail? Are conclusions supported by data (e.g., are direct quotations from participants used to illustrate points)?

Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 29 Evaluating Types of Reports Interview Studies Were the interview procedures pilot tested? Are pilot study procedures and results described? Does each item in the interview relate to a specific objective of the study? When necessary, is a point of reference given for interview items?

Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 30 Evaluating Types of Reports Interview Studies Are leading questions avoided in the interview? Is the language and complexity of the questions appropriate for the participants? Does the interview protocol indicate the type and amount of prompting and probing that was permitted? Are the qualifications and special training of the interviewers described?

Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 31 Evaluating Types of Reports Interview Studies Is the method used to record responses described? Did the researcher use the most reliable, unbiased method of recording responses that could have been used? Does the researcher specify how the responses to semi-structured and unstructured items were quantified and analyzed?

Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 32 Evaluating Types of Reports Narrative Research Does the researcher provide a rationale for the use of narrative research to study the chosen phenomenon? Is there a rationale for the choice of individual to study the chosen phenomenon?

Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 33 Evaluating Types of Reports Narrative Research Does the researcher describe data collection methods and give particular attention to interviewing? Does the researcher describe appropriate strategies for analysis and interpretation (e.g., restorying)?

Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 34 Evaluating Types of Reports Ethnographic Research Does the written account (i.e., the ethnography) capture the social, cultural, and economic themes that emerged from the study? Did the researcher spend a full cycle in the field studying the phenomenon?

Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 35 Evaluating Types of Reports Case Study Research Was the phenomenon under investigation appropriate for investigation using a case study research method? Is there a rationale for the selection of the case (i.e., unit of analysis)? Does the researcher provide a clear description of the case?

Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 36 Evaluating Types of Reports Case Study Research Was an appropriate analysis of the case, or cross-site analysis, conducted? Is there a clear link between the data presented in the case study and the themes that are reported?

Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 37 Evaluating Types of Reports Mixed Methods Research Did the study use at least one quantitative and at least one qualitative data research method? Did the study investigate both quantitative and qualitative research questions? Is a rationale for using a mixed methods research design provided? Is the type of mixed methods research design stated?

Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 38 Evaluating Types of Reports Mixed Methods Research Is the priority given to quantitative and qualitative data collection and the sequence of their use described? Are qualitative and quantitative data collection techniques clearly identified? Are the data analysis techniques appropriate for the type of mixed methods design? Was the study feasible given the amount of data to be collected and concomitant issues of resources, time, and expertise?

Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 39 Evaluating Types of Reports Action Research Does the area of focus involve teaching and learning in the researcher’s own practice? Was the area of focus within the researcher’s locus of control? Is the area of focus something the researcher was passionate about?

Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 40 Evaluating Types of Reports Action Research Is the area of focus something the researcher wanted to change or improve upon? Does the researcher state questions that were answerable given the researcher’s expertise, time, and resources? Does the researcher provide an action plan detailing the effect of the research findings on practice?

Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 41 Evaluating Qualitative Studies Validity and Reliability Threats to internal validity Did the researcher effectively deal with problems of history and maturation by documenting historical changes over time? Did the researcher effectively deal with problems of mortality by using a sample large enough to minimize the effects of attrition? Was the researcher in the field long enough to minimize observer effects?

Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 42 Evaluating Qualitative Studies Validity and Reliability Threats to internal validity Did the researcher take the time to become familiar and comfortable with participants? Were interview questions pilot tested? Were efforts made to ensure intra-observer agreement by training interview teams in coding procedures?

Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 43 Evaluating Qualitative Studies Validity and Reliability Threats to internal validity Were efforts made to cross-check results by conducting interviews with multiple groups? Did the researcher interview key informants to verify field observations? Were participants demographically screened to ensure that they were representative of the larger population?

Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 44 Evaluating Qualitative Studies Validity and Reliability Threats to internal validity Were data collected using different media (e.g., audiotape, videotape, etc.) to facilitate cross-validation? Were participants allowed to evaluate research results before publication? Are sufficient data presented to support findings and conclusions? Were variables repeatedly tested to validate results?

Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 45 Evaluating Qualitative Studies Validity and Reliability Threats to external validity Were constructs defined in a way that has meaning outside the setting of the study? Were both new and adapted instruments pilot tested to ensure that they were appropriate for the study?

Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 46 Evaluating Qualitative Studies Validity and Reliability Threats to external validity Does the researcher fully describe participants’ relevant characteristics, such as socioeconomic structure, gender makeup, level of urbanization and/or acculturation, and pertinent social and cultural history? Are researcher interaction effects addressed by fully documenting the researcher’s activities in the setting?

Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 47 Evaluating Qualitative Studies Validity and Reliability Threats to external validity Were all observations and interviews conducted in a variety of fully described settings and with multiple trained observers?

Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 48 Evaluating Qualitative Studies Validity and Reliability Reliability Is the researcher’s relationship with the group and setting fully described? Is all field documentation comprehensive, fully cross-referenced and annotated, and rigorously detailed? Were observations and interviews documented using multiple means (e.g., written notes and recordings)?

Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 49 Evaluating Qualitative Studies Validity and Reliability Reliability Was the interviewer’s training documented, and is it described? Was the construction, planning, and testing of all instruments documented, and are they described?

Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 50 Evaluating Qualitative Studies Validity and Reliability Reliability Are key informants fully described, and is information on groups they represent and their community status included? Are sampling techniques fully documented and sufficient for the study?