1. Data Collection Data is your friend

2. Agenda Action research check-up Measures (aka, ways to collect data) Midterms

3. Public Service Announcement APA style guide websites http://owl.english.purdue.edu/owl/resource/560/01/ http://citationmachine.net/index.php?reqstyleid=2

4. Action Research Projects Where you should be Finding research related to, but not necessarily identical to, your topic Research questions may shift

5. Action Research Projects literature reviewOrganizing Sources for literature review Example Topic A ( Source A, C, G, H, K ) Subtopic 1 (Source G, K) Subtopic 2 (Source A, G, H) Subtopic 3 (Source A, C, H, K) Topic B ( Source B, D, E, F, G, H ) Subtopic 1 (Source B, F, G, H) Subtopic 2 (Source D, E, F, G)

6. Action Research Projects Collecting Data What types of data should I collect to answer my research question? What types of data should I collect to help answer the why questions in my study?

7. Measures of Data Collection Interviews Questionnaires Observations Tests

8. Measures (Means of Data Collection) You must match the instrument to the research question!

9. Examples to critique Measures Questionnaire – Psychological School Membership Survey used with middle school students Interview protocol – for teachers & counselors regarding professional development issues Observation instrument – PDE 430 for student teachers What are 2 benefits and 2 limitations of this measure?

10. Interviews What are some important things happening in this video related to interviewing? http://www.youtube.com/watch?v=eAfTHwVTOrY

11. Interviews Advantages Establish rapport & enhance motivation Clarify responses through additional questioning Capture the depth and richness of responses Allow for flexibility Reduce “no response” and/or “neutral” responses Disadvantages Time consuming Expensive Small samples Subjective – interviewer characteristics, contamination, bias

12. Questionnaires Used to obtain a subject’s perceptions, attitudes, beliefs, values, opinions, or other non-cognitive traits Example of psychology questionnaires measuring aspects of happiness, “signature strengths,” well-being http://www.authentichappiness.sas.upenn.edu/

13. Questionnaires Scales - a continuum that describes subject’s responses to a statement Likert Checklists Ranked items

14. Questionnaires Likert scales Response options require the subject to determine the extent to which they agree with a statement Debate over odd v. even number responses Statements must reflect extreme positive or extreme negative positions

15. Questionnaires Checklists Choose options Ranked items Sequential order Avoids marking everything high or low

16. Questionnaires Problems with measuring non-cognitive traits Difficulty clearly defining what is being measured Self-concept or self-esteem Response set Responding same way (Ex - all 4’s on CATS) Social desirability /faking “PC filter” Agreeing with statements because of the negative consequences associated with disagreeing

17. Questionnaires Controlling problems Equal numbers of positively and negatively worded statements Alternating positive and negative statements Providing confidentiality or anonymity to respondents

18. Designing Questionnaires Online resources www.surveymonkey.com http://pareonline.net/getvn.asp?v=5&n=3 http://www.peecworks.org/PEEC/PEEC_Inst/I0 004E536http://www.peecworks.org/PEEC/PEEC_Inst/I0 004E536 http://www.statpac.com/surveys/

19. Observations Observations - direct observations of behaviors Provide first hand account (ameliorates issues of self-reporting in questionnaires) Natural or controlled settings Ex – classroom vs. lab (child attachment studies) Structured or unstructured observations Ex – frequency counts vs. narrative record Detached or involved observers

20. Observations Inference Low inference - involves little if any inference on the observers’ part Student participation High inference - involves high levels of inference on the observers’ part Teacher effectiveness – PDE form 430

21. Observations Controlling observer effects Observer bias Training Inter-rater reliability (Cronbach’s alpha) Multiple observers Contamination - knowledge of the study influences the observation Training Targeting specific behaviors Observers do not know of the expected outcomes Observers are “blind” to which group is which

22. Observations Observer effects Halo effect - initial ratings influence subsequent ratings Hawthorne effect - increased performance results from awareness of being part of study Leniency - wanting everyone to do well Central Tendency - measuring in the middle Observer Drift - failing to record pertinent information

23. Tests PSSA Writing Assessment Purpose Domains Scoring Questions to Consider

24. Score Interpretation of Tests Know the norming group on a norm-referenced test Self-report information is not very reliable Inferences must be limited to what is being tested Grade equivalency scores should not be interpreted to indicate grade assignment Comparisons on a norm-referenced test can not be made to populations outside the given norming group.

25. Tests Standard scores - transformations of raw scores into easily interpreted standard metrics Z-score - how far away the data value is from the mean All standard scores are interpreted relative to the scores of others in the norming group Barbara’s SAT score of 700 is very, very good relative to the scores of the norm group because it is two (2) standard deviations above the mean (i.e., in the 99 th percentile)

26. Tests Standardized tests Uniform procedures for administration, scoring, and interpreting test scores Benefits and limitations? Validity and reliability?

27. Benefits Forces teachers to get students on track with specific standards; quality control; assures students getting skills Red flag for a poor teacher who is negligent Consistency across school districts; use scores as consistent measure Consistency of subject matter Accountability of teachers Find areas where students are lacking and teach to their deficits Force students to be accountable

28. Limitations “If test scores improve (as I believe they will) and parent and policy makers are pleased with the results, are children receiving a better education?” (Cuban, 1983, p. 696). Once we know the expectation we limit what we teach to just what is covered on the test; restricts content taught Teaching to the test Alignment between test and standard; is test valid? Which came first—the standard or the assessment? How to provide opportunities for remediation for those not meeting the standard without cheating the “smart” kids. Doesn’t measure creativity Leads to labels of students Bad test takers penalized

29. Validity & Reliability What are issues of validity and reliability that must be addressed in relation to standardized testing? PSSA issues

30. Standardized Testing Resources AERA statement about standardized tests http://www.aera.net/policyandprograms/?id=378 AERA “Research Points” http://www.aera.net/uploadedFiles/Journals_and_Publications/Resea rch_Points/RP_Spring03.pdfhttp://www.aera.net/uploadedFiles/Journals_and_Publications/Resea rch_Points/RP_Spring03.pdf Article by James Popham http://school.familyeducation.com/educational-testing/educational- philosophy/38778.htmlhttp://school.familyeducation.com/educational-testing/educational- philosophy/38778.html Dept of Education guide for policy makers and educators http://www.ed.gov/offices/OCR/archives/pdf/TestingResource.pdf