1. Today’s Agenda Quick Review Validity/Reliability/Generalizability
Begin Discussion on Ways to Collect Data
Observations
Field Notes
Documents
Sociogram

2. Qual or Quant or Both? Deals with how/why Deals with what
Based on empirical evidence
Analyzed with statistical tests
Researcher is the instrument
Small, purposive sample
Valid and reliable

3. Reliable
What is meant by reliability?
Demonstration

4. Validity
What is meant by valid?
Demonstration

5. Issue
Quantitative: use statistical measures to help determine validity and reliability
Qualitative: we need to convince the reader that the data and analysis are valid and reliable

6. Reliability Are we reporting what is truly being observed
Is what is being observed typical/true behavior
The researcher must come across as unbiased and not subjective
HOW?
Support observations (videos, tape recordings, multiple observers)
Address observer bias upfront
How do you fit into the context

7. Validity
Matter of fit: does the method of data collection match the problem; are claims supported
How?
Supporting quotes
Supporting examples
Triangulation

8. A X B
Triangulation

9. Triangulation Used to support validity and reliability
Use multiple data sources (interviews, lesson plans, observations) to show that the data collected are an accurate/true representation of what you are studying
Achieved by using multiple researchers (doesn’t work for us), multiple subjects (collecting multiple viewpoints about the same topic)

10. Example Text example on Student Reflections
Let’s take a closer look at the OCEPT project
Interested in whether student teachers were using reformed based teaching practices in their math and science classes

11. OCEPT Methods of data collection
Three observations with an observation protocol and global scan notes
Interview
Periodically used multiple observers to find interrater reliability with the protocol
Multiple student teachers
From multiple programs
(See overhead)

12. By presenting data from multiple sources, subjects and researchers, our findings were believed to be an accurate reflection of what was occurring in the classrooms.

13. Generalizability
Along with validity and reliability, generalizability is discussed
Qualitative studies are not designed to be generalizable (what you find is not meant to apply to other settings) so this is a non-issue!

14. One more note If something is valid is it necessarily reliable?
If something is reliable is it necessarily valid?
Consider this simple example:
Assign points for filling in the name on a test.
Is that reliable?
Is that valid?
Can something be reliable and not valid?
Can something be valid and not reliable?

15. Moving on to data collection

16. Participant/Observer
Continuum of participation from complete participant to complete observer
Complete Complete
Participant Observer
We often fall somewhere in-between

17. Considerations
Quality of direct on-site observation (is your presence affecting normal behavior, are you having an observer effect )
Freedom of access for unbiased observations (can you visit any time, speak to whomever…)
Intensity of observations (need many/long observations for validity)
Observer bias
Halo effect

18. More considerations
Sampling of data (what is needed from whom/when—example, if examining how a teacher teaches may need to observe planning meetings, department meetings, classroom visits)
Subtle cues – be alert to unobtrusive signals (is teacher always late to your meetings)
Will you get too involved?

19. And still more
Non-participant/observer – minimizes interaction with subjects being observed
Less likely to become emotionally involved
May be difficult to really get a handle on perception of subjects
Hard to be unobtrusive
In cases where you are studying your own students and often your own teachers, you are already involved and so are a participant.

20. Observations as Data Collection
Field Notes-description of the setting and what’s going on
Includes portraits of the subject
Reconstruction of dialog
Description of physical setting
Account of particular events and activities
Description of observer’s behavior
Your reflection of the above

21. Field Notes
Include personal reflective notes (O.C. observer comments—Why are these important?)
Lederman example in text
Overhead of another example

22. Taping May want to audiotape/videotape Hard to be unobtrusive
Transcription
What it covers/who it covers
Doesn’t get the whole situation (all people, all gestures, entire class)
Quality is a concern (even volume!)

23. Practice
Purpose: Determine good teaching practices to use with ELL (ESOL) students. Observing teachers that have been identified as doing an excellent job with ELL students in their classes
View the excerpts with an eye toward teaching practices used and student engagement.

24. Documents Personal documents of subjects Official
Lesson plans
Samples of student work
Journals
Power point presentations
Official
Minutes
reports
Student records and personal files
Video-audio files
Photographs

25. Considerations Typically work with photocopies
Note how it was received, collected, when, etc.
Do you need permission?

26. Personal Journals
If you’re doing a case study, have subject keep a journal
If you’re studying your own students/staff, keep your own journal
Jot quick notes during/at the end of each class or meeting
Set aside time later that same day to write a more detailed description of the day’s events

28. Sociogram Useful to understand relationships in a classroom or group
Ask questions that student responds to using the name of another student
Example: The purpose is to determine if there is a pattern to student self-selected groups

29. Sociogram
Ask students: If you could work with anyone in the class, with whom would you work?
Note 1st, 2nd, 3rd choices
Graph names and lines showing choices
See OH from Hubbard and Power

30. Limitations Can see stars and isolates BUT Cannot stand alone
Don’t know the reasons behind the students choices

31. Closure What is triangulation? Why are field notes important?
What are some key pieces of information noted in field notes?
What are examples of documents that might be used as data?