Today’s Agenda Quick Review Validity/Reliability/Generalizability Begin Discussion on Ways to Collect Data Observations Field Notes Documents Sociogram

Qual or Quant or Both? Deals with how/why Deals with what Based on empirical evidence Analyzed with statistical tests Researcher is the instrument Small, purposive sample Valid and reliable

Reliable What is meant by reliability? Demonstration

Validity What is meant by valid? Demonstration

Issue Quantitative: use statistical measures to help determine validity and reliability Qualitative: we need to convince the reader that the data and analysis are valid and reliable

Reliability Are we reporting what is truly being observed Is what is being observed typical/true behavior The researcher must come across as unbiased and not subjective HOW? Support observations (videos, tape recordings, multiple observers) Address observer bias upfront How do you fit into the context

Validity Matter of fit: does the method of data collection match the problem; are claims supported How? Supporting quotes Supporting examples Triangulation

A X B Triangulation

Triangulation Used to support validity and reliability Use multiple data sources (interviews, lesson plans, observations) to show that the data collected are an accurate/true representation of what you are studying Achieved by using multiple researchers (doesn’t work for us), multiple subjects (collecting multiple viewpoints about the same topic)

Example Text example on Student Reflections Let’s take a closer look at the OCEPT project Interested in whether student teachers were using reformed based teaching practices in their math and science classes

OCEPT Methods of data collection Three observations with an observation protocol and global scan notes Interview Periodically used multiple observers to find interrater reliability with the protocol Multiple student teachers From multiple programs (See overhead)

By presenting data from multiple sources, subjects and researchers, our findings were believed to be an accurate reflection of what was occurring in the classrooms.

Generalizability Along with validity and reliability, generalizability is discussed Qualitative studies are not designed to be generalizable (what you find is not meant to apply to other settings) so this is a non-issue!

One more note If something is valid is it necessarily reliable? If something is reliable is it necessarily valid? Consider this simple example: Assign points for filling in the name on a test. Is that reliable? Is that valid? Can something be reliable and not valid? Can something be valid and not reliable?

Moving on to data collection

Participant/Observer Continuum of participation from complete participant to complete observer Complete Complete Participant Observer We often fall somewhere in-between

Considerations Quality of direct on-site observation (is your presence affecting normal behavior, are you having an observer effect ) Freedom of access for unbiased observations (can you visit any time, speak to whomever…) Intensity of observations (need many/long observations for validity) Observer bias Halo effect

More considerations Sampling of data (what is needed from whom/when—example, if examining how a teacher teaches may need to observe planning meetings, department meetings, classroom visits) Subtle cues – be alert to unobtrusive signals (is teacher always late to your meetings) Will you get too involved?

And still more Non-participant/observer – minimizes interaction with subjects being observed Less likely to become emotionally involved May be difficult to really get a handle on perception of subjects Hard to be unobtrusive In cases where you are studying your own students and often your own teachers, you are already involved and so are a participant.

Observations as Data Collection Field Notes-description of the setting and what’s going on Includes portraits of the subject Reconstruction of dialog Description of physical setting Account of particular events and activities Description of observer’s behavior Your reflection of the above

Field Notes Include personal reflective notes (O.C. observer comments—Why are these important?) Lederman example in text Overhead of another example

Taping May want to audiotape/videotape Hard to be unobtrusive Transcription What it covers/who it covers Doesn’t get the whole situation (all people, all gestures, entire class) Quality is a concern (even volume!)

Practice Purpose: Determine good teaching practices to use with ELL (ESOL) students. Observing teachers that have been identified as doing an excellent job with ELL students in their classes View the excerpts with an eye toward teaching practices used and student engagement.

Documents Personal documents of subjects Official Lesson plans Samples of student work Journals Power point presentations Official Minutes reports Student records and personal files Video-audio files Photographs

Considerations Typically work with photocopies Note how it was received, collected, when, etc. Do you need permission?

Personal Journals If you’re doing a case study, have subject keep a journal If you’re studying your own students/staff, keep your own journal Jot quick notes during/at the end of each class or meeting Set aside time later that same day to write a more detailed description of the day’s events

Sociogram Useful to understand relationships in a classroom or group Ask questions that student responds to using the name of another student Example: The purpose is to determine if there is a pattern to student self-selected groups

Sociogram Ask students: If you could work with anyone in the class, with whom would you work? Note 1st, 2nd, 3rd choices Graph names and lines showing choices See OH from Hubbard and Power

Limitations Can see stars and isolates BUT Cannot stand alone Don’t know the reasons behind the students choices

Closure What is triangulation? Why are field notes important? What are some key pieces of information noted in field notes? What are examples of documents that might be used as data?