Observational Methods January 20, 2010

Today’s Class Survey Results Probing Question from Wed, Jan. 20 Observational Methods Probing Question for Fri, Jan. 22 Survey

Survey Results Only 3 responses  – I will be switching to a better technology for the next survey Mostly positive I’ve read through your comments, and will try to take them into account Quick comment – if you think a key term is being incompletely or unclearly defined, please let me know, and I will try to explain it better The concepts covered in the first class will be re-visited as we go, and perhaps will become clearer as we get more examples

Today’s Class Survey Results Probing Question from Wed, Jan. 20 Observational Methods Probing Question for Fri, Jan. 22 Survey

ENTITATIVE HOLISTIC ESSENTIALISTEXISTENTIALIST

Today’s Class Survey Results Probing Question from Wed, Jan. 20 Observational Methods Probing Question for Fri, Jan. 22 Survey

Observational methods: top level You don’t manipulate the research setting in any way You just look at what is happening, or what has happened, and you note it Later, you analyze the data

Observational Methods This class we will focus on quantitative observational methods – Where observations are noted in terms of a defined set of categories – Called “quantitative” (numerical) because of the analytic methods commonly used with this kind of data

Observational Methods This week we will focus on quantitative observational methods Next week we will focus on qualitative methods – Where observations are noted in a more free-form way that does not attempt to rigorously apply a defined set of categories, but attempts to understand cases in their full particularity

What is the research setting? Where the learning (or other phenomenon you want to study) is taking place

What is the research setting? Usually a real learning setting – Classroom – University computer lab – Dorm room – Workplace – Any others? Occasionally a lab setting – Needs to be justification for why the lab setting is valid

Types of observation Enhanced video coding Video coding Field observation Screen replay coding Text replay coding (and more)

High-level notes High-level methodological similarity Can be used for many of the same research goals Advantages and disadvantages to each

Types of observation (that we’ll focus on today) Video coding Field observation Text replay coding

Types of observation that we’ll focus on today Video coding Field observation Text replay coding

Let’s watch a video QIg&feature=related QIg&feature=related – From 0:00 to 1:15

What student behaviors did you observe? Please explain both what the behavior is, and how you could recognize it from the video Does anyone disagree with how a behavior is defined? Why?

List of behaviors Let’s briefly write down a list of behaviors seen

OK, let’s watch it again Please write every behavior you see that did not fit into the original coding scheme

What additional student behaviors did you observe? Please explain both what the behavior is, and how you could recognize it from the video Does anyone disagree with how a behavior is defined? Why?

What is the total list of behaviors?

Now let’s make a coding scheme A list of “interesting” behaviors, how they are defined, and a code letter Example: – Hint abuse (a type of gaming the system) – Clicking through hints at high speed until reaching the bottom-out hint, obtaining and entering the answer, and immediately moving on to the next problem step

Now let’s make a coding scheme Should not be infinitely long – eliminate – “Uninteresting” behaviors – be guided by your intuition prior theory and empirical results – “Extremely rare” behaviors – under approximately 1 in 200 – Behaviors occuring between 1-5% of the time are often very interesting behaviors! – Effects on learning are often disproportionate Gaming the system occurs 3-5% of the time, and correlates to learning at

Which behaviors in your list Are “uninteresting”? Are “extremely rare”?

Now let’s make a coding scheme And combine related behaviors – For example, different types of gaming the system (e.g. hint abuse, systematic guessing, intentional rapid mistakes) often get classified together common – antecedents (poorly known skill) – characteristics (very fast responses) – results (poor learning gains) differences are no longer considered theoretically interesting (some debate on this still) tend to co-occur

Which behaviors Should be combined?

Our Coding Scheme

OK, let’s watch it for the 3 rd time Each person in our class will be assigned a student in the video (minimum of 2 of us for each 1 of them) Every five seconds of video, I am going to stop the video, and you need to write down the best code for the previous 5 seconds – If 2 behaviors observed, pick the dominant one – If you are not sure, put down a “?” – Do NOT talk to your partner

OK, now work with your partner Find out what % of time you agreed Find out where and how you disagreed There are better ways to assess inter-rater reliability, we’ll discuss these next class

Each group’s % error

What categories did you disagree on?

OK, now let’s watch Same video, but times 1:06 to 2:10 Observe the same student Follow the same procedure as before

Were the behaviors the same in this segment? It is important to: – Develop your coding scheme based on sufficient initial qualitative observation (and/or theory and prior work) – Note behaviors that do not fit in your coding scheme

OK, now work with your partner Find out what % of time you agreed Find out where and how you disagreed

Across sessions Did your agreement go up or go down? By how much?

A good strategy Repeatedly code the same data separately and compare Until your inter-rater reliability is “high enough” (more next time) And then code different data separately

Incidentally There is nothing “magic” about the video coding method we used It was largely selected for tractability in class

Other things we could have done Watch the same time segment multiple times Advantages Disadvantages

Other things we could have done Watch the same time segment multiple times Advantages: Greater certainty in coding Disadvantages: Takes a lot longer; allows second-guessing and hair-splitting

Other things we could have done Code when the behavior changes, not on a pre-selected time interval Advantages Disadvantages

Other things we could have done Code when the behavior changes, not on a pre- selected time interval Advantages: Get better assessment of the duration of each behavior Disadvantages: Enforces coding one student at a time; some code switches do not have “hard” edges (especially in case of coding affect)

Other things we could have done Code the first action in a time window, not the predominant one Advantages Disadvantages

Other things we could have done Code the first action in a time window, not the predominant one Advantages: Less ambiguity and decision- making Disadvantages: May be less representative; very brief behaviors may be missed

Other things we could have done What else?

A limitation of one camera

What is she doing?

Other ways to use video Multiple cameras Addresses issue of the student off-screen Synchronization becomes a challenge Time to code increases with each camera you add

Other ways to use video Individual student video Often from web-cams on top of machines

Example

his-affect-dialogue.wmv?ticket=t_dPrPbv8H his-affect-dialogue.wmv?ticket=t_dPrPbv8H

Note that The specific D’Mello study we read was a lab study, but the same method has been used in classrooms (without the emote-aloud) (cf. Arroyo et al, 2009) I chose the D’Mello paper in part because of the very clear description and comparisons of alternatives

Other ways to use video Other thoughts?

Types of observation (that we’ll focus on today) Video coding Field observation Text replay coding

Methods Similar, but live and in the field rather than via video Different challenges Some advantages

Sampling One challenge that emerges in field observations is sampling

Sampling Usually not desirable to follow only one student throughout the entire observation period Usually not tractable to have one observer per student Usually not possible to fully code multiple students at the same time

Field Sampling Strategies Switch between students in pre-determined order according to pre-determined schedule Watch or public timekeeper (image from Rodrigo et al, 2008)

Field Sampling Strategies Focus on a specific behavior and code it whenever it occurs, including which student engaged in the behavior

Observer effects Another challenge in live observation is observer effects A person hovering over your shoulder will impact your behavior

Observer effects: What you can do Obfuscation strategies Desensitization strategies

Obfuscation strategies Don’t appear to be observing the student you are actually observing – Observation from a distance – Side glances – Peripheral vision observation

Obfuscation strategies Don’t appear to be observing the student you are actually observing – Observation from a distance – Side glances – Peripheral vision observation Easiest to do Depends on classroom layout Not possible to assess some things

Obfuscation strategies Don’t appear to be observing the student you are actually observing – Observation from a distance – Side glances – Peripheral vision observation Not always effective

Obfuscation strategies Don’t appear to be observing the student you are actually observing – Observation from a distance – Side glances – Peripheral vision observation Takes training, but is the most powerful

Desensitization strategies Spend a class period walking around the classroom and noting random symbols in your clipboard Whenever a student asks you what you’re doing, show them the clipboard and explain that you’re just there to study the software for your professor and won’t be giving any information to the teacher After a certain point, they will decide you’re not important and begin to behave in normal (frighteningly anti-social) ways

Advantages of field observations

Faster to conduct About 3x faster (Baker, Corbett, & Wagner, 2006)

Faster to conduct Go out to class Versus Go out to class, then analyze in lab Also – The ability to stop and re-watch is frequently used in video coding

No camera & microphone limitations You can stand close enough to really see & hear what’s going on If something is obscured you can change your position

What is she doing? What is his facial expression?

With video… You have to get the data collection right, even before you’ve had time to do analysis

With field observations… You can quickly see whether you’re missing key information, and adjust your method (Note: adjusting your method in the *middle* of a study is a bad idea. Better to adjust your method, and restart, and just lose the first day’s data)

Major limitation Impossible to change coding categories to follow up on a question that emerges when you analyze the data Impossible for data to be re-used by alternate research group with different coding scheme

Other thoughts on field observation?

Types of observation (that we’ll focus on today) Video coding Field observation Text replay coding

Text replays Pretty-prints of student interaction behavior from the logs

Examples

Sampling You can set up any sampling schema you want, if you have enough log data 5 action sequences 20 second sequences Every behavior on a specific skill, but other skills omitted

Sampling Equal number of observations per lesson Equal number of observations per student Observations that machine learning software needs help to categorize (“biased sampling”)

Major Advantages Both video and field observations hold some risk of observer effects Text replays are based on logs that were collected completely unobtrusively

Major Advantages Blazing fast to conduct – 8 to 40 seconds per observation

Notes Decent inter-rater reliability is possible (Baker, Corbett, & Wagner, 2006) (Baker, Mitrovic, & Mathews, under review) Agree with other measures of constructs (Baker, Corbett, & Wagner, 2006) Can be used to train machine-learned detectors (Baker & de Carvalho, 2008) (Baker, Mitrovic, & Mathews, under review)

Major Limitations Limited range of constructs you can code Gaming the System – yes Collaboration in online chat – yes (Prata et al, 2008) Frustration, Boredom – sometimes Off-Task Behavior outside of software – no Collaborative Behavior outside of software – no

Major Limitations Lower precision (because lower bandwidth of observation)

Other thoughts on text replays?

Other Quantitative Observation Methods

When? When might you want to use some of the other methods listed here?

Any thoughts? About some of these related methods?

Not as common I only know of a 2 papers using screen replay in education – That group (Helen Pain and her students) now uses transcript coding, field coding, and video coding, but don’t use screen replays anymore I don’t know of any group coding using Super- Fidelity data, although D’Mello and Arroyo both use that kind of data in machine learning

High-level thoughts on quantitative observation methods

Generalizability Important to think about where you data came from, and what conclusions you can draw

What can you conclude? Data on one class day Data on one lesson Data on full year Data from 1 student Data from 10 students Data from 1 class Data from 1 school Data from 3 schools with different profiles Data from 100 schools

Feasibility of collection for each method Data on one class day Data on one lesson Data on full year Data from 1 student Data from 10 students Data from 1 class Data from 1 school Data from 3 schools with different profiles Data from 100 schools

Things you can code Teacher behaviors Student individual behaviors Student collaborative behaviors Student affect What else?

What method(s) work for each of these? Teacher behaviors Student individual behaviors Student collaborative behaviors Student affect What else?

Coding teacher behavior kPM kPM

Standard coding schemes It is worth noting that many research areas have standard coding schemes that have been developed and refined across many research projects, and are used by many researchers – Lauren Resnick and Sarah Michaels’s accountable talk (strategies used by students and teachers) – Baker et al: engagement-related behaviors in classrooms using educational software – D’Mello et al: affect in educational settings

Advantages to using these schemes You don’t need to defend your choice of scheme in your write-up Easier to compare your results to previous results and identify what’s new/different

Disadvantages to using these schemes You insulate yourself to some degree from obtaining surprising/new/exciting results

Which types of research programs would use these methods?

ENTITATIVE HOLISTIC ESSENTIALISTEXISTENTIALIST

ENTITATIVE HOLISTIC ESSENTIALISTEXISTENTIALIST

Today’s Class Survey Results Probing Question from Wed, Jan. 20 Observational Methods Probing Question for Fri, Jan. 22 Survey

Probing Question For Friday, you will read D'Mello, S., Taylor, R.S., Graesser, A. (2007) Monitoring Affective Trajectories during Complex Learning. Proceedings of the 29th Annual Meeting of the Cognitive Science Society, Which used data from a lab study If you wanted to study affective transitions in real classrooms, which of the methods we discussed today would be best? Why?

Today’s Class Survey Results Probing Question from Wed, Jan. 20 Observational Methods Probing Question for Fri, Jan. 22 Survey

Please complete this survey, and put it in this envelope

The End