Research Experience for Teachers (RET) Research Methods Dr. Randa L. Shehab Dr. Chen Ling

Overview Research Methodology Technical Writing and Presentation Problem Formulation Development and Measurement of Performance Criteria Experimental Design Data Collection and Analysis Presentation of Results Technical Writing and Presentation Introductions Sign-up List Review Syllabus: Note Project Deadlines Email Alias for Class Members Office Info Course Description The purpose of this course is to introduce the student to the techniques of experimental methodology through the vehicles of ergonomics and human performance measurement. Experimental methodology includes problem formulation, the development and measurement of human performance criteria, experimental design, data collection, data analysis, and the presentation of results. Problems in the study of human performance will be discussed along with the proper techniques for writing and presenting a technical report. As part of the course, students will be required to conduct an experimental investigation of some relevant aspect of human performance. This research project will require a significant contribution of time and the amount of credit given will reflect this contribution. RET 2006 Summer

The Scientific Method A systematic approach to answering questions using empirical investigation Steps Form a hypothesis - an idea that can be tested through observation or experimentation Collect data through observation Analyze the data - test the hypothesis “Do not reject” or “reject” the hypothesis Confirm results by repeating the experiment RET 2006 Summer

The Research Environment Naturalistic Observation Systematic, detailed observation of behavior Purely descriptive, no causality Laboratory Research Arbitrary environment where elements in environment can be controlled Control allows statements about causality Naturalistic Observation Journal, time sample, etc. e.g., observe wild animals in natural environment, children on playground, guards at a prison, assembly workers on the job RET 2006 Summer

Basic Research Acquisition of knowledge for developing or validating a theory Not necessarily for practical application Used to set forth general design principles Examples Nature of human performance? How is performance affected by variable(s)? Relationship between variables? Impact on system performance? Example: Working in a Heat Stress Environment Basic Research: Exercise at various levels of heat stress and monitor core temperature and heart rate for sole purpose of learning physiological responses (physiologist). Set Forth General Design Principles Applied Research: Perform actual work task while monitoring core temperature and heart rate with goal of setting standards such as OSHA (NIOSH) exposure limits. Evaluate Specific Design Alternatives Example: Texas Instruments LED Watch Flick wrist to turn on product. Perform experimentation to evaluate various response times. Other elements: display brightness, on time, size of digits RET 2006 Summer

Applied Research Examples Finding answers to practical problems Highly specific hypothesis Absence of general knowledge or theory advancement Used to evaluate specific design alternatives Examples Effect of technology on human performance? Effect of behavioral variables on use of technology? Most effective behavioral/technological variable to achieve desired performance? Example: Working in a Heat Stress Environment Basic Research: Exercise at various levels of heat stress and monitor core temperature and heart rate for sole purpose of learning physiological responses (physiologist). Set Forth General Design Principles Applied Research: Perform actual work task while monitoring core temperature and heart rate with goal of setting standards such as OSHA (NIOSH) exposure limits. Evaluate Specific Design Alternatives Example: Texas Instruments LED Watch Flick wrist to turn on product. Perform experimentation to evaluate various response times. Other elements: display brightness, on time, size of digits RET 2006 Summer

Why Research? To understand behavior and answer questions about people, environments, systems To evaluate the research of others We should change the system to make it more effective… Determine if research is done correctly and if it applies To conduct research of your own Research is a tool for answering questions RET 2006 Summer

Learning to Perform Scientific Research Scientific Research is Unique Acquire Research Experience Perform Solid Literature Review Use Proper Tactics and Strategy Avoid Mistakes of Other Investigators RET 2006 Summer

General Scientific Method: Steps Development of Hypothesis Body of Existing Knowledge Define the problem Controlled Experimentation Observation of Phenomena Scientific curiosity Quantification of Observations Analyze the problem Logical Analysis Test of Hypothesis RET 2006 Summer

Research Experiments Experiment: A series of controlled observations taken in an artificial situation with deliberate manipulation of variables to answer one or more specific hypotheses. Controlled Observations Conditions and events Must be repeatable! Artificial Situation Allows accurate observations, but might affect behavior Controlled Observations - Control variables that might affect the results: - Conditions: Environmental; Everything must be the same for each subject during each trial, No confounding factors - Events: No outside influences, distractions; No differences in test methodology (time of day, noise levels, etc…) - Must be able to replicate the experiment! Verify and Validate! Artificial Situation - Contrived Conditions and Circumstances; Not naturally occurring. - Unfortunately, people tend to behave artificially when they know they are being observed. Might respond differently than they normally would. (Hawthorne Effect: Workers performance improves merely because of the added attention => increased motivation). RET 2006 Summer

Research Experiments Manipulation of Variables Specific Hypotheses Systematically vary conditions to see effects Eliminate extraneous factors to determine causes Specific Hypotheses Allows clear definition of the experimental plan Ideas can be obtained from observation “Effects of (independent variable) on (measured variable)” “Comparison of the effects of (different technologies / products) on (performance measure)” Manipulation of Variables - Systematically vary ONLY the factors that you wish to evaluate. - Avoid CONFOUNDING - if you vary more than one factor at a time, you can’t be certain what was responsible for any effect you may observe. - Control all other variables that might have an effect on the outcome. Specific Hypotheses - Starting point of all your research => Very Important! - Proposes an explanation that can be tested. - State clearly what your topic of interest is and what the independent and dependent variables are. RET 2006 Summer

Developing a Hypothesis A hypothesis states an idea that can be tested through experimentation or observation The hypothesis proposes an explanation for some observed phenomenon Initial Interest Literature Review Hypothesis guides inquiry and directs thinking Working Hypothesis Formal Hypothesis RET 2006 Summer

Types of Hypotheses Working Hypothesis Exploration Formal Hypothesis Preliminary statement based on limited information Informal statement derived from initial observation Subject to modification Exploration Obtain additional information on the topic Books, journals, related knowledge, collaboration Formal Hypothesis Precise expression of a predicted relationship between or among events Capable of being tested Does not change Working Hypothesis - Oranges are larger than apples. Formal Hypothesis - Defines the concern of our research - The average diameter of oranges is larger than that of apples - In a statistical test, this becomes our alternate hypothesis Null Hypothesis - The average diameter of apples and oranges is the same - You can prove a relationship does not exist, but you can never prove that one does. RET 2006 Summer

Key Principles of Experimental Design Provide a Measure of Random Error Design must contain a measurable estimate of variation to determine statistical significance Provided through replication Avoid Systematic Bias Effects due to time-ordering such as learning Counterbalance - perform half the trials using one sequence and the other half using another Randomize - assign units randomly Do Not Confound Variables Change only one variable at a time Provide a Measure of Random Error - Design must contain a measurable estimate of variation to determine statistical significance - Provided through replication Repeated measurement of the same experimental unit (less variability but not as generalizable) Measurement of several experimental units (more generalizable, but higher variability) Avoid Systematic Bias - Effects due to time-ordering such as learning; Occurs when there is a dependence between measured values that follows some pattern - Counterbalance - perform half the trials using one sequence and the other half using another - Randomize - (if you cannot counterbalance) assign units and trials randomly Do Not Confound Variables - Two variables are confounded when their effects cannot be isolated. The levels of two factors change simultaneously. - Change only one variable at a time RET 2006 Summer

Defining the Problem Easiest way is to define the variables A variable is a quantity that may assume any one of a set of values Independent Variable Variable that is deliberately manipulated Must have at least 2 levels Forms the basis of the hypothesis RET 2006 Summer

Independent Variables The factor or treatment Presence/Absence Does the variable impact behavior? Quantitative Continuous along a single dimension Must have enough levels to describe expected relationship Must consider range and spacing of levels Presence / Absence: Non-zero value in one condition, Zero in the other. Does the variable impact behavior or performance? - Quantitatively: Choose levels along a single but continuous dimension Select a wide enough range to differentiate effects between the levels - RET 2006 Summer

Independent Variables Qualitative Combination of multiple factors not described with a single dimension Vary one dimension and control the others Select “optimal” values for all variables held constant Test at range of values to identify the optimal points Qualitatively: Combination of multiple factors that cannot be described with a single dimension. (shape - three dimensional; color - contains both hue and shade characteristics (light)) RET 2006 Summer

Quantitative Variables Number of Levels Choose enough levels to get an accurate view of the relationship between the variables Cost tradeoff when adding more levels Total Trials = (# IV's)  (IV levels)  (repetitions) Actual Relationship Dependent Variable Independent Variable Level 1 Level 2 Level 3 Suspected Relationship RET 2006 Summer

Quantitative Variables Range of Values Wide enough to cover region of interest Start large and refine (pilot study) Spacing Should match phenomenon being investigated Equal: Better for statistical analysis Unequal: Often more representative of system response (e.g., decibels) RET 2006 Summer

Qualitative Variables Example When comparing control shapes, select the best size, texture, etc. for each shape Compare Round A with Square C Fundamental Principle Select the best levels of all other variables (usually quantitative) to be held constant, then use the best combination at each qualitative level Alternatively, test across a broad range of values to locate optimum levels Dependent Variable Knob Size A B C Round Square RET 2006 Summer

Dependent Variables The criterion measure Variable that is measured Value “depends” on the value of the independent variable Selection of the criterion determines the outcome Does it answer the question? Is it relevant to the task? RET 2006 Summer

Control Variables Examples: Variables that are held constant because they may affect the results Avoid Confounding Uncontrolled influences on experimental results Interpret results within a narrow limit to determine causality Examples: Time of day Attitudes/expectation Instruction RET 2006 Summer

Variable Examples Effect of Alcohol on Reaction Time Independent Variables Alcohol Level / Consumption (0%, .08%, 1.2% BAC) Gender (Female, Male) Dependent Variables Errors Processing Time Motor Time Control Variables Food Intake Environmental RET 2006 Summer

Characteristics of Good Performance Measures Appropriate Level of Detail Measure appropriately reflects differences to detect Adequate precision and range to detect differences Reliability Degree of repeatability (can be expressed as a correlation coefficient) Interrater reliability – the degree to which multiple observers agree when scoring the same event Validity Degree to which a measure actually measures what it is supposed to measure Ensures a measure tells you “what” your data mean and that you have selected an appropriate response variable RET 2006 Summer

Characteristics of Good Performance Measures Sensitivity Can detect changes in the behavior of interest Accuracy More precise than the phenomena being measured Minimize measurement error Non-Intrusiveness Collection method does not affect performance Observer should not be a distraction Data collection should not interrupt the task (i.e., filling out forms) Measuring device should not attract participant's attention Implementation Requirements Practical with respect to time, budget personnel requirements, ease and quality of collection, and measurement robustness RET 2006 Summer

Populations and Samples Sample: subset of the population Population: all relevant cases Population - Represents all relevant cases in the entire data set - Finite, but may be very large Sample - Manageable subset of the population. - Very Important: Should represent the population Sampling Methods - To ensure representative samples Sampling Methods Random Sampling Stratified Sampling Matched Sampling Fortuitous Sampling Proportionate Stratified RET 2006 Summer

Random Sampling Randomly select members of population Independent Representative of the population for large sample sizes Reduces systematic bias Randomly assign subjects to conditions How would you randomly sample n=20 to study ability of drivers to localize warning sirens? Random Sampling - Define the population of test units and randomly sample - Each element has an equal probability of being sampled - Random numbers table: a large table of numbers used to assign participants randomly to treatment conditions - Sample is representative of the population when a large number are selected - Random assignment reduces bias Fortuitous Sampling - Take whatever you can get! RET 2006 Summer

Fortuitous Sampling Use whatever is available Does not guarantee independence Not representative of the population How would you fortuitously sample n=20 to study ability of drivers to localize warning sirens? RET 2006 Summer

Stratified Sampling Identify critical subgroups of the population Randomly select members for subgroup samples based on similar critical characteristics Mirrors population characteristics Accurate information is not always available How would you develop a stratified sample of n=20 to study ability of drivers to localize warning sirens? RET 2006 Summer

Proportionate Stratified Sampling Identify critical subgroups of the population Randomly select members for subgroup samples until proportionate to the population characteristics Mirrors population characteristics and count How would you develop a proportionately stratified sample of n=20 to study ability of drivers to localize warning sirens? RET 2006 Summer

Matched Sampling Experimental samples are identical with the exception of the independent variable Controls for extraneous variables Allows strong conclusions to be made about any significant differences Equating groups Precision matching - match identical participants Range matching – match participants within a range Average matching - match average group score How would you develop a matched sample of n=20 to study the effects of different work hardening programs on material handling endurance? Matched Sampling - Controls for Extraneous Variables: variables not controlled that may affect the outcome of the experiment (i.e., age, gender) - Match participants on these extraneous variables - Attempts to create identical samples to ensure that observed differences are due only to treatment - Equating Groups: Matches groups along a particular dimension (common features or participant characteristics) - Precision Matching: Pairs of identical participants are randomly assigned to treatment conditions. - Range Matching: Pairs of participants falling within the defined range are randomly assigned to treatment conditions. Problems: - Difficult to determine precise values. - Can’t always find matches, so discard participants ==> - Participant selection bias could affect generalization. - Yoking Procedure - Matches participant sequences to control for temporal and sequential effects. - Randomized Block Designs - Includes an extraneous variable in the design as an independent variable. Randomized assignment to treatments within blocks (levels of the blocking factor). RET 2006 Summer