1. Today: Our process Assignment 3 Q&A Concept of Control Reading: Framework for Hybrid Experiments Sampling If time, get a start on True Experiments: Single-Factor Design

2. Assignment 2 problems? Did you have trouble with IV/DV identification? Did you provide enough detail to really answer the questions? Were you able to fully operationalizing the concept of the DV into the actual data that will be measured? Discrete/continuous (for the measurement) Scaling types What are descriptive statistics Which descriptive statistics are appropriate for the different types of data Estimation of reliability/validity of measurement (not the experiment)

3. Where we are in the process Have defined broad research question and identified hypotheses or very refined research questions that must be evaluated to answer the question (Ass 1) Have identified the variable that need to be manipulated/studied in order to evaluate the hypotheses and have thought about what kind of data will need to be measured (Ass 2)

4. Assignment 3 Start thinking about threats to validity – what will you have to justify in order to convince readers that you have a valid study that addresses the research question – Note: as of yet, you don’t have a study designed. But you should be thinking about possible study designs that might be appropriate Ass. 2 was descriptive stats (individual variables) – in Ass. 3, you need to think about which comparisons you need to make – Different measurements, single variable? Satisfaction before/after treatment? – Correlations between different variables? Satisfaction as it relates to a participant’s age? – Focus on describing the data – what tables/graphs would you use to show the relationships

5. Assignment 3: Q&A For five of the threats to EXPERIMENTAL validity covered in class, give an example of a possible threat to validity for your research question. Note: as you have not yet designed your experiment, this exercise is to identify possible threats under a possible experimental design. You can use different possible experimental designs for each threat. What comparisons should you make to explore the answers to your research question? That is, which of the variables you identified in Assignment 2 should be analyzed individually and which variables should be compared against each other (which do you think might have interactions)? For each variable, specify what tabular and graphical methods you would use to describe the data.

6. Later (starting to think about a study design): Assignment 4: Hypothesis testing & Control – Starting to think about statistical evaluation of the hypotheses: Stating alternate and null hypotheses Is each hypothesis one tailed or two tailed? If there was a type I error, what would be the conclusion statement based on the error? Type II? Where you will get the data How you will rule out alternate explanations for the results (give an alternate & show why it’s not viable) – The limitations to your study

7. Term paper: Ongoing action items Look at related work: – What papers will you cite when writing your research paper – Papers to motivate the problem – Papers that have used similar methodologies to answer similar types of questions – Papers that establish baselines in your area – Papers about methodologies – Papers that you should compare your results with (if you actually end up running the study)

8. Term paper: Ongoing action items Think about what kind of study is needed to be able to answer the research questions/hypothesis that you have identified – How many factors are you looking at? – How much control do you need? – What are your biggest validity concerns that need to be addressed? – What is the usual approach in your discipline? – …

9. The research paper We’ll review the full details next week Proposing a study to evaluate your research questions The assignments are the setup to get you thinking about the details In the paper, you will be writing them formally In the presentation, you will be presenting them Your ability to discuss alternatives and justify your chosen approach will be key to a good mark

10. Control Basic idea: isolate the effect of the independent variable on the dependent variable – Ensure there are no reasonable alternative causes

11. True Experiments

12. True Experiment Experimental control Control as many potential threats to validity as possible Random assignment of participants/data to conditions Could be within-subjects or between-subjects

13. Control True experiment = complete control over the subject assignment to conditions and the presentation of conditions to subjects – Control over the who, what, when, where, how Control of the who => random assignment to conditions – Only by chance can other variables be confounded with IV Control of the what/when/where/how => control over the way the experiment is conducted

14. Quasi-Experiment When you can’t achieve complete control – Lack of complete control over conditions – Participants/elements for different conditions come from potentially non-random pre-existing groups

15. Control Basic idea: isolate the effect of the treatment on the dependent variable – Ensure there are no reasonable alternative causes Basic design: control vs. experimental group Setting: – Field vs. lab – Experiment vs. non-experiment

16. Today’s reading A taxonomy and evaluation for developing 802.11-based wireless mesh network testbeds. Discussion of what is being controlled (and how) What have you found in your own research domain? How much control is required? How clearly are populations, experimental conditions, and sampling methods and other experimental decisions defined?

17. Controlled studies

18. Controlled Experiments Predict the relationship between two or more variables. Independent variable(s) manipulated by the researcher Dependent variable(s) depends on the independent variable(s) Typical experimental designs have one or two independent variables. Validated statistically & replicable. www.id-book.com18

19. Control True experiment = complete control over the subject assignment to conditions and the presentation of conditions to subjects – Control over the who, what, when, where, how Control of the who => random assignment to conditions – Only by chance can other variables be confounded with IV Control of the what/when/where/how => control over the way the experiment is conducted

20. Sampling

21. Allocation (sampling) Probabilistic: – Simple random sample, systematic sample, stratified random sample, cluster sample – Video describing and contrasting – http://www.youtube.com/watch?v=be9e-Q-jC-0http://www.youtube.com/watch?v=be9e-Q-jC-0 Non-Probabilistic – convenience sample – purposive sample (choosing to meet an identified need)

22. Sampling in the broader sense Simple random sample, systematic sample, stratified random sample, cluster sample Convenience sample, purposive Discuss: – Elements: Data… Participants…. Tasks… Times… – Will you be able to work with the whole population/all possible scenarios? Or will you have to sample? – What kind of populations will you be dealing with?