Week 7 Advanced Research Design Research Methods Week 7 Advanced Research Design

Review - The t-test T-test returns a value – p What is the probability the means are the same? Usually we hope for the probability to be low – less that 5% (p = .05) Example – Assume we have had a group of students take a test, and then study a website and take the test again (paired sample). If the mean score for the first test was 57%, and the mean score on the second test was 60%, we want to be sure the second mean was not just a random event, but was less than 5% the result of random variation.

The t-test Process – we calculate the two means and the standard deviation for each sample. Excel makes the process simple: Formulas – more functions – statistical functions – t.test Array 1 – the results of students on the first test Array 2 – the results of students on the second test 1 tail (look for change at one end of the distribution) 1 for paired sample (2 if independent sample (like our cars)) If the number returned is <.05 we believe the two means are different (the students learned something by using the website.)

Advanced Research Design Hypothesis testing Research Designs Before and after without control After-only with control Before and after with control Completely randomized design Randomized block design

Before and after without control Commonly used with medical and educational research. I am concerned with one group. I treat all of them. I look for changes – more learning, less disease. Risk – what if the experimental sample was unusual? “Hawthorne effect” – false positive from the attention Applications – Monitor a work team after training, or after change in leadership

After-only with control I treat one group but not another, and check on differences in performance. Risk – control must be exact. Extraneous variables could easily account for the difference. Applications – I have two groups of identical students. One uses a website for math, one continues to use the standard textbook. I see who has the best test scores at the end of the year.

Before and after with control Two equivalent groups are identified. I test them both, then treat one group and then test again. Risks – low. Initial testing will help establish that the two groups were equivalent. Application – I identify two ERP implementation groups. They are equivalent in leadership and education. I test their ERP knowledge. One group uses software manuals, and one groups gets 3 days of training. I test their ERP knowledge again.

Completely randomized design Individuals within a sample are randomly assigned to two groups. One group is treated; one group is not. Post-treatment testing determines success. Risk – Random assignments might not be equivalent (especially for small numbers) Application – HR files are randomly assigned to two outsourcing companies. One company asserts it uses better trained (and more expensive) employees.

Randomized block design Individuals are put in multiple groups (blocks) and take some action in a random order. Application – multiple groups and treatments

Report #2 – Due March 24 You will approach the assignment in this manner: State an hypothesis involving two groups or conditions. Gather data points. Graph your data using a bar graph Describe your data using mean, median, mode, and standard deviation Determine if the two groups or conditions are statistically different. Explain if your hypothesis was proven or not.  

State an hypothesis involving two groups or conditions. Possible Research designs Before and after without control – After-only with control Before and after with control Completely randomized design Randomized block design

State an hypothesis involving two groups or conditions. Before and after without control Hypothesis – HR records can be updated more quickly by outsourcing companies. Methodology – Measure time it took to update HR records before outsourcing, and time it takes now. (Should be same quantity, same type)

State an hypothesis involving two groups or conditions. Before and after without control Hypothesis – Work group satisfaction is improved when team leaders receive training in project management. Methodology – Measure results of a satisfaction survey completed before team lead goes to training and again after training.

State an hypothesis involving two groups or conditions. After-only with control Hypothesis: Senior management support helps projects succeed. Methodology: Two work groups with equivalent talent and experience are assigned two equivalent tasks. The senior manager regularly attends meetings of one group but not of the other. Project evaluation results determine if one team is more successful.

State an hypothesis involving two groups or conditions. After-only with control Hypothesis: One hour exposure to Khan Academy videos each week help students learn algebra better. Methodology: Two classes with equivalent skills study algebra. One class listens to lectures and uses Khan academy videos. One class just uses lectures. Exam results show if one class is more successful.

State an hypothesis involving two groups or conditions. Before and after with control Hypothesis – Educational websites that involve frequent student engagement are more successful in student learning. Methodology – Student knowledge of a subject is tested. Students of equivalent ability are put into two groups. One group uses presentation-only websites. The other group uses a website that involves frequent student interaction. Students are tested again after one month.

State an hypothesis involving two groups or conditions. Before and after with control Hypothesis – Teams educated in project management principles will be more successful than teams without. Methodology – Two teams with equivalent talent and experience are assigned a project task. Each team creates a project plan. One team is allowed to manage the project as they wish. The other team is trained in PMI methodology. Both projects are monitored. Project success is measured in six months.

Completely randomized design Hypothesis – Outsourcing companies process HR records more accurately than local workers Methodology – Based on a random number generator, 50 records are collected from local workers, and 50 from an outsourced company. Records are checked for accuracy.

Completely randomized design Hypothesis – Websites that involve sound are more educationally effective than websites that are visual only. Methodology – Two websites are selected. They are matched for content (e.g. both teach the same algebra concept), but one uses audio and one does not. 30 students are randomly assigned to the two websites. They are tested at the end of 10 hours of instruction.

Randomized block design Hypothesis – Khan Academy websites are more educationally effective than other brands. Methodology – 5 websites from 5 companies are selected. All claim to teach the same concept. Students are grouped by ability and then randomly assigned the various websites, using them in a random order. Student knowledge is tested after each website.

State an hypothesis involving two groups or conditions. Report #2 State an hypothesis involving two groups or conditions. Gather data points. Write a possible hypothesis Select a hypothesis testing research design Create a methodology