1. Power the ability to find a difference if there is one. – Effect size, alpha-level, sample size, (all increase power) variance (decrease power)

2. PTP 560 Research Methods Week 6 Thomas Ruediger, PT

3. Sampling How can we generalize our study to the world? Our sample responses are representative of population! Population – All members – All measurements possible Sample – Subgroup of members – Measurements actually taken Bias – Conscious – Unconscious (sub?)

4. Sampling Target Population Accessible population Sample Inclusion Criteria – Trait of the Target or accessible population – Qualifies someone as a subject – Restrictions here will limit ability to generalize Exclusion Criteria – Precludes someone being a subject – Excluded because they may interfere with interpreting findings Selection – Plan – Fig 8.2

5. Sampling Techniques Probability – Everyone in the population has an equal chance of being selected – Through random selection – Not the same as random assignment – Every member has equal chance of being selected – Considered (but not guaranteed to be) representative – Allows estimate of sampling error The difference between population average and sample average Non-probability – Non-random methods – Limits ability to generalize outcomes

6. Probability Sampling Simple random – Also known as sampling without replacement – Table of random numbers (Table 8.1) Systematic – Every 2 nd one, every other person, etc. Stratified random – Subsets (strata) established – Random selection from the strata – May also be proportional to amount in population. – May be more representative than random Grade in Class, Age, Fitness level.

7. Probability Sampling Disproportional – Select random samples of appropriate size – Correct it with proportional weighting Allows us to use a smaller sample size to project to the a larger population, through proportional weighting. Cluster – Successive random sampling – Convenient and efficient …………….BUT, increased sampling error – Example Area probability sampling Random digit dialing

8. Non-Probability Sampling Convenience – Also known as accidental sample – Consecutive sampling is common method – Self selection is a major limitation Quota – Enroll subjects (Ex: selecting by decade, fill up quota) – Stop for certain strata when they are represented Purposive – Hand picked by criteria – Prone to bias Snowball – Chain-referral: friends talk to friends

9. Recruitment Feasibility issues can be daunting Advertisements Other healthcare providers/institutions Track and report – Screened for eligibility – Number actually eligible – Number enrolled POWER – “The ability to find significant differences when they exist” – Important to know a priori to get appropriate sample size Higher alpha level Increase Sample Size Estimate Effect Size, compared to chart (30 typically the best per group)

10. Validity in Experimental Design Experiment has three essential characteristics: 1. Manipulation of independent variables 2.Random assignment to groups 3.Control or comparison group Supports (Does NOT prove) cause-and effect relationship -stronger design the better cuaseial reationship Extraneous variables – Must be controlled OR, – They can confound `

11. Handling incomplete (or lost) data On-Protocol (Completer) Analysis – Only those who complete the study – Tends to bias in favor of the treatment As there is a reason why they dropped out. Intentions to treat (ITT) PREFERRED approach – What did we intend to do? – More conservative than On-Protocol – Considered to reflect clinical situations – Analysis? Non-completer equals failure Last observation carried forward – When they drop out, carry their score forward as if no change (more conservative)

12. Validity in Experimental Design Blinding – Single Blind Subject blinded to treatment or placebo – Double Blind Subject and Tester blinded to treatment or placebo condition – Triple Blind Researcher, tester, and subject blinded Data analyzed by independent source

13. Controlling Inter-subject Differences Options – Homogenize on certain characteristic(s) – Manipulate attribute variables into “Blocks” – Consider matching – Use subjects as own control – Handle statistically with ANCOVA Table 9.1

14. Threats to validity Four Threats correspond to four major questions Is there a relationship between IV and DV? – Statistical Conclusion Validity Evidence of causal relationship? – Internal Validity Can results be generalized to a theoretical construct? – Construct Validity Can it be generalized to other persons/settings/times? – External Validity

15. Statistical Conclusion Validity Is there a relationship between IV and DV? Threats – Low Statistical Power – Violated Assumptions of Statistical Tests – Error Rate – Reliability – Variance (to small) – Failure to use ITT

16. Internal Validity Evidence of causal relationship? The extent to which the results of a study/experiment can be attributed to the treatment or intervention rather than to flaws in the research design Threats to internal validity – History – Maturation – Attrition – Testing – Instrumentation – Regression – Social Threats

17. Internal Validity Threats to Internal Validity – Testing Interactions Pre-tests or subsequent testing has an effect Second test scores tends to move toward the mean Standard Deviation decreases

18. Construct Validity Can results be generalized to a theoretical construct? Threats – Limits of Operational Definitions – Time Frame Within Operational Definitions – Multiple Treatment Interactions – Experimental Bias – Hawthorne Effect

19. External Validity Can results be generalized to other persons/settings/times? Threats – Interaction of treatment and selection – Interaction of treatment and setting – Interaction of treatment and history

20. Research Designs Common Sources of Error – Experimental Bias Post Hoc Error – Events that occur in sequence without cause & effect – Change related to coincidence; rival hypothesis Error of Misplaced Precision – Statistical significance not clinically important – Measuring blood pressure to the 0.001 mm Hg

21. Research Designs Common Sources of Error – Experimental Bias “Typical” Case Studies – Typically not typical – Typically IDEAL The Law of the Instrument – Always use the same instrument – Always calibrate the instrument

22. Experimental Bias Halo Effect – Irrelevant factors effect outcome favorably or unfavorably – Ex: Health care worker with a favorable/unfavorable characteristic influences outcome of study Rating Errors – Over/Under/Central tendency raters Hawthorne Effect – 1920’s Hawthorne Plant of Western Electric – Productivity & efficiency – Factory that owners changed conditions for productivity

23. Experimental Bias “Self-Fulfilling Prophecy” – Find what researchers expect to find “John Henry” Effect – Control group discovers their status and outperforms experimental group Placebo Effect – True effect of intervention versus ‘suggestibility’

24. EXAM Types of Data Application of the ICF Model-clinical purpose Evidence Based Practice/ APTA Core Values IV/DV, labels, levels, intervention Single subject, multifactorial, etc know details about them level of strength in design (case report to RCT) Hypotheses Determinates of Power Steps in Research process Correlation/ Association=spearman rho, – Inter vs. Intra – MDD, statistical difference – Reliability and Validity, Sources of Error No Research Articles Shapes of Distribution, skews, Levene’s Test, Mean’s difference, Specify and Sensitivity, Likelihood Ratios Multiple Choice and Fill in the Blank (40 questions) Paper Exam 8am