PTP 560 Research Methods Week 6 Thomas Ruediger, PT

Sampling How can we generalize our study to the world? Our sample responses are representative of population! Population – All members – All measurements possible Sample – Subgroup of members – Measurements actually taken Bias – Conscious – Unconscious (sub?)

Sampling Target Population Accessible population Sample Inclusion Criteria – Trait of the Target or accessible population – Qualifies someone as a subject – Restrictions here will limit ability to generalize Exclusion Criteria – Precludes someone being a subject – Excluded because they may interfere with interpreting findings Selection – Plan – Fig 8.2

Sampling Techniques Probability – Through random selection – Not the same as random assignment – Every member has equal chance of being selected – Considered (but not guaranteed to be) representative – Allows estimate of sampling error The difference between population average and sample average Non-probability – Non-random methods – Limits ability to generalize outcomes

Probability Sampling Simple random – Also known as sampling without replacement – Table of random numbers (Table 8.1) Systematic Stratified random – Subsets (strata) established – Random selection from the strata – May also be proportional – May be more representative than random

Probability Sampling Disproportional – Select random samples of appropriate size – Correct it with proportional weighting Cluster – Successive random sampling – Convenient and efficient …………….BUT, increased sampling error – Example Area probability sampling Random digit dialing

Non-Probability Sampling Convenience – Also known as accidental sample – Consecutive sampling is common method – Self selection is a major limitation Quota – Enroll subjects – Stop for certain strata when they are represented Purposive – Hand picked by criteria – Prone to bias Snowball – Chain-referral

Recruitment Feasibility issues can be daunting Advertisements Other healthcare providers/institutions Track and report – Screened for eligibility – Number actually eligible – Number enrolled POWER – “The ability to find significant differences when they exist” – Important to know a priori to get appropriate sample size

Validity in Experimental Design Experiment has three essential characteristics: 1. Manipulation of independent variables 2.Random assignment to groups 3.Control or comparison group Supports (Does NOT prove) cause-and effect relationship Extraneous variables – Must be controlled OR, – They can confound `

Handling incomplete (or lost) data On-Protocol (Completer) Analysis – Only those who complete the study – Tends to bias in favor of the treatment Intentions to treat (ITT) PREFERRED approach – What did we intend to do? – More conservative than On-Protocol – Considered to reflect clinical situations – Analysis? Non-completer equals failure Last observation carried forward

Validity in Experimental Design Blinding – Single Blind Subject blinded to treatment or placebo – Double Blind Subject and Tester blinded to treatment or placebo condition – Triple Blind Researcher, tester, and subject blinded Data analyzed by independent source

Controlling Inter-subject Differences Options – Homogenize on certain characteristic(s) – Manipulate attribute variables into “Blocks” – Consider matching – Use subjects as own control – Handle statistically with ANCOVA Table 9.1

Threats to validity Four Threats correspond to four major questions Is there a relationship between IV and DV? – Statistical Conclusion Validity Evidence of causal relationship? – Internal Validity Can results be generalized to a theoretical construct? – Construct Validity Can it be generalized to other persons/settings/times? – External Validity

Statistical Conclusion Validity Is there a relationship between IV and DV? Threats – Low Statistical Power – Violated Assumptions of Statistical Tests – Error Rate – Reliability – Variance – Failure to use ITT

Internal Validity Evidence of causal relationship? The extent to which the results of a study/experiment can be attributed to the treatment or intervention rather than to flaws in the research design Threats to internal validity – History – Maturation – Attrition – Testing – Instrumentation – Regression – Social Threats

Internal Validity Threats to Internal Validity – Testing Interactions Pre-tests or subsequent testing has an effect Second test scores tends to move toward the mean Standard Deviation decreases

Construct Validity Can results be generalized to a theoretical construct? Threats – Limits of Operational Definitions – Time Frame Within Operational Definitions – Multiple Treatment Interactions – Experimental Bias – Hawthorne Effect

External Validity Can results be generalized to other persons/settings/times? Threats – Interaction of treatment and selection – Interaction of treatment and setting – Interaction of treatment and history

Research Designs Common Sources of Error – Experimental Bias Post Hoc Error – Events that occur in sequence without cause & effect – Change related to coincidence; rival hypothesis Error of Misplaced Precision – Statistical significance not clinically important – Measuring blood pressure to the mm Hg

Research Designs Common Sources of Error – Experimental Bias “Typical” Case Studies – Typically not typical – Typically IDEAL The Law of the Instrument – Always use the same instrument – Always calibrate the instrument

Experimental Bias Halo Effect – Irrelevant factors effect outcome favorably or unfavorably – Ex: Health care worker with a favorable/unfavorable characteristic influences outcome of study Rating Errors – Over/Under/Central tendency raters Hawthorne Effect – 1920’s Hawthorne Plant of Western Electric – Productivity & efficiency

Experimental Bias “Self-Fulfilling Prophecy” – Find what researchers expect to find “John Henry” Effect – Control group discovers their status and outperforms experimental group Placebo Effect – True effect of intervention versus ‘suggestibility’