DENT 514: Research Methods Masahiro Heima, DDS, PhD

Lecture 6: Methodology 2: Cross sectional, Longitudinal Study, Split Mouth Design, Crossover Designs, and Questionnaires (reliability and validity),

Cross-Sectional study vs. Longitudinal study Cross-sectional study: an observational study which involves a observation of variables from subjects at one specific point in time. We can see differences in the variables between groups. Longitudinal study: an observational study which involves repeated observations of the same variables from the same subjects over a period of time. We can see differences in the variables between groups as well as time.

Split Mouth Design One side of the mouth is the “control” side and the other is the “study” side (within-subject designs) This design removes all differences between subjects. Carry-over Efects: one side affects another side mouth (ex. Fluoride applied on one side of the mouth can affect the other side of the mouth.) Counterbalancing: necessary in order to control the carryover/order effect, through randomization.

Dental Caries Examination Isaacs (Caries Research, 1999) Subjects: 150 children, 9‐12‐years‐old (high risk population) Methods: Split mouth Half of the mouth was examined with loupes and the other half with explorer. DFS was counted at baseline and in 8 months. Results: Loupe side = 2.1 fold increase Explorer side = 4.5 fold increase

Carryover Effects: needs “washout” Crossover design Each participant received a control condition and an intervention. Carry-over effects (“Order" effects, “Learning" effect) “Washout:” Any carryover effect is washed out as a consequence of more than sufficient time between visit 1 and 2 Carryover Effects: needs “washout” Visit 1 Visit 2 Group 1 Control Intervention Group 2 Intervention Control

Surveys(Questionnaires) “Easy” Relatively low cost Methods: Face to face, Telephone, Letter, Email (Text), Web based and mixed Sampling method (possible bias) Sample size (Large) Response rate (at least 60-70 %: “adequate”) Reliability and Validity

Methodology Development of questionnaire PROMIS® Instrument Development and Validation Scientific Standards Version 2.0 (revised May 2013) http://www.nihpromis.org/Documents/PROMISStandards_Vers2.0_Final.pdf Design of questionnaire The Total Design Method/Tailored Design Method (TDM) (by Dillman): to improve quality of response and to increase response rate

Dental visit In the past 12 months, how many times did you see a dentist? Last year, how many times did you see a dentist?

Questionnaires (reliability and validity) Reliability (consistency) and validity (accuracy) Reliability is the degree to which an assessment tool produces stable and consistent results. Validity refers to how well a test measures what it is purported to measure.  High reliability Low validity Low reliability High validity Low reliability Low validity High reliability High validity

Reliability For tools (questionnaire, assessment, evaluation, etc.) Stability Reliability (test-retest) Internal Consistency Reliability Parallel forms reliability For researchers (the reliability of researchers) Interobserver Reliability (Interrater Reliability) Intraobserver Reliability

Reliability Stability Reliability (test-retest): To measure stability of instruments over time-Same test different time. It is used when the interest or phenomenon is stable/unchanging. Works for the Trait Anxiety Inventory ,which measures how easily a person becomes anxious. However, does not work for the State Anxiety Inventory, which measures the level of anxiety at various points in time. You want to know how much the two variables (first time and second time) are similar. What kind of statistics do you want to use? Answer: a correlation analysis Time 2 Time 1

Reliability Internal Consistency Reliability (Inter-item Reliability): tests questions designed to measure the same concept. (Cronbach’s alpha) (Split-half Reliability test is also used) Example How long do you study? (less than 1 min,…) Do you study hard? (yes or no) Do you discuss your questions with your professor? (yes or no) Do you drive a car? (yes or no) This is different.

Reliability Parallel forms reliability: tests two measurements which have identical (similar) concepts-different forms at the same time e.g.: Anxiety questionnaire and Fear questionnaire Comparing the new developing form with the “standard” form You want to know how much the two variables are similar. What kind of statistics do you want to use? Answer: correlation analysis Variable 2 Variable 1

Reliability Interobserver Reliability (Interrater Reliability): tests observers (raters) of a research team measuring the same thing. Addresses the consistency of the implementation of a rating system. Multiple observers test one subject Kappa statistics (2x2 table, two examiners and “Yes” or “No”) Correlation coefficients (two examiners and using a continuous scale or ordinal scale)

Reliability Intraobserver Reliability: test the observer rating in the same manner every time. Same assessment twice. You want to know how much the two variables are similar. What kind of statistics do you want to use? Answer: correlation analysis Time 2 Time 1

Validity Construct validity Criterion validity Content validity Convergent validity Discriminate validity Criterion validity Concurrent validity Predictive validity Content validity Representation validity Face validity

Validity Construct Validity: Whether the measurement tools (ex. questionnaire) measure the constructs being investigated  Factor analysis Convergent validity: Two (conceptually) similar constructs correspond with one another Discriminate validity (divergent validity): Two (conceptually) dissimilar constructs do not correspond with one another Questionnaire 1 Questionnaire 2 Factor 1 Factor 1 Factor 2 Factor 2 Factor 3 Factor 4

Validity Criterion Validity: a measure of how well a set of variables predicts an outcome A researcher is developing a new “behavior questionnaire” which tries to predicts a child’s behavior on the dental chair. Good Poor Outcome of the Q Outcome of the Q Behavior rate Behavior rate

Validity Content Validity: the extent to which the content of the test reflects the specific intended (theoretical) domain of content . E.g.: A semester or quarter exam that only includes content covered during the last six weeks is not a valid measure of the course's overall objectives -- it has very low content validity.

Questions?