Reliability & validity issues in European web survey in Czech Republic Lisbon Addictions 2017

Conflict of interest statement. Work funded under the EMCDDA project CT.15.EPI.0112.1.0 (Assessment of the Methodology of internet web-surveys to collect amounts of drugs used: reliability and validity).

Aims - to trial online data collection and various solicitation channels - to assess test/re-test reliability of the individual items and of the derived consumption estimates - to improve questionnaire through the qualitative cognitive assessment

Conclusions: Methodological challenges It is crucial to know the target population. Use simple tools to measure complex and rather irregular phenomena.

Content and timeline Literature review (Autumn 2015) Translation, adaptation and testing of the common questionnaire (Winter 15 / Spring 16) Pilot data collection (Feb-April 16) *Follow-up data collection (re-test, April-May 16) Two-weeks daily recalls (diary method, May 16) *Cognitive interviews (May-July 16)

Overview of the data Pilot (N=610) - 65% male, Mage=28 (80% aged 18 to 34 years) - prevailing stable working and living conditions, 38% tertiary education - LYP: 88% cannabis, 29% cocaine, 42% XTC, 25% amphetamines Follow-up (N=153) - 61% male, Mage=28 - LYP: 93% cannabis, 33% cocaine, 55% XTC, 30% amphetamines Cognitive interviews (N=19) - recreational drug users, problem methamphetamine users, one dealer of methamphetamine Ads targeted – recreational drug users / party goers, cannabis users / cannabis legalization supporters, psychonauts, and problem users of met/amphetamines. (79% came through Facebook)

Test/re-test reliability (individual items) Moderate to high reliability of most items (w/sufficient N) Prevalence items single-question vs. arrays Traditional vs. unusual/unfamiliar drugs Frequency questions Ordinal scale vs. days of use (LMP/LYP) High reliability of clue-induced amounts (use, purchase) Only calculated for cannabis resin Low reliability for amounts used in “dry pipe” Low reliability for XTC/MDMA questions (powder/tabs) Low sample sizes for cocaine questions, no data on amphetamine tablets Follow-up data collection 2 weeks after pilot, recruited via email address request Same questionnaire (copy/paste in the Lime Survey) Reduced background and redundant questions   N= 153, RR=66.2% (out of 231 emails collected) kappa statistics (𝜅) for categorical and dichotomous variables interclass correlation coefficient (ICC) for ordinal and continuous measures modest (0.30 to 0.49), moderate (0.50 to 0.69), or high (0.70 to 1.00) reliability

Test/re-test reliability (typologies of users) Cannabis users: Other substances: Infrequent users or chippers = less than 11 days in the past year (≈ “less than once a month”). Infrequent users = less than 11 days in the past year (≈ “less than once a month”) Occasional users = 11-50 days (≈ “less than once a week but at least once a month”). Occasional users = 11-50 days (≈ “less than once a week but at least once a month”) Regular users = 51-250 days (≈ “once a week and ‘more than once a week, but not daily or almost daily”). Frequent users = 51 to more than 350days (≈ “once a week or more”). Intensive users = more than 250 days (≈ daily or almost daily”).

Test/re-test reliability (yearly consumption) Last 30 days not representative for last year. Number of assumptions and data manipulations for weed/skunk and resin. Consumption by type of user calculated. Last 30 days not representative for last year Number of assumptions and data manipulations for weed/skunk and resin Consumption by type of user calculated

Validity (cognitive interviews) Overall comprehensible, easy to read and understand Classical cognitive issues: Less effort in online data collection (“did not know precision is required”) Telescoping (e.g. accounting for distant heavy periods) Identification w/pattern (recreational vs. heavy vs. cannabis users) Exceptional experiences easier to recall (events/holidays, cocaine and resin) How many doses (joints/pipes) would participant prepare from one gram of cannabis or cannabis resin? low reliability for cannabis, moderate for resin (n.s.) No “usual” or “typical” dose/purchase/money spent for many users (“It depends… on weekday, available drug, quality, budget...”). Occasional extreme purchase and/or use. Daily cannabis users: grams/day vs. joints/day Irregular patterns in frequency questions (missing “once or twice”) Resin in frequent cannabis users Heavy methamphetamine users/binges Holiday season binges   Types of cannabis – buy vs. consume Amphetamine vs. methamphetamine (double-barrelled section) Local production of meth (precursors, exchange for goods) Specific terminology: ”one gram” = 0.7 gram = 10 syringe units Unclear amount of purchase: One dose/one line ”for one thousand”

Summary of the main points Moderate to high test/re-test reliability Unambiguity and effort increase reliability Frequency questions perform better when number of days is asked (compared to ordinal scale item) Individual consumption easier from direct questions (=less error) Recommendations from cognitive testing - never assume regular patterns and same understanding of items.

Thank you for your attention. Katerina Skarupova skarupovakat@gmail.com