1. Reliability & validity issues in European web survey in Czech Republic
Lisbon Addictions 2017

2. Conflict of interest statement.
Work funded under the EMCDDA project CT.15.EPI (Assessment of the Methodology of internet web-surveys to collect amounts of drugs used: reliability and validity).

3. Aims
- to trial online data collection and various solicitation channels
- to assess test/re-test reliability of the individual items and of the derived consumption estimates
- to improve questionnaire through the qualitative cognitive assessment

4. Conclusions: Methodological challenges
It is crucial to know the target population.
Use simple tools to measure complex and rather irregular phenomena.

5. Content and timeline Literature review (Autumn 2015)
Translation, adaptation and testing of the common questionnaire (Winter 15 / Spring 16)
Pilot data collection (Feb-April 16)
*Follow-up data collection (re-test, April-May 16)
Two-weeks daily recalls (diary method, May 16)
*Cognitive interviews (May-July 16)

6. Overview of the data Pilot (N=610)
- 65% male, Mage=28 (80% aged 18 to 34 years)
- prevailing stable working and living conditions, 38% tertiary education
- LYP: 88% cannabis, 29% cocaine, 42% XTC, 25% amphetamines
Follow-up (N=153)
- 61% male, Mage=28
- LYP: 93% cannabis, 33% cocaine, 55% XTC, 30% amphetamines
Cognitive interviews (N=19)
- recreational drug users, problem methamphetamine users, one dealer of methamphetamine
Ads targeted – recreational drug users / party goers, cannabis users / cannabis legalization supporters, psychonauts, and problem users of met/amphetamines.
(79% came through Facebook)

7. Test/re-test reliability (individual items)
Moderate to high reliability of most items (w/sufficient N)
Prevalence items
single-question vs. arrays
Traditional vs. unusual/unfamiliar drugs
Frequency questions
Ordinal scale vs. days of use (LMP/LYP)
High reliability of clue-induced amounts (use, purchase)
Only calculated for cannabis resin
Low reliability for amounts used in “dry pipe”
Low reliability for XTC/MDMA questions (powder/tabs)
Low sample sizes for cocaine questions, no data on amphetamine tablets
Follow-up data collection
2 weeks after pilot, recruited via address request
Same questionnaire (copy/paste in the Lime Survey)
Reduced background and redundant questions
N= 153, RR=66.2% (out of 231 s collected)
kappa statistics (𝜅) for categorical and dichotomous variables
interclass correlation coefficient (ICC) for ordinal and continuous measures
modest (0.30 to 0.49), moderate (0.50 to 0.69), or high (0.70 to 1.00) reliability

8. Test/re-test reliability (typologies of users)
Cannabis users:
Other substances:
Infrequent users or chippers = less than 11 days in the past year (≈ “less than once a month”).
Infrequent users = less than 11 days in the past year (≈ “less than once a month”)
Occasional users = days (≈ “less than once a week but at least once a month”).
Occasional users = days (≈ “less than once a week but at least once a month”)
Regular users = days (≈ “once a week and ‘more than once a week, but not daily or almost daily”).
Frequent users = 51 to more than 350days (≈ “once a week or more”).
Intensive users = more than 250 days (≈ daily or almost daily”).

9. Test/re-test reliability (yearly consumption)
Last 30 days not representative for last year.
Number of assumptions and data manipulations for weed/skunk and resin.
Consumption by type of user calculated.
Last 30 days not representative for last year
Number of assumptions and data manipulations for weed/skunk and resin
Consumption by type of user calculated

10. Validity (cognitive interviews)
Overall comprehensible, easy to read and understand
Classical cognitive issues:
Less effort in online data collection (“did not know precision is required”)
Telescoping (e.g. accounting for distant heavy periods)
Identification w/pattern (recreational vs. heavy vs. cannabis users)
Exceptional experiences easier to recall (events/holidays, cocaine and resin)
How many doses (joints/pipes) would participant prepare from one gram of cannabis or cannabis resin?
low reliability for cannabis, moderate for resin (n.s.)
No “usual” or “typical” dose/purchase/money spent for many users (“It depends… on weekday, available drug, quality, budget...”).
Occasional extreme purchase and/or use.
Daily cannabis users: grams/day vs. joints/day
Irregular patterns in frequency questions (missing “once or twice”)
Resin in frequent cannabis users
Heavy methamphetamine users/binges
Holiday season binges
Types of cannabis – buy vs. consume
Amphetamine vs. methamphetamine (double-barrelled section)
Local production of meth (precursors, exchange for goods)
Specific terminology: ”one gram” = 0.7 gram = 10 syringe units
Unclear amount of purchase:
One dose/one line
”for one thousand”

11. Summary of the main points
Moderate to high test/re-test reliability
Unambiguity and effort increase reliability
Frequency questions perform better when number of days is asked (compared to ordinal scale item)
Individual consumption easier from direct questions (=less error)
Recommendations from cognitive testing - never assume regular patterns and same understanding of items.

12. Thank you for your attention.
Katerina Skarupova