A Comparison of Self-reported Sexual Identity Using Direct and Indirect Questioning Alessandra Gaia Institute for Social and Economic Research (ISER) University of Essex agaia@essex.ac.uk Thank you very much for having me here today. I will talk about a comparison of self-reported sexual identity using direct and indirect questioning techniques.

Motivation 1/2 “sexual orientation”: multidimentional construct: attraction, behaviour and self-identification (Laumann et al., 1994) included in the list of protected attributes in the UK Equalities Act (2010), to tackle stigma, harrassment, and discrimination crucial to obtain high quality data… …but challenging Operational problems: Multidimentional construct Sexuality is a sensitive topic “sexual orientation” is included in the list of protected attributes in the UK Equalities Act (2010), to tackle stigma, harrassment, and discrimination. Thus, obtaining good quality data on this topic is crucial. However, this is also challenging. Indeed, there are several operational problems to asking survey questions on sexual orientation. Firstly, according to early work from Laumann and his colleagues, sexual orientation is a multinational construct involving attraction, behaviour and self-identification. Secondly, sexuality is among the most sensitive topics in surveys. A Comparison of Self-reported Sexual Identity Using Direct and Indirect Questioning | July 2017

Motivation 2/2 Estimates of sexual minorities in the UK seem inconsistent Wave 3 Year 2 of Understanding Society: the UK Household Longitudinal Study (UKHLS) 2012 Integrated Household Survey Bisexuality (HIS) < bisexuality (UKHLS) especially among young people Difference in data collection protocols: modes (IHS: F2F with showcards & Tel, vs. UKHLS: CASI), and wording. Thus, there are theoretical reasons why sexual orientation may be misreported in surveys, and/or may be subject to item non response. From an empirical point of view, there seem to be an inconsistency of estimates on sexual orientation in the UK. Wave 3 Year 2 of Understanding Society: the UK Household Longitudinal Study (UKHLS) overlaps with the 2012 round of the Integrated Household Survey; a comparison of figures suggests underreporting of bisexuality, especially among young people in the IHS. The two studies implement different data collection protocols: Firstly, they use different data collection modes (with the IHS adopting a and the UKHLS asking the survey question in the self completion section of the questionnaire) and they use slightly different wording. A Comparison of Self-reported Sexual Identity Using Direct and Indirect Questioning | July 2017

Research Idea Compare the UKHLS protocol with the IHS protocol, using the Item Count Technique as a benchmark the Item Count Technique pros: no one will know the respondent’s answers cons: no individual level data will the respondent understand? the “Longitudinal Item Count Technique (LICT)” My research idea is to compare these two data collection protocols using the Item Count Technique as a benchmark. I will describe the technique in detail in the next slides, for the moment let me just anticipate that this is a technique was over 25 years ago, it is based on the idea of eliciting more candid answers as it allows higher respondents’ privacy; (at least in its original formulation) it has the caveat that it doesn’t allow to derive individual level characteristics. I propose a modification of this technique, that allows the derivation of individual level estimates; and it is called the Longitudinal Item Count Technique (LICT). And I will explain the design in more detail in the next slides. A Comparison of Self-reported Sexual Identity Using Direct and Indirect Questioning | July 2017

Method: Item Count Technique (ICT)   Where: P is the estimated prevalence of the sensitive behaviour X is the number of items counted Longitudinal ICT (LICT) The item count technique   Assumption: time invariant behaviour between t1 and t2 A Comparison of Self-reported Sexual Identity Using Direct and Indirect Questioning | July 2017

Method: Item Count Technique (ICT) Group A Group B “I have at least once been sexually attracted to someone who … • is the same sex as me • has a disability • is fit and muscular • grew up with me in my local area • is ten or more years older than me How many statements are true for you? The item count technique A Comparison of Self-reported Sexual Identity Using Direct and Indirect Questioning | July 2017

Method: Longitudinal Item Count Technique (LICT) Wave 1 Wave 2 “I have at least once been sexually attracted to someone who … • is the same sex as me • has a disability • is fit and muscular • grew up with me in my local area • is ten or more years older than me How many statements are true for you? The item count technique A Comparison of Self-reported Sexual Identity Using Direct and Indirect Questioning | July 2017

Longitudinal Item Count Technique Combined Method Wave 1 Wave 2 Group 1 List + sensitive item List Longitudinal Item Count Technique Group 2 Item Count Technique The item count technique A Comparison of Self-reported Sexual Identity Using Direct and Indirect Questioning | July 2017

Research questions What is the estimated prevalence of the lesbian gay and bisexual population obtained with the HIS protocol and the UKHLS protocol? ...and how does it compares with the measure obtained with the Item count technique? ... And with the “Longitudinal Item Count Technique” perform? Thus my research questions are What is the estimated prevalence of the lesbian gay and bisexual population obtained with an indirect questioning method, such as the “Item Count” indirect questioning Technique (ICT)? ...and how does it compares with the measure obtained with CASI or CAPI with showcards protocols? ...how does the “Longitudinal Item Count Technique” perform? (not only for measuring sexual identity, but also homosexual attraction and homoerotic behaviour) A Comparison of Self-reported Sexual Identity Using Direct and Indirect Questioning | July 2017

Data Experiment from the innovation panel wave 8 and 9 Probability sample of the UK population IP is part of UKHLS study (but separate sample) for methodological innovation & substantive research Original sample: 1,500 households at wave 1 Refreshment sample at wave 4 and 7 (~ 500 hh) Experiment from the innovation panel wave 8 and 9 A Comparison of Self-reported Sexual Identity Using Direct and Indirect Questioning | July 2017

2x2 Design IP8 IP9 UKHLS Question List List + Sensitive item HIS Question List + Sensitive item Used for every item: sexual attraction, behaviour and identity Two list Item Count Technique (total of 6 question) A Comparison of Self-reported Sexual Identity Using Direct and Indirect Questioning | July 2017

Results, IP8: sexual minorities Note: N=1464 (UKHLS), 767 (IHS). A Comparison of Self-reported Sexual Identity Using Direct and Indirect Questioning | July 2017

Results, IP9: sexual minorities Note: N=1464 (UKHLS), 544 (IHS) A Comparison of Self-reported Sexual Identity Using Direct and Indirect Questioning | July 2017

Results Low item non response, refusal, and don’t know... ...but, evidence of both “ceiling effects” and “floor effects” estimates of behaviour and attraction are considered unreliable due to “ceiling” and “floor” effects A Comparison of Self-reported Sexual Identity Using Direct and Indirect Questioning | July 2017

Results No significant difference among the two protocols, in IP8, nor in IP9 Even analysing the results by subgroups, e.g. age Low item non response, refusal, and don’t know... ...but, both “ceiling effects” and “floor effects” A Comparison of Self-reported Sexual Identity Using Direct and Indirect Questioning | July 2017

Conclusion While the item count technique was generally accepted by respondents, the derived estimates are not always reliable and there is inconsistency across the ICT and LICT No difference in the two protocols: UKHLS and IHS A Comparison of Self-reported Sexual Identity Using Direct and Indirect Questioning | July 2017

Thank you! A Comparison of Self-reported Sexual Identity Using Direct and Indirect Questioning | July 2017

Appendix A Comparison of Self-reported Sexual Identity Using Direct and Indirect Questioning | July 2017

Item count tecniques The next set of questions will ask you to count the number of statements that are true for you. Please only count the number of statements. We are not interested in knowing which statements are relevant for you. I have at least once been sexually attracted to someone who … • is the same sex as me • has a disability • is fit and muscular • grew up with me in my local area • is ten or more years older than me How many statements are true for you? None of the statements are true One of the statements are true Two of the statements are true Three of the statements are true Four of the statements are true Five statements are true A Comparison of Self-reported Sexual Identity Using Direct and Indirect Questioning | July 2017

Longitudinal Item Count Technique Results   Item Count Technique Longitudinal Item Count Technique Same sex Attraction 16.6% 30.8% Same sex Behaviour 9.5%  27.7% Gay/ lesbian/ bisexual 3.6%  N.A. A Comparison of Self-reported Sexual Identity Using Direct and Indirect Questioning | July 2017

Longitudinal Item Count Technique (ICT) – 2 list version Group 1 Group 2 “I have at least once been sexually attracted to someone who … • is the same sex as me • has a disability • is fit and muscular • grew up with me in my local area • is ten or more years older than me How many statements are true for you? • wears the latest trends and fashions • has a tattoo or body piercing • is of a different ethnicity to me • is from a different class background to me How many statements are true for you?” A Comparison of Self-reported Sexual Identity Using Direct and Indirect Questioning | July 2017

IHS Question via face to face interview (with a show card) “Which of the options on this card best describes how you think of yourself? Please just read out the number next to the description.” Note: The categories on the show card should appear as follows: 27 Heterosexual/Straight 21 Gay / Lesbian 24 Bisexual 29 Other A Comparison of Self-reported Sexual Identity Using Direct and Indirect Questioning | July 2017

Item count tecniques Lists made sense together Mix of (expected to be) high prevalence and low prevalence items A Comparison of Self-reported Sexual Identity Using Direct and Indirect Questioning | July 2017

IHS Questions via telephone interview HIS Questions via telephone interview I will now read out a list of terms people sometimes use to describe how they think of themselves: “Heterosexual or Straight”, “Gay or Lesbian”, “Bisexual”, or “Other”. As I read the list again please say 'yes' when you hear the option that best describes how you think of yourself. “Heterosexual or Straight”, “Gay or Lesbian”, “Bisexual”, or “Other”. A Comparison of Self-reported Sexual Identity Using Direct and Indirect Questioning | July 2017

UKHLS Question via self-completion (CASI or Web) “Which of the following options best describes how you think of yourself? Heterosexual/Straight Gay / Lesbian Bisexual Other Prefer not to say” A Comparison of Self-reported Sexual Identity Using Direct and Indirect Questioning | July 2017

Design List A & B Attraction List C & D Behaviour List E & F Identity A Comparison of Self-reported Sexual Identity Using Direct and Indirect Questioning | July 2017

Method: Item Count Technique (ICT) Sample I have at least once been sexually attracted to someone who … • is the same sex as me • has a disability • is fit and muscular • grew up with me in my local area • is ten or more years older than me How many statements are true for you? The item count technique A Comparison of Self-reported Sexual Identity Using Direct and Indirect Questioning | July 2017

Method: Item Count Technique (ICT) I have at least once been sexually attracted to someone who … • has a disability • is fit and muscular • grew up with me in my local area • is ten or more years older than me How many statements are true for you? I have at least once been sexually attracted to someone who … • is the same sex as me • has a disability • is fit and muscular • grew up with me in my local area • is ten or more years older than me How many statements are true for you? The item count technique A Comparison of Self-reported Sexual Identity Using Direct and Indirect Questioning | July 2017