Research Showcase: Combining multiple methods Dr Katherine Woolf Associate Professor in Medical Education NIHR Career Development Fellow Dr Eliot Rees PhD Candidate in Medical Education We were going to talk mostly about mixing primary and secondary data, including qualitative and quantitative primary data, and survey and administrative secondary data. We were going to: Find out what participants know about primary and secondary data (do they know what it is, can they give examples) Get them to consider the advantages and disadvantages of both types of data in general Give participants the research questions for our study (does applicant background influence choice of medical school, and  does choice affect probability of getting a place to study medicine), ask them to design a study in which they use both primary and secondary data, and then get them to think about the advantages and challenges, and how we might overcome the challenges. We will also present our research design and get them to critique it.   Our study uses primary data (interviews, followed by a questionnaire informed by interview findings), linked to secondary administrative data (UCAS, UKCAT, and other UKMED data). We may also link to secondary birth cohort study data (Millennium Cohort Study, Next Steps). So there is plenty to talk about. In fact, it’s pretty hard to come up with something that fits in to 45 minutes!

Since 2004, I’ve spent most of my academic career trying to understand why we see ethnic differences in attainment in medics, particularly UK medical students and graduates. The reason I do this is because I think it is vital to understand what is happening so that we can try to make medical education fairer and improve patient care.

UK Medical Applicant Cohort Study: UKMACS How does applicant background influence medical school choice and success? Study aim: improve patient care by helping medical applicants make informed choices. I’m now doing research on selection into medical school from the applicant perspective, trying to understand how someone’s background might influence their applications and their chances of getting a place. That’s the research I’m going to be talking about today.

UK Medical Applicant Cohort Study: UKMACS Three year study using a combination of primary and secondary data. Funded by the National Institute for Health Research (NIHR)’s Fellowship programme Programme. The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health. I’ll give you an overview of our methods later, but essentially it’s a three year study which uses a combination of primary and secondary data.

Today’s session Discussion about the advantages and limitations of primary and secondary data, and of combining them. Group exercise: designing a study using primary and secondary data. Overview of UKMACS methods.

In your main role, are you…? New researcher Research producer Policy maker or research user Hands up if you are: 1. New researcher (never designed and carried out own study, and not a policy maker) 2. Research producer (mainly, and have designed and carried out own study) 3. Policy maker or research user (mainly) Get people into groups so they’re a bit mixed up.

Primary and secondary data QQ Does anyone know what primary or secondary data are?

Primary and secondary data Primary data: “collected for the specific research problem at hand, using procedures that fit the research question best” Secondary data: “originally collected for a different purpose and reused for another research purpose”. Hox & Boeije (2005) Encyclopeadia of Social Measurement Primary data can of course become secondary, if someone collects survey data and then makes it generally available, it becomes secondary data.

In pairs/small groups List two or more examples of secondary data you have come across or are aware of. 2 mins 3 mins feedback – TOTAL 5 mins Birth Cohort study data UKMED data (UCAS, BMAT, UKCAT, specialty applicants, progression data, exam data) NHS digital data (hospital records, information about hospital trusts) Educational data – PISA HRMC data

Many sources of secondary data Many sources of secondary data. Mostly administrative, but also survey data including some of the best birth cohort studies in the world. www.closer.ac.uk http://www.oecd.org/pisa/data/ programme for international student assessment https://digital.nhs.uk/ Ukmed LEO – longitudinal educational outcomes – linked to HRMC data

Secondary data Often administrative data. Increasingly quantitative research data e.g. survey/cohort study data and experimental (trial) data. More rarely qualitative data.

Pros Cons Primary Secondary In pairs/small groups Write a list of at least 2 pros and 2 cons of using primary and secondary data in research 3 mins Pros Cons Primary Secondary 5 mins feedback. Total 8 mins.

Pros Cons Primary data Secondary Designed to answer your research question. Often easy to access (you collect them!). Easy to understand because they’re yours. Often don’t require linkage. Can be expensive to collect. Often small or single-institution/location samples. Typically suffer from response/self-report bias. Limited scope (e.g. variables limited due to expensive data collection). Secondary Can be free to access. Easier to get large N’s so high statistical power. Can provide lots of variables & be longitudinal (esp if linked). Can encompass whole populations. Administrative data can bypass self-report bias and problems due to low response. Not designed to answer your research question (e.g. missing appropriate variables, collected from the wrong participants). Can be hard to access (GDPR, commercial interests, slow organisations). Can be hard to understand. Can require linkage, which needs expertise & takes time. GDPR – asking for consent for data linkage. These are generalisations rather than hard and fast rules.

Combining primary and secondary data Can get benefit from the advantages of both primary (designed to answer your question) and secondary (complete data lacking in response bias). But combining data can require a lot of time and expertise in multiple methods and data linkage.

In groups Design a study that uses both primary and secondary data to answer this research question: How do applicants from different social backgrounds choose which medical schools to apply to? Be prepared to give a 2 min overview 8 mins Three groups, each give 2 minute overview (6 mins). General feedback 5 mins. TOTAL 12 mins. If no time, just ask for one group to talk through and give feedback. (8 mins)

UK Medical Applicant Cohort Study

Workstream 1: The impact of choice on selection How does an applicant’s background influence which medical schools they choose to apply to? Interviews applicants & entrants from diverse backgrounds in 2018 (Qualitative study) survey applicants in 2019 (Questionnaire study, linked to administrative data). What is the impact of choices on outcomes? Statistical modelling of linked administrative data (UCAS, UKCAT, BMAT, HESA) on medical school choice, offers and acceptances by applicant characteristics (Applications and Outcomes study)

Output: Tool to improve choice and selection Better information provision to attract the best applicants from diverse backgrounds. Developed with Medical Schools Council Selection Alliance (created in 2016 to improve selection and widen participation to medicine). Medical schools providing better information to potential applicants about choices.

Workstream 2: The foundations for future research Link questionnaire data into UKMED to follow up medical entrants administratively.  By 2025, start to assess the impact of applicant choices and medical school selection on the provision of patient care by UK-trained doctors. Medical schools providing better information to potential applicants about choices.

Questions/comments about using primary and secondary data, or about UKMACS before I summarise?

Take home messages Mixed methods doesn’t just mean qualitative and quantitative. Secondary data can be very powerful, and is increasingly available in medical education (especially UKMED). Some secondary data hard to access in practice, and can’t always be used to answer your question. Combining primary and secondary data can be advantageous but often requires time and expertise.