Smart Qualitative Data: Methods and Community Tools for Data Mark-Up (SQUAD) Louise Corti UK Data Archive, University of Essex ASC Conference 29 September 2006

Our unit - ESDS Qualidata specialist service of the ESDS led by the UK Data Archive (UKDA) specialist service of the ESDS led by the UK Data Archive (UKDA) systematically archiving and enabling sharing of qualitative data since 1995 systematically archiving and enabling sharing of qualitative data since 1995 focus is on acquiring digital data collections from purely qualitative and mixed methods contemporary research and from UK-based 'classic studies' focus is on acquiring digital data collections from purely qualitative and mixed methods contemporary research and from UK-based 'classic studies' facilitates the preservation of important large paper collections, and where appropriate, digitises samples of these collections. facilitates the preservation of important large paper collections, and where appropriate, digitises samples of these collections. works closely with data creators (e.g academics) to ensure that high quality and well-documented qualitative data are produced works closely with data creators (e.g academics) to ensure that high quality and well-documented qualitative data are produced offers user support and training to encourage professional researchers and research students alike to make full use of the rich sources of archived qualitative data offers user support and training to encourage professional researchers and research students alike to make full use of the rich sources of archived qualitative data

Access to data ESDS offers a resource discovery hub of some 4000 data collections ESDS offers a resource discovery hub of some 4000 data collections some 160 qualitative research-based datasets some 160 qualitative research-based datasets developed an online data browsing service for texts (ESDS Qualidata Online) developed an online data browsing service for texts (ESDS Qualidata Online) programme to extend and share common methods, standards and tools relating to this system programme to extend and share common methods, standards and tools relating to this system investigating new publishing forms: re-presentation of research outputs combined with data investigating new publishing forms: re-presentation of research outputs combined with data investigating natural language processing, text mining and e- science applications to enable richer access to digital data banks investigating natural language processing, text mining and e- science applications to enable richer access to digital data banks

4 Applications of formats and standards standard for data producers to store and publish data in multiple formats standard for data producers to store and publish data in multiple formats e.g UK Data Archive and ESDS Qualidata Online e.g UK Data Archive and ESDS Qualidata Online data exchange and data sharing across dispersed repositories and software packages (eg CAQDAS) data exchange and data sharing across dispersed repositories and software packages (eg CAQDAS) more precise searching/browsing of archived qualitative data beyond the catalogue record more precise searching/browsing of archived qualitative data beyond the catalogue record shared toolsets for preparing qualitative data for sharing and archiving shared toolsets for preparing qualitative data for sharing and archiving

5 Our own needs ESDS Qualidata online system ESDS Qualidata online system limited functionality - currently keyword search, KWIC retrieval, and browse of interview texts and documents limited functionality - currently keyword search, KWIC retrieval, and browse of interview texts and documents wish to extend functionality wish to extend functionality display of marked-up features (e.g.. named entities) display of marked-up features (e.g.. named entities) linking between sources (e.g.. text, annotations, analysis, audio etc) linking between sources (e.g.. text, annotations, analysis, audio etc) for 5 years we have been developing a generic descriptive standard and format for data that is customised to social science research and which meets generic needs of varied data types for 5 years we have been developing a generic descriptive standard and format for data that is customised to social science research and which meets generic needs of varied data types some important progress through TEI and Australian collaboration some important progress through TEI and Australian collaboration

UK Qualitative data initiatives Typically not technically oriented, but offer important opportunities for collaboration: QUADS - Archiving and Sharing Demonstrator scheme QUADS - Archiving and Sharing Demonstrator scheme National Centre for Research Methods National Centre for Research Methods National Qualitative Longitudinal Study National Qualitative Longitudinal Study

More technically oriented support National Centre for e-Social Science National Centre for e-Social Science JISC – e.g. Digital Repositories Programme JISC – e.g. Digital Repositories Programme International data archives community (IASSIST) International data archives community (IASSIST) NSF Cyber infrastructure NSF Cyber infrastructure CoData – data sharing organisations CoData – data sharing organisations

8 How useful is textual data? dob: 1921 Place: Oldham finalocc: Oldham [Welham] U id='1' who='interviewer' Right, it starts with your grandparents. So give me the names and dates of birth of both. Do you remember those sets of grandparents? U id='2' who='subject' Yes. U id='3' who='interviewer' Well, we'll start with your mum's parents? Where did they live? U id='4' who='subject' They lived in Widness, Lancashire. U id='5' who='interviewer' How do you remember them? U id='6' who='subject' When we Mum used to take me to see them and me Grandma came to live with us in the end, didn't she? U id='7' who='Welham' Welham: Yes, when Granddad died - '48. U id='8' who='interviewer' So he died when he was 48? U id='9' who='Welham' Welham: No, he was 52. He died in U id='10' who='interviewer' But I remember it. How old would I be then? U id='11' who='Welham' Welham: Oh, you would have been little then. U id='12' who='subject' I remember him, he used to have whiskers. He used to put me on his knee and give me a kiss....

9 What are we interested in finding in data? short term: short term: how can we exploit the contents of our data? how can we exploit the contents of our data? how can data be shared? how can data be shared? what is currently useful to mark-up? what is currently useful to mark-up? long term long term what might be useful in the future? what might be useful in the future? who might want to use your data? who might want to use your data? how might the data be linked to other data sets? how might the data be linked to other data sets?

10 SQUAD Project: Smart Qualitative Data Primary aim: to explore methodological and technical solutions for exposing digital qualitative data to make them fully shareable and exploitable to explore methodological and technical solutions for exposing digital qualitative data to make them fully shareable and exploitable collaboration between collaboration between UK Data Archive, University of Essex (lead partner) UK Data Archive, University of Essex (lead partner) Language Technology Group, Human Communication Research Centre, School of Informatics, University of Edinburgh Language Technology Group, Human Communication Research Centre, School of Informatics, University of Edinburgh 18 months duration, 1 March 2005 – 31 August months duration, 1 March 2005 – 31 August 2006

11 SQUAD: main objectives developing and testing universal standards and technologies developing and testing universal standards and technologies long-term digital archiving long-term digital archiving publishing publishing data exchange data exchange defining context for research data (e.g. interview settings and dynamics and micro/macro factors defining context for research data (e.g. interview settings and dynamics and micro/macro factors user-friendly tools for semi-automating processes already used to prepare qualitative data and materials (Qualitative Data Mark-up Tools (QDMT) user-friendly tools for semi-automating processes already used to prepare qualitative data and materials (Qualitative Data Mark-up Tools (QDMT) formatted text documents ready for output formatted text documents ready for output mark-up of structural features of textual data mark-up of structural features of textual data annotation and anonymisation tool annotation and anonymisation tool automated coding/indexing linked to a domain ontology automated coding/indexing linked to a domain ontology providing demonstrators and guidance providing demonstrators and guidance

12 What features can be marked-up? spoken interview texts provide the clearest ― and most common ― example of kinds of typical encoding features: 3 basic groups of structural features 3 basic groups of structural features utterance, specific turn taker, defining idiosyncrasies in transcription utterance, specific turn taker, defining idiosyncrasies in transcription links to analytic annotation and other data types (e.g.. thematic codes, concepts, audio or video links, researcher annotations) links to analytic annotation and other data types (e.g.. thematic codes, concepts, audio or video links, researcher annotations) identifying information such as real names, company names, place names, occupations, temporal information identifying information such as real names, company names, place names, occupations, temporal information

13 Identifying elements Identify atomic elements of information in text Identify atomic elements of information in text Person names Person names Company/Organisation names Company/Organisation names Locations Locations Dates Dates Times Times Percentages Percentages Occupations Occupations Monetary amounts Monetary amounts Example: Example: Italy's business world was rocked by the announcement last Thursday that Mr. Verdi would leave his job as vice-president of Music Masters of Milan, Inc to become operations director of Arthur Anderson. Italy's business world was rocked by the announcement last Thursday that Mr. Verdi would leave his job as vice-president of Music Masters of Milan, Inc to become operations director of Arthur Anderson.

14 How do we annotate our data? human effort? human effort? how long does one document take to mark up by hand? how long does one document take to mark up by hand? how much data do you want/need? how much data do you want/need? how many annotators do you have? how many annotators do you have? human error – like traditional coding error human error – like traditional coding error accuracy accuracy expertise in subject area expertise in subject area boredom boredom subjective opinions subjective opinions what if we decide to add more categories for mark-up at a later date? what if we decide to add more categories for mark-up at a later date? can this be automated at all? can this be automated at all?

15 Automating content extraction using rules rules can be written rules can be written lists of common names, useful to a point lists of common names, useful to a point lists of pronouns (I, he, she, me, my, they, them, etc) lists of pronouns (I, he, she, me, my, they, them, etc) “me mum”; “them cats”, but which entities do pronouns refer to? “me mum”; “them cats”, but which entities do pronouns refer to? rules regarding typical surface cues: rules regarding typical surface cues: CapitalisedWord CapitalisedWord probably a name of some sort e.g. “John found it interesting…” probably a name of some sort e.g. “John found it interesting…” first word of sentences is useless though first word of sentences is useless though title CapitalisedWord - probably a person name, e.g. “Mr. Smith” or “Mr. Average”? title CapitalisedWord - probably a person name, e.g. “Mr. Smith” or “Mr. Average”? Works ok but requires several months for a person to write these rules Works ok but requires several months for a person to write these rules each new domain/entity type requires more time each new domain/entity type requires more time requires experienced experts (linguists, biologists, etc.) requires experienced experts (linguists, biologists, etc.)

16 What about more intelligent content extraction mechanisms? machine learning: machine learning: manually annotate texts with entities manually annotate texts with entities 100,000 words can be done in 1-3 days depending on experience 100,000 words can be done in 1-3 days depending on experience the more annotated data you have, the higher the accuracy the more annotated data you have, the higher the accuracy if the system hasn’t seen it or hasn’t seen anything that looks like it, then it can’t tell what it is if the system hasn’t seen it or hasn’t seen anything that looks like it, then it can’t tell what it is So - garbage in, garbage out So - garbage in, garbage out Latest approach uses a mixture of rules and machine learning Latest approach uses a mixture of rules and machine learning Recent focus on relation and event extraction Recent focus on relation and event extraction Mike Johnson is now head of the department of computing. Today he announced new funding opportunities. Mike Johnson is now head of the department of computing. Today he announced new funding opportunities. person(Mike-Johnson) person(Mike-Johnson) head-of(the-department-of-computing, Mike-Johnson) head-of(the-department-of-computing, Mike-Johnson) announced(Mike-Johnson, new funding opportunities, today) announced(Mike-Johnson, new funding opportunities, today)

17

18 UK Data Archive - NLP collaboration ESDS Qualidata making use of options for semi-automated mark-up of some components of its data collections using natural language processing and information extraction ESDS Qualidata making use of options for semi-automated mark-up of some components of its data collections using natural language processing and information extraction new partnerships created – new methods, tools and jargon to learn! new partnerships created – new methods, tools and jargon to learn! new area of application for NLP to social science data new area of application for NLP to social science data growing interest in UK in applying NLP and text mining to social science texts – data and research outputs such as publications’ abstracts growing interest in UK in applying NLP and text mining to social science texts – data and research outputs such as publications’ abstracts

19 Project progress defined areas of context for qualitative data defined areas of context for qualitative data drafted a metadata schema with mandatory elements drafted a metadata schema with mandatory elements built a Java GUI – with step-by-step components built a Java GUI – with step-by-step components data clean up tool data clean up tool named entity mark-up tools named entity mark-up tools annotation tool - NITE XML Toolkit annotation tool - NITE XML Toolkit extended functionality of ESDS Qualidata Online system to include links to audio-visual material, other documents, research outputs and mapping systems extended functionality of ESDS Qualidata Online system to include links to audio-visual material, other documents, research outputs and mapping systems

Defining context rich context enables informed re-use of data. But defining how to provide context for raw data to make it more ‘usable’ is complex both micro and macro level features should be considered detailed information on sampling procedures, field work approaches and question guides, analysis. Personal fieldwork observations timelines e.g events and political chronologies SQUAD has identified a minimal generic set of elements that represent a baseline for contextualising data QUADS workshop to address common problems. Papers being prepared for dedicated edited collection in Journal in Methodological Innovations Online sirius.soc.plymouth.ac.uk/~andyp/

Metadata standards in use DDI for Study description, Data file description, Other study related materials, links to variable description for quantified parts (variables) DDI for Study description, Data file description, Other study related materials, links to variable description for quantified parts (variables) for data content and data annotation: the Text Encoding Initiative for data content and data annotation: the Text Encoding Initiative standard for text mark-up in humanities and social sciences standard for text mark-up in humanities and social sciences used consultant to help text the TEI-conformant DTD used consultant to help text the TEI-conformant DTD evaluating other schema evaluating other schema

TEI Schema The XML schema will specify a ‘reduced’ set of Text Encoding Initiative (TEI) elements: core tag set for transcription names, numbers, dates links and cross references notes and annotations text structure unique to spoken texts linking, segmentation and alignment advanced pointing - XPointer framework text and AV synchronisation contextual information (participants, setting, text)

23 Metadata for model transcript output Study Name Mothers and daughters Study Name Mothers and daughters Depositor Mildred Blaxter Depositor Mildred Blaxter Interview number 4943int01 Interview number 4943int01 Date of interview 3 May 1979 Date of interview 3 May 1979 Interview ID g24 Interview ID g24 Date of birth 1930 Date of birth 1930 Gender Female Gender Female Occupation pharmacy assistant Occupation pharmacy assistant Geo region Scotland Geo region Scotland Marital status Married Marital status Married

24 Transcript with XML mark-up

XML: enabling a standardised format for interview transcripts

XML and XSL: enabling web-enabled display, search and browse

Automating XML mark-up Input data file

Data processed through Edinburgh LT- XML and CME tools The main Graphical User Interface (GUI) Invokes the SQUADCoder in NXT

NXT tool Locate the NXT metadata file The NXT generic window – running the SQUAD Coder

The SQUADCoder Window Transcription view The Named Entity Hierarchy All the references to a particular entity

Annotation tool - anonymise The Coreference Action Panel

Annotation tool Enter pseudonym

Anonymised data The Anonymised Transcription View

Annotated data what formats and how stored? NXT uses ‘stand off’ annotation – annotation linked to or references individual words NXT uses ‘stand off’ annotation – annotation linked to or references individual words uses the NITE NXT XML model uses the NITE NXT XML model creates new anonymised version creates new anonymised version intend to : intend to : save original file save original file save matrix of references - names to pseudonyms save matrix of references - names to pseudonyms outputs annotations – who worked on the file etc outputs annotations – who worked on the file etc

35 Enhancing multimedia display ESDS Qualidata Online XML enabling link to and simultaneously display: memos and annotations memos and annotations other documents other documents URLs URLs photos photos audio and video audio and video maps maps

Future work from Autumn: from Autumn: funding to formalising a data exchange standard funding to formalising a data exchange standard testing Qualitative Data Interchange Format – Australia Unis testing Qualitative Data Interchange Format – Australia Unis non-proprietary exchangeable bundle - metadata, data and annotation – expressed to RDF non-proprietary exchangeable bundle - metadata, data and annotation – expressed to RDF testing import and export from CAQDAS packages eg Atlas-ti testing import and export from CAQDAS packages eg Atlas-ti develop archiving tool for annotated data develop archiving tool for annotated data key word extraction systems to help conceptually index qualitative data – text mining collaboration key word extraction systems to help conceptually index qualitative data – text mining collaboration exploring grid-enabling data – e-science collaboration exploring grid-enabling data – e-science collaboration we welcome collaboration and testers we welcome collaboration and testers

44 Information ESDS Qualidata Online site: ESDS Qualidata Online site: SQUAD website: SQUAD website: quads.esds.ac.uk/projects/squad.asp Edinburgh NLP tools Edinburgh NLP tools NITE NXT toolkit: NITE NXT toolkit: ESDS Qualidata site: ESDS Qualidata site: SQUAD staff Louise Corti - UK Data Archive, Essex (PI) Louise Corti - UK Data Archive, Essex (PI) Claire Grover - LTG, Edinburgh (PI) Claire Grover - LTG, Edinburgh (PI) Libby Bishop - UK Data Archive, Essex Libby Bishop - UK Data Archive, Essex Maria Milosavljevic - LTG, Edinburgh Maria Milosavljevic - LTG, Edinburgh Mijail A. Kabadjov- LTG, Edinburgh Mijail A. Kabadjov- LTG, Edinburgh