~ Multimodal Communication ~ HOW TO: From raw data to data annotation

Raw data: video file zfrom video camera to the computer zTASX compatible format: AVI for help with this: AVZ (Audio-visuelles Zentrum) to be found on N6, N7

Editing the video file with VirtualDub cutting the video/ making selections

Editing the video file with VirtualDub saving changes: File: Save as AVI select file name & location

Extracting audio from video with VirtualDub saving audio stream: File: Save WAV select file name & location

Data output zdigitized video file, e.g avi zdigitized sound file, e.g wav Why extract the sound file from the video file? --> separate speech description in Praat (or other tools, for that matter)

Speech annotation with Praat: Individual steps zload.wav file into Praat, zset up TextGrid (annotation tiers), and zEDIT; zannotate speech file according to individual needs (granularity of segmentation, means of transcription,...) zwrite TextGrid to text file (STRG-S), e.g TextGrid

Speech annotation with Praat: Illustration

From Praat file to TASX file applying script (praat-label2tasx) to Praat file for conversion from.TextGrid to.xml in order to make it compatible with TASX (output: e.g xml)

Confused? Brief review on previous steps zediting of video file (file.avi) zextraction of sound stream from video file (file.wav) zannotation of speech in Praat (or some other tool...) (file.TextGrid) zconversion of file.TextGrid into TASX compatible format (file.xml)

What is TASX? zTASX - Time Aligned Signal data eXchange (XML based) ztool for the annotation of multimodal (audio & video) data (cf. Praat for audio only) zSource:

The TASX annotator za TASX-annotated corpus consists of a set of sessions zeach session holds an arbitrary number of descriptive tiers or layers zeach layer consists of a set of separated events zeach event holds some kind of textual information (label) and is linked to the primary audio or video data by means of two time stamps (marking the beginning and the end of an interval)

Terminology corpus, plural corpora: A collection of linguistic data, either compiled as written texts or as a transcription of recorded speech. The main purpose of a corpus is to verify a hypothesis about language - for example, to determine how the usage of a particular sound, word, or syntactic construction varies. [...] (cf. Crystal, David An Encyclopedic Dictionary of Language and Languages. Oxford, 85)

Interface of the TASX annotator Illustration tiers or layers segments or events labels: e.g. “mom“

Getting started... Part I zStart the TASX annotator. zLoad primary video: File - Load primary video or STRG-W (.avi file) zLoad audio file (.wav file, optional; loading audio file causes TASX annotator to generate oscillogram) zImport speech annotation: File - Import from - TASX (.xml file) zIn case there is already an existing annotation file (.tbf file), file can simply be loaded into the tool: File - Open (new format) or STRG-O

Getting started... Part II zFile - Merge (new format) merges two separate annotation files zAdd (an) extra tier(s): Tier - New tier or Shift-N zRename extra tier(s): yActivate tier to be renamed by mouseclick (changes to green) yTier - Rename tier or Shift-R yType in new tier name z Save complete file: File - Save as... (new format) or STRG-S resulting output file: file.tbf

Getting started... Part III zFor further details, see the manual: And now, let‘s get started!