Humanizing closed captioning: How captions transform the meaning and experience of the text Sean Zdenek Texas Tech University

Closed captioning tends to be defined narrowly. Just copy down what you hear (unreflective and objective transcription) A machine can do it (auto-captioning and speech-to- text) Any hearing person can do it (crowdsourcing Netflix, Amara) Captioning is just a variant of subtitling (speech only) Captioning is primarily a “technique used to display text” (WiseGeek 2014) Captioning is a technical problem (early Netflix, Vimeo)

We haven’t explored captioning as a text in a multimodal analysis How do captions create and shape meaning? What if our analyses started with captions instead of assuming that the real meaning is elsewhere? What are the differences and tensions between sound and writing, reading and listening? How do readers make sense of captioned texts? What is involved in offering a humanistic rationale for closed captioning?

Captions transform the meaning of the text. Captions create a new text (cf. Nornes 2007). Captioning is about meaning, not sound. Captioning is rhetorical invention or textual performance. The captioner is a rhetorical proxy agent. Examples: Seven transformations…

1. Captions clarify Captions tell us which sounds are important, what people are saying, and what non-speech sounds mean.

2. Captions contextualize Captioning is about meaning, not sound per se. Captions don’t describe sounds so much as convey the purpose and meaning of sounds in specific contexts. Turbine engine played backwards

3. Captions formalize/homogenize Captions tend to be presented in standard written English, with information about manner of speaking relegated to identifiers such as (drunken slurring). Nothing else about the speech will mark it as inflected or accented (e.g. drunk) except for a lone identifier at the beginning of the first speech caption.

Type of NSI District 9 (2009) Inception (2010) Man of Steel (2013) Oblivion (2013) 1680 closed captions 680 NSI captions NSI: 40.5% of CC 1:52:16 runtime 1741 closed captions 314 NSI captions NSI: 18% of CC 2:28:07 runtime 1469 closed captions 399 NSI captions NSI: 27.2% of CC 2:23:02 runtime 1026 closed captions 141 NSI captions NSI: 13.7% of CC 2:04:42 runtime Classic Speaker Identifiers 516 (75.7%)160 (51%)222 (55.6%)56 (39.7%) Other References to Names 51 (7.5%)31 (9.9%)49 (12.3%)6 (4.3%) Foreign Language Identifiers 61 (9%)2 (0.6%)4 (1%)0 Sound Effects37 (5.4%)80 (25.5%)73 (18.3%)27 (18.4%) Paralanguage90 (13.2%)71 (22.6%)108 (27.1%)43 (30.5%) Manner of Speaking Identifiers 2 (0.3%)1 (0.3%)1 (0.2%)11 (7.8%) Music32 (4.7%)11 (3.5%)1 (0.2%)5 (3.5%) Channel Identifiers51 (7.5%)3 (1%)17 (4.3%)3 (2.1%)

4. Captions equalize Every sound tends to play at the same “volume” on the caption track. While there are ways of modulating the volume of captioned sounds and differentiating background from foreground sounds in the captions, these ways are limited and space-consuming. As a result, every sound tends to occupy the same sonic plane, so that every sound appears equally “loud.”

5. Captions linearize Sounds that are heard simultaneously can not be read simultaneously. Captions linearize sound by presenting the soundscape in a form that can be read one sound/caption at a time.

6. Captions time-shift Under the right conditions and with the right readers, closed captions can manipulate time, transporting readers into the future or, in more general terms, providing them with advance or additional information. I coin the term captioned irony to account for this significant difference between captioned and uncaptioned texts.

7. Captions distill The soundscape is often pared down to its essential elements in the caption track. Ambient sounds tend to be reduced to single, one-off captions or not captioned at all. Music is distilled to a simple description and/or captioned music lyrics. Captions reconstruct the narrative as a series of elemental sounds, taming and rationalizing it as a result.