1. Natural Language Processing
Discourse, Entity Linking, and Pragmatics

2. Semantics Road Map Lexical semantics Disambiguating words
Word sense disambiguation
Coreference resolution
Semantic role labeling
Meaning representation languages
Discourse and pragmatics
Compositional semantics, semantic parsing
-We’re extracting progressively more complicated forms of meaning from language.
-Started with zero semantics: just parse trees
-Then: Semantics of words in isolation; then predicates (semantic structures)
-Now we’re going to look at how sentences relate to each other
-We will spend more time than usual talking about theory

3. How Do Sentences Relate to Each Other?
John hid Bill’s car keys. He was drunk.
*John hid Bill’s car keys. He likes spinach.
-First example: these aren’t arbitrary sentences tossed together
--They have a relationship

4. Another Example
“Near-death experiences can help one see more clearly sometimes,” said Steve Jobs. He was speaking about struggling companies. Yet he could easily have been talking about his own life. In 1985 Mr Jobs was pushed out of Apple Computer, the firm he had helped found, only to return after a decade away. In doing so, he mounted one of capitalism’s most celebrated comebacks.
*Yet he could easily have been talking about his own life. “Near-death experiences can help one see more clearly sometimes,” said Steve Jobs. In doing so, he mounted one of capitalism’s most celebrated comebacks. In 1985 Mr Jobs was pushed out of Apple Computer, the firm he had helped found, only to return after a decade away. He was speaking about struggling companies.
-The second version of the passage has the same sentences in a different order.
-All the same sentences are still there in the second version. What went wrong?
-Do they make sense? Maybe, but not in the way that the writer originally intended.

5. What Is Discourse?
Discourse is the coherent structure of language above the level of sentences or clauses. A discourse is a coherent structured group of sentences.
What makes a passage coherent?
-Discourse is the scaffolding that holds statements together in a passage of text or spoken language.
-[before the reveal] Tell me in your own words what coherence means?
A practical answer: It has meaningful connections between its utterances.

6. Applications of Computational Discourse
Automatic essay grading
Automatic summarization
Dialogue systems
-automatic essay grading: ETS uses some (simple) computational discourse as part of how they automatically score essays
--analysis for coherence
--comparison to discourse features of well-written essays
-summarization: discourse analysis can tell us things about the relative importance of sentences
-dialogue systems: I’ll say more about this later
--think about the relationship between a question and an answer in a conversation
-think about the different ways we make requests without commanding a person to do something

7. Discourse Segmentation
Goal: Given raw text, separate a document into a linear sequence of subtopics.
-Think about the structure of a news article: an “inverted pyramid”
-Think about the structure of academic papers: Abstract, Introduction, Methodology, Results, Conclusion
-On the right: an example from the book; this is a sequence of topics in a science news article
--The numbers here are for paragraphs rather than sentences
--But they could be for sentences
Pyrmaid from commons.wikimedia.org

8. Cohesion
Relations between words in two units (sentences, paragraphs) “glue” them together. Before winter I built a chimney, and shingled the sides of my house… I have thus a tight shingled and plastered house. Peel, core, and slice the pears and apples. Add the fruit to the skillet.
-In the first example here, we have lexical cohesion between two sentences: cohesion via words.
-In the second example we have anaphora: replacing a word by other words

9. Supervised Discourse Segmentation
Our instances: place markers between sentences (or paragraphs or clauses)
Our labels: yes (marker is a discourse boundary) or no (marker is not a discourse boundary)
What features should we use?
-Let’s think about treating this as a supervised machine learning problem.
-If we’re looking at sentences, paragraph boundaries; if clauses, sentence and paragraph boundaries
Discourse markers or cue words
Word overlap before/after boundary
Number of coreference chains that cross boundary
Others?

10. Evaluating Discourse Segmentation
-To evaluate, we need gold standard data to compare against (“ref”)
--This is usually text in which boundaries have been labeled by humans
-We generally don’t use precision and recall to measure discourse segmentation
--They’re not sensitive to near misses
-The method shown here is WinDiff
-The basic idea: we run a sliding window over the two sequences of sentences and count the differences in the number of boundaries
--Size of the lsiding window: half the average segment length
-The calculation at the bottom results in 0 if all boundaries are correct

11. Some Coherence Relations
How can we label the relationships between utterances in a discourse? A few examples:
Explanation: Infer that the state or event asserted by S1 causes or could cause the state or event asserted by S0.
Occasion: A change of state can be inferred from the assertion of S0, whose final state can be inferred from S1, or vice versa.
Parallel: Infer p(a1, a2,…) from the assertion of S0 and p(b1, b2,…) from the assertion of S1, where ai and bi are similar for all i.
-Recall from a few slides ago what coherence is (the presence of meaningful connections between utterances)
-Let’s consider our utterances to be individual sentences or sequences of sentences.
--We can characterize relationships between them.
-Give me examples of these.
-Parallel is loosely defined. Think of two utterances that serve a similar purpose in some context.
--Example: The Scarecrow wanted some brains. The Tin Man wanted a heart.
--Example: I parked my car in the driveway. My friend chained his bike to the railing.

12. Discourse Structure from Coherence Relations
-We can consider a hierarchical structure between coherence relations in a discourse.
-Read the sentences, then parse the diagram.

13. Automatic Coherence Assignment
Given a sequence of sentences or clauses , we want to automatically:
determine coherence relations between them (coherence relation assignment)
extract a tree or graph representing an entire discourse (discourse parsing)
-Do these tasks sound easy or hard?

14. Automatic Coherence Assignment
Very difficult. One existing approach is to use cue phrases.
John hid Bill’s car keys because he was drunk.
The scarecrow came to ask for a brain. Similarly, the tin man wants a heart.
Identify cue phrases in the text.
Segment the text into discourse segments.
Classify the relationship between each consecutive discourse segment.
-Automatic coherence assignment is considered an open problem.
-Cue phrases are also known as discourse markers or cue words.
-To some extent cue phrases are domain dependent.
--In transcripts of news broadcasts: “joining us now”, or “up next”
--In scientific papers: “in this section”, “here, we present”

15. Entity Linking

16. A Lead-In: Reference Resolution
John Chang, Chief Financial Officer of Megabucks Banking Corp since 2004, saw his pay jump 20%, to $1.3 million, as the 37-year-old also became the Denver-based financial-services company’s president. It has been ten years since he came to Megabucks from rival Lotsabucks.
-What’s the name for the NLP problem of determining when two linguistic entities refer to the same entity?
-In contrast, now we’re interested in determining the things that those linguistic entities (particularly noun phrases) refer to.

17. Reference Resolution
Goal: determine what entities are referred to by which linguistic expressions. The discourse model contains our eligible set of referents.
-You can think of this problem as a layer on top of coreference resolution
--Often it is necessary to identify coreference chains as part of entity linking

18. Five Types of Referring Expressions
Indefinite noun phrases
I saw a beautiful Ford Falcon today.
Definite noun phrases
I read about it in the New York Times.
Pronouns
Emma smiled as cheerfully as she could.
Demonstratives
Put it back. This one is in better condition.
Names
Miss Woodhouse certainly had not done him justice.
-Some of these can take a few different forms.
--For example, indefinite noun phrases are sometimes marked by “some” or “this”

19. Entity Linking
Apple updated its investor relations page today to note that it will announce its earnings for the second fiscal quarter (first calendar quarter) of 2015 on Monday, April 27.
-We’ll talk briefly about a specific kind of reference resolution called entity linking
-In entity linking, we have a knowledge base of entities and some information about them
-We want to know specifically when a given piece of text refers to entities in the knowledge base
-Why is this hard?
--Names are ambiguous
News text from

20. One Approach to Entity Linking
Use supervised learning: Train on known references to each entity. Use features from context (bag of words, syntax, etc.).
-This approach has worked well when training on Wikipedia text.
-Each incoming link to a Wikipedia article is a training instance for the entity represented by that article.

21. Pragmatics

22. Pragmatics
Pragmatics is a branch of linguistics dealing with language use in context. When a diplomat says yes, he means ‘perhaps’; When he says perhaps, he means ‘no’; When he says no, he is not a diplomat. (Variously attributed to Voltaire, H. L. Mencken, and Carl Jung)
-This third topic for today has some relevance to text but a lot of relevance to conversation.
--These slides are framed with the context of conversation: easier to discuss
-How can “yes” mean “perhaps”, and how can “perhaps” mean “no”?
--It’s the context of diplomacy and a diplomat speaking.
-To be clear, there are definitely contexts when “yes” and “no” are literal
--Because people aren’t always diplomats
Quote from

23. In Context? Social context Physical context Linguistic context
Social identities, relationships, and setting
Physical context
Where? What objects are present? What actions?
Linguistic context
Conversation history
Other forms of context
Shared knowledge, etc.
-Social context: example on the previous slide
-Physical context: (You’re sitting by a window) “It’s cold in here.”
-Linguistic context: (We’ve been talking about leaving for a restaurant) “Let’s go now.”

24. Speech Act Theory
“I’ll give the lecture today.” “It’s cold in here.” "This administration today, here and now, declares unconditional war on poverty in America.” “I now pronounce you man and wife.”
-By saying things, we actually do things.
-We intend consequences when we speak (maybe like “sideaffects” in programming languages)
-#1 on this slide is a promise of sorts: the person I’m speaking to recognizes my commitment and intent
-#2 we saw on a previous slide: if I say it to someone sitting by the thermostat, it’s a request
-#3 is a different kind of promise: LBJ (US president) said this In 1964; it’s a commitment of resources
-#4 is a classic example: the officiant at a wedding ceremony says this
--We recognize it as making the marriage exist in a ceremonial way

25. Speech Act Theory in NLP
Let’s say that I’m building a system that will interact with people conversationally. Is speech act theory relevant? Why?
-I won’t formally introduce dialog systems here, but this can be a teaser to one of Alan’s later lectures.
-Try giving Siri a speech act like “I’m hungry” and see what it does.
-It doesn’t just accept commands, because people don’t make requests of each other solely in commands.

26. Grice’s Maxims
Quantity: Make your contribution as informative as required, but no more
Quality: Try to make your contribution one that is true
Relation: Be relevant
Manner:
Don’t be obscure
Avoid ambiguity
Be brief
Be orderly
The Pragmatics Handbook

27. Grice’s Maxims in NLP
Let’s say that I’m building a system that will interact with people conversationally. How are Grice’s Maxims relevant?
-Let’s try this again with Grice’s Maxims.
-We want to create a system that communicates efficiently and clearly.

28. -The topics we’ve covered in this lecture are some examples of particularly super-ambitious NLP.
--Different kinds of difficulty: improving upon 97% accuracy vs. improving upon 30%
--These are problems that require advanced language facilities that we have difficulty mimicing computationally.
-Automatic systems that attack them straight-on do not perform well.
--Discourse parsing: hard
-Worth noting: still some useful applications
--Automatic essay grading: used in production systems
--Systems like Siri are built to identify certain speech acts
Cover of Shel Silverstein’s Where the Sidewalk Ends (1974)