Annotating Committed Belief Mona Diab, Lori Levin, Teruko Mitamura, Owen Rambow CMU/Columbia-CCLS

Motivating Example 1 Suppose: IR System Query: show documents discussing instances of peasants being robbed of their land o Document found 1: The people robbing Iraqi peasants of their land should be punished o Document found 2: Robbing Iraqi peasants of their land would be bad Irrelevant Relevant

Motivating Example 2 Suppose: QA System Question: Did the humanitarian crisis in Iraq end? o Text found 1: He arrived on Tuesday, bringing an end to the humanitarian crisis in Iraq. o Text found 2: He arrived on Tuesday, calling for an end to the humanitarian crisis in Iraq Answer: I do not know Answer: Yes

What is “Committed Belief”? Model: two discourse participants o speaker/writer (S) o hearer/reader (H) S and H have beliefs about the world o Note: knowledge = belief + encyclopedic truth; not relevant here o Note: discourse participants also have desires and intentions; not relevant here

What is “Committed Belief”? A lot of (but not all) communication is about S informing H of S’s beliefs o Patrick is a nice man : H believes nice(Patrick) o I hope Patrick is a nice man : H has no belief about Patrick being nice or not, only a desire Typically, S’s goal is that H adopt those beliefs Beliefs about the world can be of different strengths; linguistic means can signal strength: o I know Patrick is a nice man : Committed Belief o I think Patrick is a nice man : Non-Committed Belief Can use numerical weights to represent strength; we use 2 discrete categories o Committed belief o Non-committed belief

Annotation Categories We annotate propositions on the verb or noun Committed belief: S indicates in this utterance that S believes the proposition o I know Mark and Sandra have eloped. Non-committed belief: S identifies the proposition as something which S could believe, but S happens not to have a strong belief in the proposition o Mark and Sandra may have eloped. Not applicable: for S, the proposition is not of type in which S is expressing a belief, or could express a belief. Usually, this is because the proposition does not have a truth value in this world. o I wish Mark and Sandra would finally elope.

Why Is Recognizing Committed Belief Important? Committed-Belief Annotation Distinguishes o Propositions that are asserted as true (CB) o Propositions that are asserted but speculative (NCB) o Propositions that are not asserted at all (NA) Important whenever we need to identify facts o IR o QA o Summarization o …

Map from Committed Belief Detection to Useful Functionality Traditional IR System Or QA System Automatic Committed Belief Filter Response including unasserted propositions Response based on asserted propositions only

Committed Belied is not Tense CB = committed belief, NA = No asserted belief We have a special feature to indicate future tense on CB (committed belief) and NCB (non-committed belief) PastFuture CB Smith was assassinated. Smith will be assassinated tomorrow. NA I hope Smith regretted his acts. I hope Smith will regret his acts.

Committed Belied is not Factivity CB = committed belief, NA = No asserted belief Committed-belief annotation and factivity annotation are complementary FactOpinion CB Smith was assassinated. Smith was a nasty dictator. NA Smith will be assassinated. Smith will become a nasty dictator (once he is in power).

Some Examples Word File (English) Mona on Arabic

Status Have complete manual Have core corpus annotated (including Arabic), working on 15% double annotation

Proposed Annotation Work Perhaps annotate desire in same manner (much of NA is desire) Work on one language until prediction (across genres!) works, then move to next language o We do not know how much data is needed o We do not know how much variability in expressing committed beliefs between languages there is