Analysing the ELICIT factoids - example of uncertainty David Mott (ETS, IBM UK) April 2014 v1.1 David Mott (ETS, IBM UK) April 2014 v1.1 International Technology Alliance In Network & Information Sciences International Technology Alliance In Network & Information Sciences

ELICIT is… An experimental framework for researching into collective sensemaking by teams of collaborating humans –A set of Natural Language sentences providing information about a scenario in a fictitious geographic region with a number of different countries –There is a possible attack by a group on a target in one of these countries –The collaboration of teams must identify, from the information the who, what, where, and when of the attack –In different experiments, the teams are given only partial information, and the task is to communicate and collaborate to do the identification

ELICIT sentences (factoids) 1 The Lion is involved 2 Word has it that an unprotected target is preferred to ensure the likelihood of success (can assume is true) 3 The Lion doesn't operate in Chiland 4 The Lion attacks in daylight 5 The Azure, Brown, Coral, Violet, or Chartreuse groups may be planning an attack 6 The Azure and Violet groups use only their own operatives, never employing locals 7 The Chartreuse group is not involved 8 The Lion is known to work only with the Azure, Brown, or Violet groups 9 The Purple or Gold group may be involved 10 All of the members of the Azure group are now in custody 11 Reports from the Coral group indicate a reorganization 12 There is a lot of activity involving the Violet group 13 The Brown group is recruiting locals - intentions unknown 14 The Lion will not risk working with locals 15 The Jackal has been seen in Tauland 16 Members of the Purple group have been visiting Omegaland 17 The Chartreuse group has close ties with local media 18 The Azure group has a history of attacking embassies 19 The Purple and Gold groups have blood ties 20 The Brown group has been known to use IED's 21 Only the Coral and Violet groups have a capacity to hit protected targets 22 All high value targets belonging to Tauland and Epsilonland are well protected 23 The attackers are focusing on a high visibility target 24 Caches of explosives have recently been found in Epsilonland, Chiland, and Psiland 25 Financial institutions in Tauland, Chiland, and Omegaland were recently attacked there is evidence of more attacks 26 Reports that uniforms were stolen in Tauland, Epsilonland and Psiland 27 Bloggers are discussing the role of financial institutions in oppressing the Coral, Violet and Chartreuse groups 28 Members of the Violet and Chartreuse groups were active in planning protests at a recent financial summit 29 Security forces are providing highly visible, around the clock protection to all visiting dignitaries in the region 30 Dignitaries in Epsilonland employ private guards 31 Tau, Epsilon, Chi, Psi and Omega-lands are providing visible, around the clock protection to their own dignitaries at home 32 A new train station is being built in the capital of country Tauland 33 Tauland's embassy in Epsilonland has a flat roof 34 Until recently most of the dignitaries in Tauland rode in Mercedes 35 Dignitaries in Chiland have motorcycle escorts 36 Epsilonland's embassy in Tauland has two helicopter pads 37 The Azure, Brown, Coral, and Violet groups have the capacity to operate in Tau, Epsilon, Chi, Psi and Omega-lands 38 Locals in Tauland, Epsilonland and Omegaland are being recruited 39 Countries Chiland, Psiland and Omegaland are taking steps to protect their embassies abroad 40 The Brown group members have entered Tauland and Epsilonland 41 Reports from Tauland, Chiland and Psiland indicate surveillance ongoing at coalition embassies 42 The target is a coalition member embassy, visiting dignitary, or financial institution (Tau, Epsilon, Chi, Psi or Omega-lands) 43 No traces of members from the Coral group have been found in countries Psiland or Omegaland 44 Chiland is in the process of deploying troops to protect the embassies of coalition partners 45 The Azure, Brown, and Coral groups want to attack the interests of Tauland, Epsilonland or Chiland 46 The Coral and Violet group operatives have entered Psiland 47 All high value targets of Omegaland are well protected 48 There has been an increase in messages intercepted in Psiland 49 The Lion was born in Tauland 50 There is no new information about Brown group operations in Chiland 51 Epsilonland is mountainous 52 Tauland is land locked 53 The attack will be at 11:00 54 The Azure and Brown groups prefer to attack at night 55 The Tauland embassy in Epsilonland is hosting a international conference on the 10th 56 The Chartreuse, Purple and Gold groups are known to attack at any time of the day 57 Attacking buildings when there are many people present increases casualties 58 The Coral, Chartreuse and Purple groups are capable of attacking year round 59 The Lion is planning something in April on the anniversary of his father's death 60 There are fewer attacks in the dead of winter (January thru March) 61 The Violet and Chartreuse groups want to attach the interests of Chiland, Psiland and Omegaland 62 The Violet group is planning something big on the 5th 63 The Violet group prefers to operate in daylight 64 The lion was born in June 65 The Coral group prefers to attack at night 66 The Purple group prefers to attack in daylight 67 The Brown group needs time to regroup 68 The Azure group does not attack on its holy days WHO (agents involved), WHAT (target of attack), WHEN, WHERE?

A stretched goal for task 2 I was told that the sentences were trivially easy to turn into a formal representation which was then a “logic puzzle” to be solved, so why not: 1.Pass the ELICIT sentences through the ERG 2.Convert the output to Controlled English, after modelling the domain concepts 3.Run a logic puzzle solver that interprets CE 4.Solve the problem! 5.Thus demonstrating the power of our system to perform analysis in a valid problem, allowing us to participate in the sensemaking experiments of task 1? The sentences ran through the ERG system –Some minor syntactic changes X’s Y did not parse, so I changed to the Y of X … –Most sentences parsed, though I have not checked all of the parses for reasonableness. However there is a big problem…

Ambiguity On analysing the sentences formally there are many ambiguities and difficulties of interpretation: –The Purple group or the Gold group may be involved What is the meaning of “or” in this context ? Inclusive? exhaustive? –The attackers are focusing on a high visibility target What does high visibility mean? Only domain knowledge or common sense can tell –The Lion will not risk working with locals Does this mean the Lion will not work with locals? –Dignitaries in Epsilonland employ private guards Does “in” mean “belonging to” or “located in”? –Reports from Tauland, Chiland and Psiland indicate surveillance ongoing at coalition embassies Are the reports from the embassies or the host countries? –The Violet group prefers to operate in daylight Does this exclude artificial light, in this context? How do we compare this with “at night” –The Azure and Violet groups use only their own operatives, never employing locals Actually this is a rule not a fact, (using present tense?) but how do we generalise the statement just enough to capture the intent? I analysed all 68 sentences in some detail as to ambiguities! So even “simple” sentences are difficult to analyse; common sense knowledge is required

Handling uncertainty We have: –defined CE domain model –generated CE facts from sentences by  Constructing CE sentences “by hand”  This has required “interpretation” of the sentences, introducing uncertainty factors as assumptions –Run the sentences in the CE system, leading to a possible solution to the who, what, where and when –Recorded the rationale for the reasoning  Including how the conclusions are based upon assumptions This should provide an example: –how the reasoning can be affected by the use of assumptions

Flow of Information Original sentences MRS WHO Simplified sentences Domain CE MRS Domain CE Domain Model (Concepts, Rules) WHEN Problem Solving Strategy WHAT WHERE ERG system CE system Manual Reasoning Engine Manual Explanation This shows the approach in the context of our three different possible approaches

Uncertainty about protected targets (1) Word has it that an unprotected target is preferred to ensure the likelihood of success (can assume is true) Original sentence: if ( the potential target T is a protected thing ) then ( the potential target T is a non-target ). Actually, because we are taking a constraint-based problem solving strategy, we would represent this in the opposite negative form if ( the thing T is a non-protected thing ) then ( the thing T is a possible target ). We could just take this preference as absolutely true, and infer that non- protected things are possible targets (The negative form could be derived automatically from the positive form via modus tollens, but this latter reasoning rule has not been implemented in the CE reasoning engine)

Uncertainty about protected targets (2) [ rule_prot_targ_non_target ] if ( the potential target T is a protected thing ) and ( the domain interpretation prot_targ_non_target can be made by the rule rule_prot_targ_non_target ) then ( the potential target T is a non-target ). However the assumption that this is actually true is an “interpretation” of the sentence in the context of the ELICIT domain and may be too strong it is assumed by the agent dm that the domain interpretation prot_targ_non_target can be made by the rule rule_prot_targ_non_target. there is a domain interpretation named prot_targ_non_target that has "the preference is treated as an absolute truth" as description. and we ASSUME that this interpretation can be made by the rule as a necessary premise to infer its conclusion We wish to explicitly record this interpretation as part of domain knowledge

Uncertainty about protected targets (3) [ rule_prot_targ_non_target ] if ( the potential target T is a protected thing ) and ( the domain interpretation prot_targ_non_target can be made by the rule rule_prot_targ_non_target ) then ( the potential target T is a non-target ). Now any application of the rule will make the conclusions dependent upon the assumption as well as the basic premise propositions the potential target t1 is a non-target the potential target t1 is a protected thing it is assumed by the agent dm that the domain interpretation prot_targ_non_target can be made by the rule rule_prot_targ_non_target. Providing this information in the rationale can show the analyst this source of uncertainty

Uncertainty about security in the region (1) Security forces are providing highly visible, around the clock protection to all visiting dignitaries in the region Original sentence: But how do we interpret “in the region”? We could take the view that all of the countries mentioned in the ELICIT sentences are in the region we have called “this region” [ visiting_digs_protected ] if ( the visiting dignitary D is visiting the country HOSTC ) and ( the country HOSTC is located in the region thisregion ) then ( the visiting dignitary D is a protected thing ). We want to represent the inference that visiting dignitaries in the region are protected, as in this rule if ( there is a region named thisregion ) and ( there is a country named C ) and then ( the country C is located in the region thisregion ).

Uncertainty about security in the region (2) However the assumption that this is actually true is an “interpretation” of the sentence and may be too strong it is assumed by the agent dm that the sentence interpretation si29 can be made by the rule assume_allcountries_in_region. the sentence interpretation si29 has "we take the region to include all countries" as description. and we ASSUME that this interpretation can be made by the rule as a necessary premise to infer its conclusion We wish to explicitly record this interpretation as part of interpreting the sentence [ assume_allcountries_in_region ] if ( there is a region named thisregion ) and ( there is a country named C ) and ( the sentence interpretation si29 can be made by the rule assume_allcountries_in_region ) then ( the country C is located in the region thisregion ). Note we have different types of interpretation, those general to the domain and those specific to a sentence

Identification requires assumptions The reasoning that identifies the WHAT and WHERE seems dependent upon these and other assumptions e.g.: –That protected things are non-targets (domain interpretation) –That all countries are in “this region” (sentence interpretation) –That we know the total set of possible participants and possible targets (domain interpretation) –That “in” means working for, not located in (sentence interpretation) –That there is only one attack situation being talked about (domain interpretation) We can generate the rationale for the reasoning, and this shows the assumptions used

“Proof tables” Why cant the Lion work with the Coralgroup? propositions rule Each column is a rule application Red cell is the conclusion Salmon cells are the premises In the second column a green cell indicates a predefined fact and pink represents an assumption and a word represents the name of a rule leading to the conclusion The Equivalent Rule in CE [ no_time_overlap : ] if ( the operative A operates in the time interval TA ) and ( the group B operates in the time interval TB ) and ( the time interval TA does not overlap the time interval TB ) then ( the operative A cannot work with the group B ). We will show the rationale in the form of “proof tables”

Determining the “WHAT” In our constraint-based approach we find the WHAT (i.e. the target) by eliminating all other possibilities One such possibility is “ChilandDVE” representing the Chiland dignitaries visiting Epsilonland: –there is a dignitary named ChilandDVE that is an official of the country Chiland and is located in the country Epsilonland. We eliminate ChilandDVE by proving: –the possible target ChilandDVE is a non-target.

Reasoning for “Chiland is a non-target” Assume all countries are in the region Assume protected things are non- targets

Assumptions used It can be seen that the proof of “the possible target ChilandDVE is a non-target” involves several assumptions, based upon an interpretation of the original sentences –Word has it that an unprotected target is preferred to ensure the likelihood of success (can assume is true) –Security forces are providing highly visible, around the clock protection to all visiting dignitaries in the region All countries mentioned are in this “region”

Could Argumentation be used? Could arguments be had in order to construct the assumed interpretations of the sentences and the domain logic? –The paper on “On Interpreting the ELICIT sentences” has some examples of informal arguments and what the sentences meanhttps://

For example: For the sentence: –Security forces are providing highly visible, around the clock protection to all visiting dignitaries in the region In the initial analysis (done on first exposure to the sentences, with no preconceived knowledge on the author’s part), it is stated: –This sentence is highly ambiguous, since it does not state the region in question. It seems unlikely that this refers to all regions, so it is hard to see how this information could be used in inference. In subsequent work to formulate the sentences, the view changed to the interpretation that “all countries are in the region”, since the alternative view that “only some are in the region” would not make sense given that: – there is no information given about the membership of countries in any region –There is no mention of specific regions, just the idea of “the region” –The sentence would be meaningless if either the initial analysis were true or there were just “some” countries in the region without stating which –It is unlikely that a sentence is given that is meaningless