1. Tasking the Tweeters: Obtaining Actionable Information from Human Sensors Alun Preece, Will Webberley (Cardiff) Dave Braines (IBM UK)

2. Introduction

3. Social Media for Real-time Intel Social Media streams as sources of actionable intelligence for Situation Awareness (SA) Acknowledging a human-based sensor network A good source today: Twitter –Real-time characteristics –Follower-based model –Open APIs Real-world examples from recent events: –Boston marathon bombing (US) –Lee Rigby murder (UK) Some SA platforms emerging: Twitcident, Apollo, ReDites, Sentinel

4. Mapping Social Media to DCPD Direction - what data to collect from where Collection e.g. Twitter: APIs for streaming, searching sampling (other platforms available too) Post-filtering for noise reduction Processing e.g. probabilistic, NLP, sentiment, event detection Provide semantic enrichment (contextual) for Shared Understanding. Detect trends, clusters, anomalies etc Dissemination Visualisation, alerting, summarisation Further querying Direct further collections A generic social media processing pipeline mapped to DCPD steps

5. Dynamic ISR asset management  Missions-and-means framework formalised as a collection of ontologies  Tasks characterised by the data needed to achieve them  type of data (visual, IR, radar etc )  “quality” rating 0 to 9  Assets rated by the data they provide MMF framework NIIRS-based approach  Software tool for agile sensor-task assignment  Extensible knowledge-base of sensor-task suitability  Uses existing models and frameworks to map capabilities Sensor Assignment to Missions (SAM) In previous work we have defined a framework for dynamic ISR asset management:

6. A Pilot Study

7. July 26 th, 2014: Cardiff protest march Planned protest march in Cardiff, UK Against Israeli incursions into Gaza Potential for public order disruption Approximately 2,000 people Some limited local trouble Evidence of protest and activities found on Twitter –Real-time during the event –In various stages afterwards Source: Wales online – www.walesonline.co.ukwww.walesonline.co.uk

8. Social Media Timeline Timeline of the July 26 th 2014 protest and its aftermath

9. Social Media Timeline Timeline of the July 26 th 2014 protest and its aftermath UK-wide tweets

10. Social Media Timeline Timeline of the July 26 th 2014 protest and its aftermath Verbal and physical abuse at bars [15:15] UK-wide tweets

11. Social Media Timeline Timeline of the July 26 th 2014 protest and its aftermath Verbal and physical abuse at bars [15:15] March ends [15:40] UK-wide tweets

12. Social Media Timeline Timeline of the July 26 th 2014 protest and its aftermath Verbal and physical abuse at bars [15:15] March ends [15:40] Tweeting after the march UK-wide tweets

13. Social Media Timeline Timeline of the July 26 th 2014 protest and its aftermath Verbal and physical abuse at bars [15:15] March ends [15:40] Tweeting after the march Police mentions increase after broadcast news UK-wide tweets

14. Social Media Timeline Timeline of the July 26 th 2014 protest and its aftermath Verbal and physical abuse at bars [15:15] March ends [15:40] Tweeting after the march Police mentions increase after broadcast news UK-wide tweets Important: We are observing perception of the event, not the event itself…

15. Practical details Sentinel Twitter Stream Analysis Geo-tagged tweets Topical search terms Mentions of local places People on the ground Access to live twitter (+ search) Manually identify “key” tweets Some issues Generality of tweets Crowd size estimation: “a few hundred”, “thousands” Very few tweets geo-tagged The Sentinel application

16. Observations from the pilot Sweet spot for initial relevancy: Search terms + geo- spatial Social Media reflects perception, not reality We are not claiming that this simple study is representative. Key events and activities can be detected: –…but how early can these be found through “small signals”? Some issues with Social Media: –Propagation of misinformation –Detection of bias –Quantification of contextual factors There is the potential to inform action via this kind of situation awareness

17. Modeling Tweets and Tweeters

18. Background: CNL for conversation Need an appropriate form for human-machine interaction:  humans prefer natural language (NL) or images  these forms are difficult for machines to process, leading to ambiguity and miscommunication Compromise: controlled natural language (CNL) there is a person named p1 that is known as ‘John Smith’ and is a person of interest. low complexity | no ambiguity ITA Controlled English (CE)

19. Defining sources and people Sources, e.g. a Twitter account: conceptualise a ~ twitter account ~ A that is an online identity and is a temporal thing and has the value L as ~ location ~ and has the value NT as ~ number of tweets ~ and has the web image PP as ~ profile picture ~ and has the value NT as ~ number of tweets ~ and has the value NFR as ~ number of friends ~ and has the value NFO as ~ number of followers ~. conceptualise a ~ twitter account ~ A that is an online identity and is a temporal thing and has the value L as ~ location ~ and has the value NT as ~ number of tweets ~ and has the web image PP as ~ profile picture ~ and has the value NT as ~ number of tweets ~ and has the value NFR as ~ number of friends ~ and has the value NFO as ~ number of followers ~. there is a journalist named ‘Paul Heaney’ that uses the twitter account ‘paulheaney67’ and works for the media organization ‘bbc’. there is a journalist named ‘Paul Heaney’ that uses the twitter account ‘paulheaney67’ and works for the media organization ‘bbc’. People (and their derivation from a source): …we are actually building profiles of “human sensors”.

20. Human Sensor profiles The following information is available for inclusion in the human sensor profile: All data from their Twitter profile (including location) Who they frequently interact with Who they talk about Who are their influencers Recently posted media (photos, videos) Terms names from recent tweets Locations from recent tweets –Including travel to/from locations Sentiment analysis for tweets and terms The use of our human friendly CNL means that additional “local knowledge” can easily be added too. e.g. “stance” – to capture some importance contextual detail This is a dynamic social network

21. Talking to Moira An example Moira query showing some elements of the tweeter model All this information (people, sources, tweets, terms, events etc) is available in a CNL knowledge base. The Moira agent is able to access this and support conversation with human team members…

22. Tasking Tweeters

23. Defining ISR tasks From our previous work: conceptualise the task T ~ requires ~ the intelligence capability IC and ~ is looking for ~ the detectable thing DT and ~ operates in ~ the spatial area SA and ~ operates during ~ the time period TP and ~ is ranked with ~ the task priority PR. conceptualise the task T ~ requires ~ the intelligence capability IC and ~ is looking for ~ the detectable thing DT and ~ operates in ~ the spatial area SA and ~ operates during ~ the time period TP and ~ is ranked with ~ the task priority PR.

24. Defining ISR tasks From our previous work: conceptualise the task T ~ requires ~ the intelligence capability IC and ~ is looking for ~ the detectable thing DT and ~ operates in ~ the spatial area SA and ~ operates during ~ the time period TP and ~ is ranked with ~ the task priority PR. conceptualise the task T ~ requires ~ the intelligence capability IC and ~ is looking for ~ the detectable thing DT and ~ operates in ~ the spatial area SA and ~ operates during ~ the time period TP and ~ is ranked with ~ the task priority PR. The “action” – what you are trying to achieve

25. Defining ISR tasks From our previous work: conceptualise the task T ~ requires ~ the intelligence capability IC and ~ is looking for ~ the detectable thing DT and ~ operates in ~ the spatial area SA and ~ operates during ~ the time period TP and ~ is ranked with ~ the task priority PR. conceptualise the task T ~ requires ~ the intelligence capability IC and ~ is looking for ~ the detectable thing DT and ~ operates in ~ the spatial area SA and ~ operates during ~ the time period TP and ~ is ranked with ~ the task priority PR. The “action” – what you are trying to achieve What you are trying to do, e.g. “detect”, “localize”

26. Defining ISR tasks From our previous work: conceptualise the task T ~ requires ~ the intelligence capability IC and ~ is looking for ~ the detectable thing DT and ~ operates in ~ the spatial area SA and ~ operates during ~ the time period TP and ~ is ranked with ~ the task priority PR. conceptualise the task T ~ requires ~ the intelligence capability IC and ~ is looking for ~ the detectable thing DT and ~ operates in ~ the spatial area SA and ~ operates during ~ the time period TP and ~ is ranked with ~ the task priority PR. The “action” – what you are trying to achieve What you are trying to do, e.g. “detect”, “localize” From a predefined ISR ontology

27. Defining ISR tasks From our previous work: conceptualise the task T ~ requires ~ the intelligence capability IC and ~ is looking for ~ the detectable thing DT and ~ operates in ~ the spatial area SA and ~ operates during ~ the time period TP and ~ is ranked with ~ the task priority PR. conceptualise the task T ~ requires ~ the intelligence capability IC and ~ is looking for ~ the detectable thing DT and ~ operates in ~ the spatial area SA and ~ operates during ~ the time period TP and ~ is ranked with ~ the task priority PR. The “action” – what you are trying to achieve What you are trying to do, e.g. “detect”, “localize” From a predefined ISR ontology From a gazetteer or similar

28. Defining ISR tasks From our previous work: conceptualise the task T ~ requires ~ the intelligence capability IC and ~ is looking for ~ the detectable thing DT and ~ operates in ~ the spatial area SA and ~ operates during ~ the time period TP and ~ is ranked with ~ the task priority PR. conceptualise the task T ~ requires ~ the intelligence capability IC and ~ is looking for ~ the detectable thing DT and ~ operates in ~ the spatial area SA and ~ operates during ~ the time period TP and ~ is ranked with ~ the task priority PR. The “action” – what you are trying to achieve What you are trying to do, e.g. “detect”, “localize” From a predefined ISR ontology From a gazetteer or similar To establish temporal bounds

29. Defining ISR tasks From our previous work: conceptualise the task T ~ requires ~ the intelligence capability IC and ~ is looking for ~ the detectable thing DT and ~ operates in ~ the spatial area SA and ~ operates during ~ the time period TP and ~ is ranked with ~ the task priority PR. conceptualise the task T ~ requires ~ the intelligence capability IC and ~ is looking for ~ the detectable thing DT and ~ operates in ~ the spatial area SA and ~ operates during ~ the time period TP and ~ is ranked with ~ the task priority PR. The “action” – what you are trying to achieve What you are trying to do, e.g. “detect”, “localize” From a predefined ISR ontology From a gazetteer or similar To establish temporal bounds For simple resource scheduling

30. Defining Social Media ISR tasks conceptualise the task T ~ requires ~ the intelligence capability IC and ~ is looking for ~ the detectable thing DT and ~ operates in ~ the spatial area SA and ~ operates during ~ the time period TP and ~ is ranked with ~ the task priority PR. conceptualise the task T ~ requires ~ the intelligence capability IC and ~ is looking for ~ the detectable thing DT and ~ operates in ~ the spatial area SA and ~ operates during ~ the time period TP and ~ is ranked with ~ the task priority PR.

31. Defining Social Media ISR tasks conceptualise the task T ~ requires ~ the intelligence capability IC and ~ is looking for ~ the detectable thing DT and ~ operates in ~ the spatial area SA and ~ operates during ~ the time period TP and ~ is ranked with ~ the task priority PR. conceptualise the task T ~ requires ~ the intelligence capability IC and ~ is looking for ~ the detectable thing DT and ~ operates in ~ the spatial area SA and ~ operates during ~ the time period TP and ~ is ranked with ~ the task priority PR. Direction: The search terms (topics) are derived from the “detectable” The spatial extent from the “spatial area”

32. Defining Social Media ISR tasks conceptualise the task T ~ requires ~ the intelligence capability IC and ~ is looking for ~ the detectable thing DT and ~ operates in ~ the spatial area SA and ~ operates during ~ the time period TP and ~ is ranked with ~ the task priority PR. conceptualise the task T ~ requires ~ the intelligence capability IC and ~ is looking for ~ the detectable thing DT and ~ operates in ~ the spatial area SA and ~ operates during ~ the time period TP and ~ is ranked with ~ the task priority PR. Direction: The search terms (topics) are derived from the “detectable” The spatial extent from the “spatial area” Collection: Stream-processing of tweets based on “direction” phase.

33. Defining Social Media ISR tasks conceptualise the task T ~ requires ~ the intelligence capability IC and ~ is looking for ~ the detectable thing DT and ~ operates in ~ the spatial area SA and ~ operates during ~ the time period TP and ~ is ranked with ~ the task priority PR. conceptualise the task T ~ requires ~ the intelligence capability IC and ~ is looking for ~ the detectable thing DT and ~ operates in ~ the spatial area SA and ~ operates during ~ the time period TP and ~ is ranked with ~ the task priority PR. Direction: The search terms (topics) are derived from the “detectable” The spatial extent from the “spatial area” Processing: The required “intelligence capability” determines the type of processing: “localization” – derive location data from tweets or tweeter “detection” – use existing event detection algorithms. Collection: Stream-processing of tweets based on “direction” phase.

34. Defining Social Media ISR tasks conceptualise the task T ~ requires ~ the intelligence capability IC and ~ is looking for ~ the detectable thing DT and ~ operates in ~ the spatial area SA and ~ operates during ~ the time period TP and ~ is ranked with ~ the task priority PR. conceptualise the task T ~ requires ~ the intelligence capability IC and ~ is looking for ~ the detectable thing DT and ~ operates in ~ the spatial area SA and ~ operates during ~ the time period TP and ~ is ranked with ~ the task priority PR. Direction: The search terms (topics) are derived from the “detectable” The spatial extent from the “spatial area” Processing: The required “intelligence capability” determines the type of processing: “localization” – derive location data from tweets or tweeter “detection” – use existing event detection algorithms. Collection: Stream-processing of tweets based on “direction” phase. Dissemination: Alerting (or otherwise) via contextual application such as Sentinel, or agent such as Moira.

35. Identifying “key tweeters” In practice “key tweeters” emerge: –Use spatial terms: they want people to know where they are –Use terms/hashtags: they want their tweets to be found –Social network: who are they and who they connect to From these we can determine: –Whether they are in a “position to know” –Their skills in Twitter usage –Their influence and reach All of this helps build knowledge of trust and information quality

36. Findings so far Existing ISR task representation can drive Twitter collection Human & machine agents can use this information in many ways The Moira agent helps us to interact with the knowledge base: –Engage the system in a conversation –Assert new local knowledge –Extend the model –Invoke additional functions such as “fact extraction” Use of the “stance” relationship in a conversation with Moira An example of fact extraction from tweet text using Moira

37. Wrapping up

38. Related work Conversational interaction: –Bi-directional chains for ISR pipelines –Humans and machines in collaboration Experiments with Human subjects: –Using the Moira interface –Crowd-sourced Situational Understanding –Combine Human input and physical sensors –Handling incomplete and conflicting information –Use of relevancy criteria to minimise resource utility

39. Some conclusions Streamed insight from Social Media could be incorporated into traditional ISR asset management. This could be streamlined through: –Automatic assignment of assets (for stream processing) –Automatic identification of Social Media collections Lots of issues: –e.g. misinformation and coordinated rumours Awareness improves potential for action: –Early countering strategies, opportunities for community intervention Limitations and opportunities: –We have focused on text-based analysis –Imagery potential: image processing, face detection, object recognition etc

40. Tasking the Tweeters: Obtaining Actionable Information from Human Sensors SPIE DSS 2015 – Ground/Air Multisensor Interoperability, Integration & Networking for Persistent ISR IV Any questions? dave_braines@uk.ibm.com Research was sponsored by US Army Research Laboratory and the UK Ministry of Defence and was accomplished under Agreement Number W911NF-06-3-0001. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the US Army Research Laboratory, the U.S. Government, the UK Ministry of Defense, or the UK Government. The US and UK Governments are authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation hereon. Development of the Sentinel platform was funded by the European Commission under the project “Tackling Radicalisation in Dispersed Societies (TaRDiS)”, and the ESRC via the project “After Woolwich: Social Re- actions on Social Media” (ES/L008181/1). Cardiff University provided funding for the pilot study examining community impacts of the NATO Summit. We thank Kieran Evans and David Rogers (Cardiff University) for setting up the data collection pipeline for the pilot study in Section 2 and assistance with the data analysis. We thank Darren Shaw (IBM Emerging Technology Services, UK) for creating the tweeter locator service in Section 3. Valuable insights on policing and community reaction to events such as the ones featured in our pilot study were provided by Martin Innes, Colin Roberts and Sarah Tucker (Cardiff Universities Police Science Institute, http://www.upsi.org.uk).http://www.upsi.org.uk