INSIGHT: Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data Heterogeneous Stream Processing and Crowdsourcing for Urban Traffic Management Alexander Artikis, Matthias Weidlich, Francois Schnitzler, Ioannis Boutsis, Thomas Liebig, Nico Piatkowski, Christian Bockermann, Katharina Morik, Vana Kalogeraki, Jakub Marecek, Avigdor Gal, Shie Mannor, Dermot Kinane and Dimitrios Gunopulos 1

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data  New technologies are creating a data revolution: – Sensor network deployments at large-scale. – Smart-phones as tools for data sensing, sharing and processing. – Social networks for disseminating news, advertisements and organizing social actions.  These technologies bring problems and challenges: – Heterogeneous data, different scales, noisy data, imperfect knowledge, massive data. – User centered focus, event understanding. – Specific Problem: Urban Traffic Management Big Data Challenge Heterogeneous Stream Processing and Crowdsourcing for Urban Traffic Management

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data INSIGHT  Objective 1: develop an adaptive, scalable and dependable, real-time infrastructure for improving our ability of coping with emergencies.  Objective 2: develop new methods for monitoring and analysing in real-time massive streams of heterogeneous data.  Objective 3: ensure reusability and facilitate faster adaptation of the proposed methodology. 3 Heterogeneous Stream Processing and Crowdsourcing for Urban Traffic Management

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data INSIGHT Architecture 4 Heterogeneous Stream Processing and Crowdsourcing for Urban Traffic Management

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data Urban Traffic Management 5 Heterogeneous Stream Processing and Crowdsourcing for Urban Traffic Management

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data Complex Event Processing  Data variety problem: heterogeneous event sources  Buses: position, direction, route, congestion.  SCATS sensors: traffic flow, traffic density.  Solution: complex event processing  Compute bus punctuality, bus driving quality, traffic congestion (trends).  Event Calculus for Run-Time reasoning (RTEC)  Formal, declarative semantics.  Interval-based reasoning.  Highly efficient (for event hierarchies).  Machine learning support for automated construction of complex event patterns. 6 Heterogeneous Stream Processing and Crowdsourcing for Urban Traffic Management

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data Complex Event Processing Buses reporting congestion at some location (Lon, Lat) of interest: busCongestion(Lon, Lat) initiated iff move(Bus, Lon B, Lat B, 1) happens, close(Lon B, Lat B, Lon, Lat) busCongestion(Lon, Lat) terminated iff move(Bus, Lon B, Lat B, 0) happens, close(Lon B, Lat B, Lon, Lat) 7 Heterogeneous Stream Processing and Crowdsourcing for Urban Traffic Management

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data Complex Event Processing Identifying mismatches among different streams: disagree(Bus, Lon I, Lat I, 1) happens if move(Bus, Lon B, Lat B, 1) happens, close(Lon B, Lat B, Lon I, Lat I ), not (scatsCongestion(Lon I, Lat I )=true holds) disagree(Bus, Lon I, Lat I, 0) happens if move(Bus, Lon B, Lat B, 0) happens, close(Lon B, Lat B, Lon I, Lat I ), scatsCongestion(Lon I, Lat I )=true holds 8 Heterogeneous Stream Processing and Crowdsourcing for Urban Traffic Management

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data Complex Event Processing Dealing with event source disagreement: noisy(Bus)=true initiatedAt T iff disagree(Bus, Lon I, Lat I, BusVal) happensAt T, crowd(Lon I, Lat I, CrowdVal) happensAt T', BusVal <> CrowdVal, 0 < T'-T < threshold noisy(Bus)=true terminated if agree(Bus) happens noisy(Bus)=true terminatedAt T if disagree(Bus, Lon I, Lat I, BusVal) happensAt T, crowd(Lon I, Lat I, CrowdVal) happensAt T', BusVal=CrowdVal, 0 < T'-T < threshold 9 Heterogeneous Stream Processing and Crowdsourcing for Urban Traffic Management

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data Self-Adaptive Complex Event Processing Discarding temporarily unreliable event sources: busCongestion(Lon, Lat) initiated iff move(Bus, Lon B, Lat B, 1) happens, not (noisy(Bus) holds), close(Lon B, Lat B, Lon, Lat) busCongestion(Lon, Lat) terminated iff move(Bus, Lon B, Lat B, 0) happens, not (noisy(Bus) holds), close(Lon B, Lat B, Lon, Lat) 10 Heterogeneous Stream Processing and Crowdsourcing for Urban Traffic Management

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data Complex Event Processing in Dublin 11 Heterogeneous Stream Processing and Crowdsourcing for Urban Traffic Management

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data Crowdsourcing  Data veracity problem: Inaccurate measurements, network failures, interference of mediators.  Solution: Query human volunteers (imperfect experts) close to the location of event source disagreement.  Crowdsourced information  Can also be directly sent to operators  Can also be used in the case of sensor unavailability  Model the reliability of each participant  Online Expectation-Maximisation.  Use participant reliability to improve the aggregation of answers. 12 Heterogeneous Stream Processing and Crowdsourcing for Urban Traffic Management

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data Crowdsourcing 13 Heterogeneous Stream Processing and Crowdsourcing for Urban Traffic Management

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data Crowdsourcing 14 Heterogeneous Stream Processing and Crowdsourcing for Urban Traffic Management

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data Traffic Modelling  Data sparsity problem: Several parts of the city are never/infrequently monitored.  Solution: Generalise observations of monitored locations to produce estimates for locations without sensors.  Scalability to city-sized areas is achieved by modelling the usual case.  Traffic network is represented with a Gaussian Process regression framework  SCATS intersections: observed traffic flow values.  Variables are highly correlated if they are adjacent in the traffic network. 15 Heterogeneous Stream Processing and Crowdsourcing for Urban Traffic Management

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data Traffic Modelling: Map of Dublin 16 Heterogeneous Stream Processing and Crowdsourcing for Urban Traffic Management

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data Traffic Modelling: Street Network & SCATS Locations 17 Heterogeneous Stream Processing and Crowdsourcing for Urban Traffic Management

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data Traffic Modelling: Traffic Flow Estimates 18 Heterogeneous Stream Processing and Crowdsourcing for Urban Traffic Management

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data Summary Insight solution to Urban Traffic Management:  Variety  Complex event processing.  Veracity  Crowdsourcing.  Sparsity  Traffic modelling. 19 Heterogeneous Stream Processing and Crowdsourcing for Urban Traffic Management

Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data  New technologies are creating a data revolution:  These technologies bring problems and challenges: – Heterogeneous data, different scales, noisy data, imperfect knowledge, massive data. – User centered focus, event understanding. – Focused on Urban Traffic Management Insight solution to Urban Traffic Management:  Variety: Complex event processing.  Veracity: Crowdsourcing.  Sparsity: Traffic modelling.  Volume: Stream Processing Active research on several technical fronts, Integrating solutions into one system Come to the MUD 2014 workshop Big Data Challenge Heterogeneous Stream Processing and Crowdsourcing for Urban Traffic Management