Distributed AI and Edge-oriented IoT Infrastructures for Smart Cities Prof Hock Beng Lim Centre for Smart Systems Singapore University of Technology and Design 27 Nov 2018 Read research paper for distributed ai part Total slide count should not exceed 40 Distributed AI part should not be limited to cameras, can consider other sensors. 1

Outline Introduction and Motivation Ontology for IoT Infrastructure Smart IoT Gateway Smart IoT Backend Distributed AI Concepts Use Case: Person Re-identification Implementation Scenario 2

Introduction and Motivation Many cities in the world, including Singapore, are deploying IoT infrastructures to support smart cities applications. Existing IoT infrastructures are complex, consisting of sensors and sensor nodes, gateways, cloud-based backends. The key challenge is how to enable IoT infrastructures to be smart, easy to manage, and scalable. Our approach: Virtualization-based IoT gateway to support sensor nodes with heterogeneous network protocols. Ontology-driven IoT infrastructure to ease deployment and enhance scalability. Distributed AI on edge-oriented IoT infrastructure to enable smart use cases. 3

Outline Introduction and Motivation Ontology for IoT Infrastructure Smart IoT Gateway Smart IoT Backend Distributed AI Concepts Use Case: Person Re-identification Implementation Scenario 4

Ontology Ontologies are used to capture knowledge on different domains of interest. An ontology describes the concepts in the domain and also the relationships that hold between those concepts. Some popular technologies for supporting ontology development : Web Ontology Language (OWL) from World Wide Web Consortium (W3C) Protégé from Stanford University 5

Ontology Meta-data that is organised using a domain specific convention/format 6

Outline Introduction and Motivation Ontology for IoT Infrastructure Smart IoT Gateway Smart IoT Backend Distributed AI Concepts Use Case: Person Re-identification Implementation Scenario 7

IoT Gateway Functionalities Virtualization based universal gateway to support heterogeneous protocols, e.g. Serial, Bluetooth/BLE, Zigbee, WiFi, Batman, DDS, LoRa, etc. Automated device discovery and interfacing enabled by ontology. Minimise configuration when integrating a new sensor device. Low cost, non proprietary and highly configurable. 8

Operating System Kernel Bluetooth Protocol Stack IoT Gateway Architecture Health Care Smart Home/Energy Information display Operating System Kernel SoC Gateway Controller : Bluetooth Protocol Stack Zigbee Protocol Stack Virtualization Ontology Management AirQ CSI GPIO USB 9

Ontology Mapping Library Software Architecture MQTT Publisher Virtualization Ontology Mapping Library Gateway Controller Bluetooth Zigbee Device Management User Access Dynamically updated Ontology 10

IoT Gateway Implementation Raspberry Pi 3, quad core ARMv7 Raspbian OS In-built Bluetooth Low Energy (BLE) In-built WiFi connectivity Zigbee dongle for Zigbee stack 11

Virtualization Implementation Standardized packet format and communication protocol Data upload format is generated by the relevant ontology. Each data has tagged with its ontology correspondents. Gateway virtualization performance enhancements Dedicated message queues were implemented for Read & Write separately from the gateway controller to other virtualization components, and vice versa. Well-formatted message encodings. No wait for read/write swapping, just continue asynchronous. More accurate loggers with timely status updates in process. Gateway Controller READ MQ Zigbee Virtualization Ontology based Intelligence Controller e.g: Hue WRITE MQ READ MQ Bluetooth Virtualization WRITE MQ IoT Gateway 12

Dynamic Ontology Management MQTT SERVER SUBSCRIBER BROKER BACKEND MQTT PUBLISHER HTTP Client IOT Gateway Dynamic ontology management with backend integration Gateway maintains a local ontology file. Discovery of new sensor devices will query the backend and update the system ontology depository. 13

IoT Gateway Ontology Dynamic synchronization of ontology Backend manages the ontologies for sensor devices created from specifications provided by device manufacturers. Only the relevant ontologies for the connected devices are downloaded to the gateway and stored in a local ontology file. Device discovery will find the available devices Check for matching ontologies in the local ontology file. If no match found, the ontology will be requested from the backend and updated in the Gateway ontology file (“.ont”). Gateway reads that file and generates a local ontology map. Gateway performs Load-Ontology accordingly for the sensor nodes Based on ontology map, the appropriate sensor devices are queried. 14

IoT Gateway GUI 15

Outline Introduction and Motivation Ontology for IoT Infrastructure Smart IoT Gateway Smart IoT Backend Distributed AI Concepts Use Case: Person Re-identification Implementation Scenario 16

Ontology-based IoT Backend High scalability (reuse of knowledge). Low deployment cost. Quick development & deployment. Easy integration (RESTful API). Flexibility in managing projects. Better Plug-ability. Robust big data processing (cloud computing). 17

IoT Backend Software Architecture Communication Interface (MQTT , REST API Django) Web App Framework : Django Data Exchange and Sharing Interface for external applications (REST API) Web App Framework : Django Data Management Data Repository (MongoDB) Data Analytics Ontology Modelling Ontology Management MQTT Client + HTTP Request Handler 18

IoT Backend Components WEB SERVER WEB APP MongoDB HTTP MQTT subscriber (Listener-Service) receives the IoT data and store in the relevant data-store. Web server with two major features: Ontology management. Project data management and visualization. The web app GUI uses the REST API to access the data and plot dynamic graphs. 19

Ontology Reuse and Inheritance Usage of same sensors in separate devices will be managed as inherited ontologies. Usage of the same device in multiple projects will re-use the same ontology. 20

Ontology based Data Visualization WEB SERVER Web App Sensor Data Ontology Data The ontology data is stored in a .owl file. Sensor data is stored in MongoDB. Data retrieved from the relevant sensor-data collection. Based on the unique-identifier, the respective ontology will be retrieved. This identifies the min-max limits of data based on it’s ontology. Possibility to retrieve the units of data. 21

RESTful APIs for Data Retrieval Easy retrieval of data from huge historical dataset covering various projects and their devices for: Data analytics Data visualisation Sharing of sensor data Third party software developers Improvement Able to retrieve data between a specified time interval highlight diff 22

Data Visualization 23

Outline Introduction and Motivation Ontology for IoT Infrastructure Smart IoT Gateway Smart IoT Backend Distributed AI Concepts Use Case: Person Re-identification Implementation Scenario 24

Edge-oriented IoT Infrastructure Conventional IoT infrastructures are cloud-centric, and have the following limitations: Bandwidth intensive for data transmission, in particular for video data. Latencies incurred for event processing and actuation. Advantages of edge-oriented IoT infrastructures: Edge devices are now powerful enough for processing data. Processing data close to the source of sensor data will reduce bandwidth requirement, and enable faster event processing. Distributed AI models and processing Conventional 25

Conventional IoT Infrastructure Data Flow Cloud Backend Gateway Gateway Sensor Device Sensor Device Sensor Device Sensor Device Sensor Device Gateways aggregates the data from sensor devices, and forward the data to the backend. They perform no/minimal data processing. The data analytics and decision making is done at the cloud backend. 26

Edge-oriented IoT Architecture Data Flow Cloud Backend Preliminary Processing Results Gateway Gateway Edge Sensor Device Sensor Device Sensor Device Sensor Device Sensor Device Backend offloads some of the data processing to the gateway. Gateways process the sensor data, and forward the results and relevant data to the backend. Gateways can make some decisions and perform actions based on sensor data. 27

Motivation for Distributed AI Conventional AI processing Large sophisticated machine learning / deep learning models are computationally-intensive and are often executed on cloud backend (with powerful CPUs and GPUs). Distributed AI techniques Edge nodes such as our IoT gateway are powerful enough to perform simple AI processing, e.g. Simple TensorFlow models for object detections. Develop distributed AI techniques where the processing is distributed over edge-oriented IoT infrastructures. The edge nodes will run simple machine learning models for event detections and preliminary analysis. The event detection and preliminary analysis results, together with the relevant sensor data, are sent to the cloud backend for more detailed processing. 28

Outline Introduction and Motivation Ontology for IoT Infrastructure Smart IoT Gateway Smart IoT Backend Distributed AI Concepts Use Case: Person Re-identification Implementation Scenario 29

Person Re-identification Given an image of a person-of-interest, the task is to determine whether this person has appeared in certain surveillance cameras. 30

Existing Surveillance System Big data storage Massive bandwidth Difficult to organize Searching is time consuming Video data is saved in cloud. Cloud Backend manual process to find where and when a target person appeared in video stream Gateway Gateway Video Data Flow 31

Distributed Person Re-identification Only image snapshot data is saved in cloud. Less data to store. Storing images instead of video. Save time to search for target person Cloud Backend automated process to find where and when a target person appeared in video stream by querying image snapshot gallery which returns matching image snapshots with time stamp and location Gateway Gateway Image Snapshot Transfer Video Data Flow 32

Distributed Person Re-identification Person Detection and Person Tracking: Finding the location of person in each frame of a video sequence. The results are a collection of images that each contains a person. 33

Challenges Deep learning models are large. Model Model Size (MB) Million Mult-Adds Parameters AlexNet [1] >200 720 60 VGG16 [2] >500 15300 138 GoogleNet [3] ~50 1550 6.6 Inception-v3 [4] 90-100 5000 23.2 [1] ImageNet Classification with Deep Convolutional Neural Networks [2] Very Deep Convolutional Networks for Large-Scale Image Recognition [3] Going Deeper with Convolutions [4] Rethinking the Inception Architecture for Computer Vision 34

Challenges Huge amount of calculations required for deep learning models. Common deep learning network operation Video processing: GPU: 40 frames/s Mobile/embedded: 0.04-0.4 frames/s 35

Technical Innovation Deep Compression: Reduce the storage and computation required by neural networks by an order of magnitude without affecting their accuracy by learning only the important connections. Person Detection &Tracking Calculating on Edge Nodes Deep Compression 36 Three-Step Training Strategies Pruning neurons & synapses

Technical Innovation Video footage processing now performed at the edge. Step1: Video camera records surveillance footage Step 2: Gateway performs person detecting and tracking using compressed model and generates image snapshots of detected people with time stamp and camera location Step3: Backend processes re-id query based on gallery of image snapshots compiled from all gateways Generated snapshots sent to cloud Video footage Video Camera Gateway Cloud Backend 37

Outline Introduction and Motivation Ontology for IoT Infrastructure Smart IoT Gateway Smart IoT Backend Distributed AI Concepts Use Case: Person Re-identification Implementation Scenario 38

Implementation Scenario Lamppost-as-a-Platform project recently initiated as part of the Smart Nation sensing infrastructure in Singapore. Smart lampposts to be fitted with sensors to collect data for government agencies with the goal of improving public services. Video surveillance is the key use case for public security. 39

Implementation Scenario fvfd Gateway with 3G/4G connection Camera 40

Questions ? 41