1. Distributed AI and Edge-oriented IoT Infrastructures for Smart Cities
Prof Hock Beng Lim
Centre for Smart Systems
Singapore University of Technology and Design
27 Nov 2018
Read research paper for distributed ai part
Total slide count should not exceed 40
Distributed AI part should not be limited to cameras, can consider other sensors. 1

2. Outline Introduction and Motivation Ontology for IoT Infrastructure
Smart IoT Gateway
Smart IoT Backend
Distributed AI Concepts
Use Case: Person Re-identification
Implementation Scenario 2

3. Introduction and Motivation
Many cities in the world, including Singapore, are deploying IoT infrastructures to support smart cities applications.
Existing IoT infrastructures are complex, consisting of sensors and sensor nodes, gateways, cloud-based backends.
The key challenge is how to enable IoT infrastructures to be smart, easy to manage, and scalable.
Our approach:
Virtualization-based IoT gateway to support sensor nodes with heterogeneous network protocols.
Ontology-driven IoT infrastructure to ease deployment and enhance scalability.
Distributed AI on edge-oriented IoT infrastructure to enable smart use cases. 3

4. Outline Introduction and Motivation Ontology for IoT Infrastructure
Smart IoT Gateway
Smart IoT Backend
Distributed AI Concepts
Use Case: Person Re-identification
Implementation Scenario 4

5. Ontology
Ontologies are used to capture knowledge on different domains of interest.
An ontology describes the concepts in the domain and also the relationships that hold between those concepts.
Some popular technologies for supporting ontology development :
Web Ontology Language (OWL) from World Wide Web Consortium (W3C)
Protégé from Stanford University 5

6. Ontology
Meta-data that is organised using a domain specific convention/format 6

7. Outline Introduction and Motivation Ontology for IoT Infrastructure
Smart IoT Gateway
Smart IoT Backend
Distributed AI Concepts
Use Case: Person Re-identification
Implementation Scenario 7

8. IoT Gateway Functionalities
Virtualization based universal gateway to support heterogeneous protocols, e.g. Serial, Bluetooth/BLE, Zigbee, WiFi, Batman, DDS, LoRa, etc.
Automated device discovery and interfacing enabled by ontology.
Minimise configuration when integrating a new sensor device.
Low cost, non proprietary and highly configurable. 8

9. Operating System Kernel Bluetooth Protocol Stack
IoT Gateway Architecture
Health Care
Smart Home/Energy
Information display
Operating System Kernel
SoC
Gateway Controller :
Bluetooth Protocol Stack
Zigbee Protocol Stack
Virtualization
Ontology Management
AirQ
CSI
GPIO
USB 9

10. Ontology Mapping Library
Software Architecture
MQTT Publisher
Virtualization
Ontology Mapping Library
Gateway Controller
Bluetooth
Zigbee
Device Management
User Access
Dynamically updated
Ontology 10

11. IoT Gateway Implementation
Raspberry Pi 3, quad core ARMv7
Raspbian OS
In-built Bluetooth Low Energy (BLE)
In-built WiFi connectivity
Zigbee dongle for Zigbee stack 11

12. Virtualization Implementation
Standardized packet format and communication protocol
Data upload format is generated by the relevant ontology.
Each data has tagged with its ontology correspondents.
Gateway virtualization performance enhancements
Dedicated message queues were implemented for Read & Write separately from the gateway controller to other virtualization components, and vice versa.
Well-formatted message encodings.
No wait for read/write swapping, just continue asynchronous.
More accurate loggers with timely status updates in process.
Gateway Controller
READ MQ
Zigbee Virtualization
Ontology based Intelligence Controller
e.g: Hue
WRITE MQ
READ MQ
Bluetooth
Virtualization
WRITE MQ
IoT Gateway 12

13. Dynamic Ontology Management
MQTT
SERVER
SUBSCRIBER
BROKER
BACKEND
MQTT PUBLISHER
HTTP Client
IOT Gateway
Dynamic ontology management with backend integration
Gateway maintains a local ontology file.
Discovery of new sensor devices will query the backend and update the system ontology depository. 13

14. IoT Gateway Ontology Dynamic synchronization of ontology
Backend manages the ontologies for sensor devices created from specifications provided by device manufacturers.
Only the relevant ontologies for the connected devices are downloaded to the gateway and stored in a local ontology file.
Device discovery will find the available devices
Check for matching ontologies in the local ontology file.
If no match found, the ontology will be requested from the backend and updated in the Gateway ontology file (“.ont”).
Gateway reads that file and generates a local ontology map.
Gateway performs Load-Ontology accordingly for the sensor nodes
Based on ontology map, the appropriate sensor devices are queried. 14

15. IoT Gateway GUI 15

16. Outline Introduction and Motivation Ontology for IoT Infrastructure
Smart IoT Gateway
Smart IoT Backend
Distributed AI Concepts
Use Case: Person Re-identification
Implementation Scenario 16

17. Ontology-based IoT Backend
High scalability (reuse of knowledge).
Low deployment cost.
Quick development & deployment.
Easy integration (RESTful API).
Flexibility in managing projects.
Better Plug-ability.
Robust big data processing (cloud computing). 17

18. IoT Backend Software Architecture
Communication Interface (MQTT , REST API Django)
Web App Framework : Django
Data Exchange and Sharing Interface for external applications (REST API)
Web App Framework : Django
Data Management
Data Repository (MongoDB)
Data Analytics
Ontology Modelling
Ontology Management
MQTT Client + HTTP Request Handler 18

19. IoT Backend Components
WEB SERVER
WEB APP
MongoDB
HTTP
MQTT subscriber (Listener-Service) receives the IoT data and store in the relevant data-store.
Web server with two major features:
Ontology management.
Project data management and visualization.
The web app GUI uses the REST API to access the data and plot dynamic graphs. 19

20. Ontology Reuse and Inheritance
Usage of same sensors in separate devices will be managed as inherited ontologies.
Usage of the same device in multiple projects will re-use the same ontology. 20

21. Ontology based Data Visualization
WEB SERVER
Web App
Sensor Data
Ontology Data
The ontology data is stored in a .owl file.
Sensor data is stored in MongoDB.
Data retrieved from the relevant sensor-data collection.
Based on the unique-identifier, the respective ontology will be retrieved.
This identifies the min-max limits of data based on it’s ontology.
Possibility to retrieve the units of data. 21

22. RESTful APIs for Data Retrieval
Easy retrieval of data from huge historical dataset covering various projects and their devices for:
Data analytics
Data visualisation
Sharing of sensor data
Third party software developers
Improvement
Able to retrieve data between a specified time interval
highlight diff 22

23. Data Visualization 23

24. Outline Introduction and Motivation Ontology for IoT Infrastructure
Smart IoT Gateway
Smart IoT Backend
Distributed AI Concepts
Use Case: Person Re-identification
Implementation Scenario 24

25. Edge-oriented IoT Infrastructure
Conventional IoT infrastructures are cloud-centric, and have the following limitations:
Bandwidth intensive for data transmission, in particular for video data.
Latencies incurred for event processing and actuation.
Advantages of edge-oriented IoT infrastructures:
Edge devices are now powerful enough for processing data.
Processing data close to the source of sensor data will reduce bandwidth requirement, and enable faster event processing.
Distributed AI models and processing
Conventional 25

26. Conventional IoT Infrastructure
Data Flow
Cloud Backend
Gateway
Gateway
Sensor Device
Sensor Device
Sensor Device
Sensor Device
Sensor Device
Gateways aggregates the data from sensor devices, and forward the data to the backend. They perform no/minimal data processing.
The data analytics and decision making is done at the cloud backend. 26

27. Edge-oriented IoT Architecture
Data Flow
Cloud Backend
Preliminary Processing Results
Gateway
Gateway
Edge
Sensor Device
Sensor Device
Sensor Device
Sensor Device
Sensor Device
Backend offloads some of the data processing to the gateway.
Gateways process the sensor data, and forward the results and relevant data to the backend.
Gateways can make some decisions and perform actions based on sensor data. 27

28. Motivation for Distributed AI
Conventional AI processing
Large sophisticated machine learning / deep learning models are computationally-intensive and are often executed on cloud backend (with powerful CPUs and GPUs).
Distributed AI techniques
Edge nodes such as our IoT gateway are powerful enough to perform simple AI processing, e.g. Simple TensorFlow models for object detections.
Develop distributed AI techniques where the processing is distributed over edge-oriented IoT infrastructures.
The edge nodes will run simple machine learning models for event detections and preliminary analysis.
The event detection and preliminary analysis results, together with the relevant sensor data, are sent to the cloud backend for more detailed processing. 28

29. Outline Introduction and Motivation Ontology for IoT Infrastructure
Smart IoT Gateway
Smart IoT Backend
Distributed AI Concepts
Use Case: Person Re-identification
Implementation Scenario 29

30. Person Re-identification
Given an image of a person-of-interest, the task is to determine whether this person has appeared in certain surveillance cameras. 30

31. Existing Surveillance System
Big data storage
Massive bandwidth
Difficult to organize
Searching is time consuming
Video data is saved in cloud.
Cloud Backend
manual process to find where and when a target person appeared in video stream
Gateway
Gateway
Video Data Flow 31

32. Distributed Person Re-identification
Only image snapshot data is saved in cloud.
Less data to store. Storing images instead of video.
Save time to search for target person
Cloud Backend
automated process to find where and when a target person appeared in video stream by querying image snapshot gallery which returns matching image snapshots with time stamp and location
Gateway
Gateway
Image Snapshot Transfer
Video Data Flow 32

33. Distributed Person Re-identification
Person Detection and Person Tracking: Finding the location of person in each frame of a video sequence.
The results are a collection of images that each contains a person. 33

34. Challenges Deep learning models are large. Model Model Size (MB)
Million
Mult-Adds
Parameters
AlexNet [1]
>200
720 60
VGG16 [2]
>500
15300
138
GoogleNet [3]
~50
1550
6.6
Inception-v3 [4]
90-100
5000
23.2
[1] ImageNet Classification with Deep Convolutional Neural Networks
[2] Very Deep Convolutional Networks for Large-Scale Image Recognition
[3] Going Deeper with Convolutions
[4] Rethinking the Inception Architecture for Computer Vision 34

35. Challenges
Huge amount of calculations required for deep learning models.
Common deep learning network operation
Video processing: GPU: 40 frames/s
Mobile/embedded: frames/s 35

36. Technical Innovation
Deep Compression: Reduce the storage and computation required by neural networks by an order of magnitude without affecting their accuracy by learning only the important connections.
Person Detection &Tracking
Calculating on Edge Nodes
Deep Compression 36
Three-Step Training Strategies
Pruning neurons & synapses

37. Technical Innovation
Video footage processing now performed at the edge.
Step1: Video camera records surveillance footage
Step 2: Gateway performs person detecting and tracking using compressed model and generates image snapshots of detected people with time stamp and camera location
Step3: Backend processes re-id query based on gallery of image snapshots compiled from all gateways
Generated snapshots sent to cloud
Video footage
Video Camera
Gateway
Cloud Backend 37

38. Outline Introduction and Motivation Ontology for IoT Infrastructure
Smart IoT Gateway
Smart IoT Backend
Distributed AI Concepts
Use Case: Person Re-identification
Implementation Scenario 38

39. Implementation Scenario
Lamppost-as-a-Platform project recently initiated as part of the Smart Nation sensing infrastructure in Singapore.
Smart lampposts to be fitted with sensors to collect data for government agencies with the goal of improving public services.
Video surveillance is the key use case for public security. 39

40. Implementation Scenario
fvfd
Gateway with 3G/4G connection
Camera 40

41. Questions ? 41