1. Graph Neural Networks
Amog Kamsetty
January 30, 2019

2. Motivations

3. Traditional Deep Learning
Slide from Thomas Kipf

4. Graph Structured Data
A lot of real-world data does not live on “grids”
Knowledge Graphs
Social Networks
Citation Networks
Communication Networks
Multi-Agent Systems
Protein Interaction Networks
Node Classification, Graph classification, Clustering, Link predictions
CNNs and RNNs can’t handle graph inputs because they rely on a specific ordering
Slide from Thomas Kipf

5. Inspiration from CNNs Advantages of CNNs
Local Connections
Shared Weights
Use of multiple Layers
But, hard to define convolutional and pooling layers for non-Euclidean data

6. Deep Learning on Graphs
Need architecture for machine learning on graph structured data
Also need to take advantage of the structural relationships in graphs, similar to CNNs
Collect information via message passing and then aggregation
“Graph neural networks (GNNs) are connectionist models that capture the dependence of graphs via message passing between the nodes of graphs.”
Input is a graph, and output is a graph
GNNs keep track of a state unlike standard neural network
Image from Thomas Kipf

7. Types of Graph Networks

8. The original Graph Neural Network (GNN)
Each node is defined by its own features and those of its neighbors
Learn some state embedding for each node
Scarselli et al., 2009

9. The original Graph Neural Network (GNN)
Update equation if parametrized by MLP

10. Inductive Capability
Slide from Stanford Snap Lab

11. Limitations of GNN
Very computationally intensive to recursively compute fixed point solution
Same parameters are used in every timestep, while other neural networks use different parameters in each layer
Informative edge features are difficult to model
Can’t alter message propagation depending on edge type
These pitfalls led to many variations of GNNs, particularly in how the propagation occurs

12. Graph Convolutional Networks
Kipf & Welling, 2017
Slide from Thomas Kipf

13. Graph Convolutional Networks
Slide from Stanford Snap Lab

14. Graph Convolutional Networks
Semi-supervised learning on Zachary’s Karate Club Network
Only one node from each class is labeled
300 iterations
Video from Thomas Kipf

15. Graph Convolutional Networks

16. Gated Graph Neural Networks
Uses LSTM for aggregation
Allows for deep graph networks without worrying about overfitting or vanishing/exploding gradient
Li et al., 2016
Slide from Stanford Snap Lab

17. Many more variants...perhaps too many

18. Generalizations Message Passing Neural Networks (MPNN)
Unifies GNN and GCN
Message Passing Phase that constructs message based on local neighborhood
Update function which interprets message and updates node’s hidden state
Readout phase computes feature vector for the whole graph
All of these functions can have different settings
Non-local Neural Networks (NLNN)
Unification of self-attention style methods
Node update based on neighbor nodes, not edges

19. Graph Nets

20. Relational Inductive Biases
Battaglia et al., 2018
Combinatorial Generalization
New inferences, predictions, and behaviors from known building blocks
Relies on humans’ mechanism for representing structure and reasoning about relations
But, modern deep Learning relies on end-to-end design and does away with any explicit structure
Instead, advocates for use end-to-end and hand-engineering jointly
Combination of “nature” and “nurture”
Biases/constraints on structure and relations is necessary
Structured representations have entities and relations (aka graphs) at their core
Thus, argues for graph networks as foundational building block for AI

21. Relational Inductive Biases

22. Relational Inductive Biases
Makes use of sharing and locality
Authors say there

23. Graph Networks
While RNNs and CNNs use relational inductive biases, they do not generalize to arbitrary relations
Need a framework that can learn from graphs
Proposes graph networks, which generalizes MPNN, NLNN, and other variants
Graph is defined as a 3-tuple
Where u is global attribute
V is set of vertices with attributes
E is set of edges with attributes

24. Graph Networks
Attributes are updated in a sequence of updates and aggregations, as we’ve seen before
Update functions can be parametrized with neural networks
Same 6 functions are used for all nodes and edges

25. Graph Networks

26. Graph Networks 3 main properties 3 design principles
Nodes and edges provide a strong relational bias
Entities and relations are represented as sets and are thus order invariant
Per-edge and per-node functions are shared across the entire network
3 design principles
Flexible representations
Configurable within block structure
Composable multi-block architectures

27. Flexible Representations

28. Composable Multi-block Architectures

29. Impact and Challenges

30. Impact Graph networks have applications in many areas
Works well for any data that can be represented as graphs
Node/Graph classification
Link prediction
Clustering
Use case in biology, chemistry, physical systems
Can be used for non-structured data but difficult to generate graphs from raw data
Whether the idea of relational inductive biases will be adopted remains to be seen

31. Non-structural scenarios Scalability
Challenges
Shallow Structure
Dynamic Graphs
Non-structural scenarios
Scalability

32. References
“Graph Neural Networks: A Review of Methods and Applications” Zhou et al. 2019
“Gated Graph Sequence Neural Networks” Li et al. 2017
“The Graph Neural Network Model” Scarselli et al. 2009
“Relational inductive biases, deep learning ,and graph networks” Battaglia et al. 2018
The morning paper blog, Adrian Coyler
Structured Deep Models: Deep Learning on Graphs and Beyond, talk by Thomas Kipf
“Convolutional Networks on Graphs for Learning Molecular Fingerprints” Duvenaud et al. 2015
“Semi-Supervised Classification with Graph Convolutional Networks” Kipf & Welling 2017
“Graph Convolutional Networks”, Kipf 2016
“Representation Learning on Graphs: Methods and Applications” Hamilton et al. 2017
“Geometric Deep Learning: Going Beyond Euclidean Data” Bronstein et al. 2017
“Gated Graph Sequence Neural Networks” Li et al., 2016