1. Matt, Ridwan, and Spencer
ANN Ontology
Matt, Ridwan, and Spencer

2. Background Refresher
Artificial Neural Networks achieve state of the art performance in various Machine Learning tasks
Decisions on hyperparameters to the models are often empirically motivated (and important)
Weight initialization choice is an important hyperparameter decision

3. Our Use Case
A recommendation system for weight initialization of Keras ANN models.
Keras is a python library
Our solution will be a python library relying on an ontology for its reasoning
Useful for any researchers working with Keras library.
Ontology will encode best practice information about weight initialization with provenance from relevant literature.

4. Our Use Case cont.
Our recommendation system will work by suggesting Keras weight initialization functions based on
Network Depth
Layer types
Layer activation functions
A scoring algorithm will be used to rank the ten initialization functions provided by Keras
PROV-O used for capturing provenance
Papers used as provenance include
“A weight initialization method for improving training speed in feedforward neural network”, JYF Yam, TWS Chow
“Improving neural networks by preventing co-adaptation of feature detectors”, GE Hinton, N Srivastava, A Krizhevsky
“Understanding the difficulty of training deep feedforward neural networks”, X Glorot, Y Bengio
And five more (with even more to likely be added)
Manually extracted

5. Our Ontology The ontology defines the following classes
Weight initialization function
Neural Network
Activation Function
Network Layer
The system will translate between Keras classes and Ontology Classes
Possibly use subclasses (e.g certain activation functions may be of a subclass “non-linear”).

6. Example Question
Question: What weight initialization functions work with layers whose activation function is non-linear and with networks of depth greater than 10?
Answer: He Normal, He Uniform.
To answer this question, we need
Ontology with lists of initialization functions
Class “Non-Linear Activation Function”, a subclass of “Activation Function.”
Link Keras classes to Ontology classes
Keras classes can be defined as subclasses of the ontology classes in the ontology
The system would perform a query for initialization functions which work with depth>10, and activation functions which are subclasses of “Non-Linear Activation Function.”
The scorer would rank the classes and return a list with scores for each class. Ideally the correct answers would be clearly at the top of this list.
The provenance, “Delving deep into rectifiers: Surpassing human-level performance on imagenet classification”, would also be returned, possibly with relevant text excerpts highlighted.

7. Example Question cont.

8. Questions?