1. General-Purpose Learning Machine
Seyed Hamid Hamraz

2. Introduction to Machine Learning
How to construct computer programs that automatically improve with experience?
Machine learning problems are usually reduced to a few function approximation problems:

3. Introduction to Machine Learning
Variable (Inputs or output) types:
Numeric: e.g. weight, height, etc
Nominal: e.g. booleans, seasons, etc
Numeric variables may be either continuous or discrete
Output:
Numeric: Regression Problems
Nominal: Classification Problems

4. Introduction to Machine Learning
Machine Learning problem types from the perspective of the feedback provided to the learning organ:
Supervised: training is in the form of
Reinforcement: training is in the form of occasional rewards to the learning system
Unsupervised: no clear training is provided
Machine Learning problem types from the perspective of the feedback provided to the learning organ:
Supervised: training is in the form of
Reinforcement: training is in the form of occasional rewards to the learning system
Unsupervised: no clear training is provided

5. Introduction to GPLM
Most of machine learning activities are in the shape of solution-finding for a specific problem.
From the user’s view, General-Purpose Learning Machine (GPLM) is a black box which receives supervised training instances and predicts answers for unknown instances.
A comprehensive class library (maybe a hardware part) for supervised machine learning jobs that can be exploited by a simple user.

6. GPLM outer view GPLM unknown instance for prediction
set of training instances for a special problem
GPLM
output value
additional Learning

7. GPLM Essential Characteristics
Disuse of intellectual cost for adapting the machine to a problem
Fast and online learning
Fast and efficient prediction
Additional learning
No strict limitation for the type of problems it can be applied

8. Nominated Learning Methods as the GPLM Internal Engine
Decision Trees
Artificial Neural Networks (ANN)
Instance-Based Learning

9. Decision Tree Learning
Approximating nominal-valued functions (classification)
The learned function is represented by a tree (if-then rules)
Learning is to construct the tree regarding that the more strong attributes should reside in higher nodes
Information theory methods

10. Play Tennis Decision Tree

11. Is Decision Tree Learning suitable for GPLM?
Intellectual cost not required
Fast and efficient prediction (linear to the number of inputs)
Slow learning
No additional learning
Only for classification
Difficulty in dealing with real inputs

12. Artificial Neural Networks
Inspired by the complex web of interconnected neurons in the brain
Robust method for approximating both numeric- and nominal-valued functions
Feed-forward networks are mostly utilized
The BackPropagation algorithm is the most commonly used ANN Learning technique

13. ANN for Steering Autonomous Vehicle

14. Is ANN suitable for GPLM
Fast and efficient estimation
No limitation in the type of inputs or output
Requires a knowledgeable user for each problem: finding the appropriate network topology
Very slow learning
No additional learning

15. Instance-Based Learning
Does not provide explicit representation for the function which is to be approximated
Saves the instances during the training session
Retrieves a few similar instances to the unknown one, and estimates the output based on them
Lazy: does not do anything special during learning session, and postpones the process to the estimation time

16. K-Nearest Neighbors (KNN) Algorithm
More specific sub-type of Instance-Based algorithms
Instances are represented in the form of points in a multi-dimensional space
The similarity metric is the Euclidean distance

17. Is KNN suitable for GPLM?
Intellectual cost not required
Fast and online learning
Additional learning at any time
No limitation in the type of inputs or output
Slow estimation

18. KNN Fatal Issues for GPLM
Overcoming laziness
Output estimation based on retrieved neighbors
Regression
Classification
Feature (input) weighing
Nominal inputs
Null-valued inputs

19. KNN Optimization Issues for GPLM
Finding appropriate value of the K
Reducing storage requirements
Noisy training instances
Missing attributes …

20. Overcoming Laziness Indexing structures 1 B-tree, hash indexing 2-10
Dimensionality
Indexing Technique 1
B-tree, hash indexing
2-10
quad-tree, grid-file, KD-B-tree, R-tree, R*-tree
10-30
X-tree, TV-tree, M-tree, Pyramid-Techniques
> 30 ?

21. Approximate Nearest Neighbors
No feasible indexing method for high-dimensional problems
Machine learning is based on approximation
Approximate nearest neighbors retrieval can lead to acceptable learning result
Approximate nearest neighbors can be retrieved far easier than the exact ones

22. Approximating f(x,y): K=1
Implemented KDTree X Y Y Y X Y X
Approximating f(x,y): K=1

23. Discussion Adding a new instance: Finding the exact nearest neighbor:
Finding the container rectangle (approximate nearest neighbor):
Splitting methods:
Equal Distribution
Middle (recursive)

24. Output Estimation
Numeric-valued output:

25. Output Estimation
Nominal-valued output:

26. Feature Weighing
The effect degree of each attribute on the value of the output
Simple distance function
Different effect degrees, different scales

27. Feature Weighing
learning rate
number of instances

28. Feature Weighing Methods
Wrapper methods: search the domain space of the W, receive feedback from the learning tool
Filter methods: Information Theory
Filter methods can determine the W faster
Wrappers can determine a more suitable W

29. Implemented Feature Weighing Mechanism
A wrapper method which (binary) searches [0,1] for each
The search is done concurrently for all
A gradient-descent like method; the gradient is not calculated analytically
The value each should be changed at each step is estimated through a race between two values of the vector W
The race is upon cross-validation (LOOCV)
Requires standard methods to avoid local minima

30. Binary Search Effect Specifier
Holds a local copy of W, that is updated at the end of each iteration
Index in the W vector
float high = 1, low = 0;
while (high - low > precision) {
float[] W1 = (float[])localWeights.clone();
float[] W2 = (float[])localWeights.clone();
W1[id] = high;
W2[id] = low;
WeightedEuclidianDistance sm1 = new WeightedEuclidianDistance(W1);
WeightedEuclidianDistance sm2 = new WeightedEuclidianDistance(W2);
double result = match(sm1, sm2);
if (result > 0) {
weightHolder.setWeight(id, high);
low += (high - low) / division;
} else {
weightHolder.setWeight(id, low);
high -= (high - low) / division; }
updateLocalWeights();
An object that holds the values of the vector W. The object is shared among all the threads.

31. Discussion The algorithm can be realized in
Suitable for multiple parallel processor programming
Limits the weights; the weights can grow and shrink in a bounded domain

32. 9 inputs; 6-valued output
Glass Classification
9 inputs; 6-valued output
Weights {0.5, 0.77, 1.0, 0.9, 0.28, 0.4, 0.77, 0.76, 0.38}

33. 4 inputs; 3-valued output
Iris Classification
4 inputs; 3-valued output
Weights {1.0, 0.825, 0.9, 0.34}

34. Pendigits Classification
16 inputs; 10-valued output
Weights {0.64, 0.81, 0.2, 0.88, 0.58, 0.92, 0.46, 0.75, 0.67, 0.89, 0.63, 0.9, 0.49, 0.75, 0.4, 0.92}

35. Domains {(0,100), (0,100), (0,100), (0,100)}
Weights {0.0, 0.06, 0.17, 1.0}

36. Domains {(0,100), (0,100), (0,100), (0,1)}
Weights {0.0, 0.048, 0.833, 0.423}

37. Domains {(0,100), (0,100), (0,100), (0,1)}
Weights {0.06, 0.0, 0.005, 0.95}

38. Summary GPLM idea introduced: GPLM characteristics
Candidate machine learning methods as internal GPLM engine: Decision Trees, ANN, IBL
KNN issues for applying to GPLM
Efficient query (overcoming laziness)
Output estimation
Feature weighing
Empirical result presented