1. Introduction to the Basic Principles of Machine Learning
U. Haifa
Department of Information and Knowledgement,
Faculty of Management, U. Haifa
Lecturers: Prof. Larry Manevitz, Alex Frid

2. Course Requirements
This course is scheduled to meet for 8 weeks on Tuesdays. It will be team taught by Larry Manevitz and Alex Frid. Full attendance is expected. The syllabus is on your web site at the university. Grading will be by two projects; which will implement some of the ideas discussed in the course. In addition there may be an oral exam and a “defense” of the projects. Probably about 30-40% project 1; about 40-70% project 2. Lets see how you do on project 1 before we fix this in stone. Manevitz: Office 516 Jacobs. Tuesday after the course 7 – 8 PM. Frid: Office 667 Education. Monday by appointment.

3. Programming versus Learning
How do we get a computer to do anything?
What is an algorithm?
What is an efficient algorithm?
What is a heuristic?
What is an adaptive or learning methodology?

4. Programming versus Learning
How do we get a computer to do anything?
Problem
Representation of Problem
Algorithm Solution
Implementation
Testing

5. Programming versus Learning
Problem
Representation of Problem
Get Data Base
Learning Algorithm

6. Some Examples of Recent Successes where Machine Learning was Crucial
Go (as opposed to chess …)
In 1997 Deep Blue finally produced a chess program that defeated the world chess champion Gary Kasparov. This program was based on old principles of AI using search techniques and heuristic evaAluation functions. The breakthrough was more in hardware and so on than in novel computational ideas.
Go became a challenge because it did not seem attackable to the same extent by these methods and much pessimism in the world as to success in the next decade.
Nonetheless, in 2014 AlphaGo succeeded in beating a 9-Dan (like a grandmaster in chess) who was European champion; in 2016 defeated a famous Chinese champion and there may be a tournament this spring for world championship.
What was the difference?

7. AlphaGo
alphago Nature film 2016

8. Self Driving Cars and MobilEye
Movie of MobilEye Self Driving Car

9. Reading the Mind

10. Challenge: Given an fMRI
Can we learn to recognize from the MRI data, the cognitive task being performed?
Automatically?
WHAT ARE THEY?
Omer Boehm
Thinking Thoughts
Tainan 2012
L. Manevitz

11. Personalized Advertising

12. Automated Learning Techniques
Perceptron: an algorithm that develops a classification based on examples and counter-examples.
Non-linearly separable techniques (neural networks, support vector machines).

13. Learning in Neural Networks
Perceptrons

14. Natural versus Artificial Neuron
Natural Neuron McCullough Pitts Neuron

15. One Neuron McCullough-Pitts
This is very complicated. But abstracting the details,we have S w1 w2 wn x1 x2 xn
Threshold
Integrate
Integrate-and-fire Neuron

16. Perceptron
weights A
Pattern Identification
(Note: Neuron is trained)

17. Three Main Issues
Representability
Learnability
Generalizability

18. Programming: Just find the weights!
AUTOMATIC PROGRAMMING (or learning)
One Neuron: Perceptron or Adaline
Multi-Level: Gradient Descent on Continuous Neuron (Sigmoid instead of step function).

19. One Neuron (Perceptron)
What can be represented by one neuron?
Is there an automatic way to learn a function by examples?

20. Feed Forward Network
weights
weights A

21. Representability
What functions can be represented by a network of McCullough-Pitts neurons?
Theorem: Every logic function of an arbitrary number of variables can be represented by a three level network of neurons.

22. Proof Show simple functions: and, or, not, implies
Recall representability of logic functions by DNF form.

23. Perceptron What is representable? Linearly Separable Sets.
Example: AND, OR function
Not representable: XOR
High Dimensions: How to tell?
Question: Convex? Connected?

24. AND

25. OR

26. XOR

27. Convexity: Representable by simple extension of perceptron
Clue: A body is convex if whenever you have two points inside; any third point between them is inside.
So just take perceptron where you have an input for each triple of points

28. Connectedness: Not Representable

29. Representability Perceptron: Only Linearly Separable
AND versus XOR
Convex versus Connected
Many linked neurons: universal
Proof: Show And, Or , Not, Representable
Then apply DNF representation theorem

30. Learnability Perceptron Convergence Theorem:
If representable, then perceptron algorithm converges
Proof (from slides)
Multi-Neurons Networks: Good heuristic learning techniques

31. Generalizability
Typically train a perceptron on a sample set of examples and counter-examples
Use it on general class
Training can be slow; but execution is fast.
Main question: How does training on training set carry over to general class? (Not simple)

32. Perceptron Convergence Theorem
If there exists a perceptron then the perceptron learning algorithm will find it in finite time.
That is IF there is a set of weights and threshold which correctly classifies a class of examples and counter-examples then one such set of weights can be found by the algorithm.

33. Perceptron Training Rule
Loop: Take an positive example or negative example. Apply to network.
If correct answer, Go to loop.
If incorrect, Go to FIX.
FIX: Adjust network weights by input example
If positive example Wnew = Wold + X; increase threshold
If negative example Wnew = Wold X; decrease threshold
Go to Loop.

34. Programming versus Learning
What is an algorithm?
What is an efficient algorithm?
What is a heuristic?
What is an adaptive or learning methodology?

35. Some Examples of Recent Successes where Machine Learning was Crucial
Go (as opposed to chess …)
In 1997 Deep Blue finally produced a chess program that defeated the world chess champion Gary Kasparov. This program was based on old principles of AI using search techniques and heuristic evaAluation functions. The breakthrough was more in hardware and so on than in novel computational ideas.
Go became a challenge because it did not seem attackable to the same extent by these methods and much pessimism in the world as to success in the next decade.
Nonetheless, in 2014 AlphaGo succeeded in beating a 9-Dan (like a grandmaster in chess) who was European champion; in 2016 defeated a famous Chinese champion and there may be a tournament this spring for world championship.
What was the difference?

36. AlphaGo
alphago Nature film 2016

37. Self Driving Cars and MobilEye
Movie of MobilEye Self Driving Car

38. Offi

39. Perceptron Conv Theorem (again)
Preliminary: Note we can simplify proof without loss of generality
use only positive examples (replace example X by –X)
assume threshold is 0 (go up in dimension by
encoding X by (X, 1).

40. Perceptron Training Rule (simplified)
Loop: Take a positive example. Apply to network.
If correct answer, Go to loop.
If incorrect, Go to FIX.
FIX: Adjust network weights by input example
If positive example Wnew = Wold + X
Go to Loop.

41. Proof of Conv Theorem Note: 1. By hypothesis, there is a e >0
such that V*X >e for all x in F
1. Can eliminate threshold
(add additional dimension to input) W(x,y,z) > threshold if and only if
W* (x,y,z,1) > 0
2. Can assume all examples are positive ones
(Replace negative examples
by their negated vectors)
W(x,y,z) <0 if and only if
W(-x,-y,-z) > 0.

42. Perceptron Conv. Thm.(ready for proof)
Let F be a set of unit length vectors. If there is a (unit) vector V* and a value e>0 such that V*X > e for all X in F then the perceptron program goes to FIX only a finite number of times (regardless of the order of choice of vectors X).
Note: If F is finite set, then automatically there is such an e.

43. Proof (cont). Consider quotient V*W/|V*||W|.
(note: this is cosine between V* and W.)
Recall V* is unit vector .
= V*W*/|W|
Quotient <= 1.

44. Proof(cont) Consider the numerator
Now each time FIX is visited W changes via ADD.
V* W(n+1) = V*(W(n) + X)
= V* W(n) + V*X
> V* W(n) + e
Hence after n iterations:
V* W(n) > n e (*)

45. Proof (cont) Now consider denominator: |W(n+1)|2 = W(n+1)W(n+1) =
( W(n) + X)(W(n) + X) =
|W(n)|**2 + 2W(n)X (recall |X| = 1)
< |W(n)|** (in Fix because W(n)X < 0)
So after n times
|W(n+1)|2 < n (**)

46. Proof (cont) Putting (*) and (**) together: Quotient = V*W/|W|
> ne/ sqrt(n) = sqrt(n) e.
Since Quotient <=1 this means
n < 1/e2.
This means we enter FIX a bounded number of times.
Q.E.D.

47. Geometric Proof
See hand slides.

48. Additional Facts
Note: If X’s presented in systematic way, then solution W always found.
Note: Not necessarily same as V*
Note: If F not finite, may not obtain solution in finite time
Can modify algorithm in minor ways and stays valid (e.g. not unit but bounded examples); changes in W(n).

49. Percentage of Boolean Functions Representable by a Perceptron
Input Perceptrons Functions
, ,536
, **9
6 15,028, **19
7 8,378,070, **38
8 17,561,539,552, **77

50. What wont work?
Example: Connectedness with bounded diameter perceptron.
Compare with Convex with
(use sensors of order three).

51. What wont work?
Try XOR.

52. What about non-linear separable problems?
Find “near separable solutions”
Use transformation of data to space where they are separable (SVM approach)
Use multi-level neurons

53. Multi-Level Neurons
Difficulty to find global learning algorithm like perceptron
But …
It turns out that methods related to gradient descent on multi-parameter weights often give good results. This is what you see commercially now.

54. Applications Detectors (e. g. medical monitors)
Noise filters (e.g. hearing aids)
Future Predictors (e.g. stock markets; also adaptive pde solvers)
Learn to steer a car!
Many, many others …