Pattern Recognition and Image Analysis Dr. Manal Helal – Fall 2014 Lecture 3

BAYES DECISION THEORY In Action 2

Recap Example

Example (cont.)

Assign colours to objects. Example (cont.) Assign colours to objects.

Example (cont.)

Example (cont.)

Example (cont.)

Assign colour to pen objects. Example (cont.) Assign colour to pen objects.

Example (cont.)

Assign colour to paper objects. Example (cont.) Assign colour to paper objects.

Example (cont.)

Example (cont.)

Bayes Discriminate Functions

Bayes Discriminate Functions Bayes Discriminate Functions (gi(x), i = 1, . . . , c), assigns feature vector x to class 1 … c 1. Minimum Error Rate Classification 2. Minimum Risk Classification Special Cases for 3) Euclidean Distance and 4) Mahalanobis Distance Discriminate Functions given last week. Other geometric functions are introduced in the following slides and many others in literature.

DISCRIMINANT FUNCTIONS 5. DECISION SURFACES If are contiguous: is the surface separating the regions. On one side is positive (+), on the other is negative (-). It is known as Decision Surface + -

If f(.) monotonic, the rule remains the same if we use: is a discriminant function In general, discriminant functions can be defined independent of the Bayesian rule. They lead to suboptimal solutions, yet if chosen appropriately, can be computationally more tractable.

Case 5: Decision Surface

6. BAYESIAN CLASSIFIER FOR NORMAL DISTRIBUTIONS Multivariate Gaussian pdf called covariance matrix

is monotonic. Define: Example:

quadrics, ellipsoids, parabolas, hyperbolas, pairs of lines. That is, is quadratic and the surfaces quadrics, ellipsoids, parabolas, hyperbolas, pairs of lines. For example:

Case 6: Hyper-planes

Case 7: Arbitrary

EXAMPLE:

EXAMPLE (cont.):

Find the discriminant function for the first class. EXAMPLE (cont.): Find the discriminant function for the first class.

Find the discriminant function for the first class. EXAMPLE (cont.): Find the discriminant function for the first class.

Similarly, find the discriminant function for the second class. EXAMPLE (cont.): Similarly, find the discriminant function for the second class.

The decision boundary: EXAMPLE (cont.): The decision boundary:

The decision boundary: EXAMPLE (cont.): The decision boundary:

EXAMPLE (cont.): >> s = 'x^2-10*x-4*x*y+8*y-1+2*log(2)'; Using MATLAB we can draw the decision boundary: (to draw the decision boundary in MATLAB) >> s = 'x^2-10*x-4*x*y+8*y-1+2*log(2)'; >> ezplot(s)

Using MATLAB we can draw the decision boundary: EXAMPLE (cont.): Using MATLAB we can draw the decision boundary:

EXAMPLE (cont.):

Voronoi Tessellation

Receiver Operating Characteristics Another measure of distance between two Gaussian distributions. found a great use in medicine, radar detection and other fields.

Receiver Operating Characteristics

Receiver Operating Characteristics

Receiver Operating Characteristics

Receiver Operating Characteristics If both diagnosis and test are positive, it is called a true positive. The probability of a TP to occur is estimated by counting the true positives in the sample and divide by the sample size. If the diagnosis is positive and the test is negative it is called a false negative (FN). False positive (FP) and true negative (TN) are defined similarly.

Receiver Operating Characteristics The values described are used to calculate different measurements of the quality of the test. The first one is sensitivity, SE, which is the probability of having a positive test among the patients who have a positive diagnosis.

Receiver Operating Characteristics Specificity, SP, is the probability of having a negative test among the patients who have a negative diagnosis.

Receiver Operating Characteristics Example:

Receiver Operating Characteristics Example (cont.):

Receiver Operating Characteristics Overlap in distributions:

BAYESIAN NETWORKS Bayes Probability Chain Rule Assume now that the conditional dependence for each xi is limited to a subset of the features appearing in each of the product terms. That is: where

For example, if ℓ=6, then we could assume: The above is a generalization of the Naïve – Bayes. For the Naïve – Bayes the assumption is: Ai = Ø, for i=1, 2, …, ℓ

A graphical way to portray conditional dependencies is given below According to this figure we have that: x6 is conditionally dependent on x4, x5. x5 on x4 x4 on x1, x2 x3 on x2 x1, x2 are conditionally independent on other variables. For this case:

A Bayesian Network is specified by: Bayesian Networks Definition: A Bayesian Network is a directed acyclic graph (DAG) where the nodes correspond to random variables. Each node is associated with a set of conditional probabilities (densities), p(xi|Ai), where xi is the variable associated with the node and Ai is the set of its parents in the graph. A Bayesian Network is specified by: The marginal probabilities of its root nodes. The conditional probabilities of the non-root nodes, given their parents, for ALL possible combinations.

The figure below is an example of a Bayesian Network corresponding to a paradigm from the medical applications field. This Bayesian network models conditional dependencies for an example concerning smokers (S), tendencies to develop cancer (C) and heart disease (H), together with variables corresponding to heart (H1, H2) and cancer (C1, C2) medical tests.

Once a DAG has been constructed, the joint probability can be obtained by multiplying the marginal (root nodes) and the conditional (non-root nodes) probabilities. Training: Once a topology is given, probabilities are estimated via the training data set. There are also methods that learn the topology. Probability Inference: This is the most common task that Bayesian networks help us to solve efficiently. Given the values of some of the variables in the graph, known as evidence, the goal is to compute the conditional probabilities for some of the other variables, given the evidence.

Example: Consider the Bayesian network of the figure: P(y1) = P(y1|x1) * P(x1) + P(y1|x0) * P(X0) P(y1) = 0.40* 0.60+ 0.30* 0.40 P(y1) = 0.24 + 0.12 = 0.36 a) If x is measured to be x=1 (x1), compute P(w=0|x=1) [P(w0|x1)]. b) If w is measured to be w=1 (w1) compute P(x=0|w=1) [ P(x0|w1)].

For a), a set of calculations are required that propagate from node x to node w. It turns out that P(w0|x1) = 0.63. For b), the propagation is reversed in direction. It turns out that P(x0|w1) = 0.4. In general, the required inference information is computed via a combined process of “message passing” among the nodes of the DAG. Complexity: For singly connected graphs, message passing algorithms amount to a complexity linear in the number of nodes.

Practical Labs On Moodle you will find two Baysian Classification examples: Image Classification Text Classification