Machine Learning 101 – Application of Bayes’ Theorem

In the previous lesson (Lesson 9), we derived Bayes theorem. So let’s write it out:

Also recall that Bayes’ theorem helps us find conditional probabilities given marginal probability.

Lets’ now apply Bayes’ theorem in the example of red and blue boxes. From the example we know the marginal probabilities as:

P(r) = 4/10  (marginal probability that the red box is picked)

P(b) = 6/10 (marginal probability that the blue box is picked)

Also not that P(r) + P(b) = 1

 

Now assuming that a box is selected randomly and it turns out to be the blue one. Then the probability of selecting an apple is simple the fraction of apples in the blue box. And that is 3/4

In terms of conditional probability, we can write it as:

P(F = a | B = r) = 1/4

That is the conditional probability of picking an apple given that the blue box is selected.

Similarly, the conditional probability of picking an orange given that the blue box is selected would be:

P(F = o | B = b) = 1/4

Therefore, we can write out all the remaining two conditional probabilities as:

P(F = a | B = r)   =   3/4

P(F = o | B = r)    = 1/4

Also:

P(F = a | B = b)   =   3/4

P(F = o | B = b)    = 1/4

What is the overall probability(marginal probability) of choosing an apple, P(F=a)?

We can then solve this using sum rule and product rule

P(F = a) = P(F = a, B = b)  +  P(F = a, B = r)

P(F = a, B = b) = P(F=a | B = b) . P(B = b)     =     3/4 x 6/10    =     9/20

P(F=a, B = r ) = P(F = a | B = r) . P(B = r)      =     3/4 x 4/10    =      3/10

Therefore:

P(F = a)     =    9/20 + 3/10    =    15/20 = 3/4

In the same way,

P(F = o) = 1 – P(F = a)     =     1 – 3/4     =     1/4

 

What if a fruit is picked and it turns out to be an orange, what is the probability that it came from the blue box?

In this case, we can apply Bayes’ theorem. So we want to find the conditional probability that the box is blue, given that the fruit is an orange. We can write it as:

And we know that:

P(F = o | B = b)  = 1/4

P(B = b) = 6/10

P(F = o) = 9/20

So plugging these into the equation, we have

P(B = b | F = o)   =    1/3 (you can work it out to verify)

 

Exercises

1. A fruit is picked from a box and it turns out to be an apple. What is the probability that it came from the red box

2. A fruit was picked from a box and it turns out to be an orange. What is the probability that it came from the red box.

 

Prior and Posterior Probabilities

If we know the probability of selecting a box, say P(B), then we can find the probability of selecting a fruit using this information. This prior information we have is called prior probability. This is because it is the probability available to us before we observe the identity of the fruit.

However, if we know the identity of the fruit first, say the fruit is an orange, then we can used Bayes’ theorem to find the conditional probability P(B | F) of which box it was taken from. This probability is called posterior probability. This is because we obtain it after we have observed the identity of the fruit F.

 

About Independence

Two events are independent if their joint probability is a product of the marginal probabilities of both events. This can be written as:

P(X, Y) = P(X)P(Y)

In this case, the product rule then becomes:

P(Y|X) = P(Y)

This then means that the conditional distribution of Y given X is independent of the value of X