Bayes as a rule for learning

Bayes’ Rule provides us with a mathematically sound way to update our knowledge in the light of new information.

Here’s a simple activity we do at our workshops, or indeed at conference presentations or other gatherings:

  • We chose one participant at random and ask them a simple yes-no question, such as “Have you ever been to Thailand?”
  • We tell them we’ll draw another person at random and ask the same question; what’s the probability the next person will say “Yes”? Unless they know the other participants well, they are guessing, perhaps they guess 0.6.
  • We draw 10 people at random and ask the question, keeping count of the number that answer “Yes”. Maybe only 4 out of 10 say “Yes”.
  • Now we go back to the first person and ask if they want to change their estimate.

Here we have very little prior information about participants visiting Thailand, so if the proportion answering “Yes” is different to the original guess, it makes sense to update our estimate to reflect the actual data collected.

We then repeat the same procedure but with a different question (and newly selected participants). This time the question is: “Were you born in January, February or March?” Now the best guess of the probability the next person says “Yes” is 1/4 or 0.25. And even if 4 out of 10 people say “Yes”, this is still the best estimate for the next person to be drawn.

In this case, our first estimate is based on good prior information about humans’ breeding behaviour. Births are distributed pretty evenly through the year, about a quarter of the total in each quarter of the year. Data from just 10 people drawn at random isn’t going to make us change our mind.

Combining the old and new information leads to our posterior belief. We can summarise the learning process as $$\textrm{prior information} + \textrm{new data} \to \textrm{posterior belief}$$Of course, learning is an on-going process: today’s posterior becomes tomorrow’s prior. Bayes’ Rule provides a mathematically sound way of updating our beliefs when we receive new information.

Leave a Reply

Your email address will not be published. Required fields are marked *

The maximum upload file size: 1 MB. You can upload: image, document, text, archive.