Why Untrained People Are Naturally Bad at Scientific Reasoning
Posted on January 3, 2017
When computer scientists started building systems to engage in conservation with humans, they augmented low-level computer understanding skills with a huge database of knowledge. They discovered that an ordinary child has quite literally millions of words and facts and relationships at her fingertips. That means she has been learning thousands of things per day all her life. (Actually, part of it is that she learns some higher level principles from which she can deduce lots of things, but even so.) Now how the devil can she possibly do that?
The answer is that she mostly learns heuristically rather than scientifically. And much of her learning is by generalizing from anecdotes. Since she is collecting thousands of anecdotes per day, she is capable of learning thousands of lessons. She will get lots of things wrong that way, but there will be plenty of time later to unlearn them.
That is, if she is willing to change her mind when she sees a counterexample. Unlike grown-ups, who mostly hold battle-hardened beliefs that seem to work pretty well and therefore become unchallengeable.
Anecdotal thinking is the extreme case of sample selection bias, i.e. of generalizing from an unrepresentative set of cases. But what’s worse is when we turn it around and use it as a (false) principle of confirmation. We keep coming across anecdotes that confirm our prior beliefs, which reinforces them. Of course we come across disconfirming anecdotes as well, but we dismiss them as nonrepresentative.