| Home Forums Library Media Gallery Glossary Links |
 |
May 09, 2004
Common errors in reasoning
In a comment to a previoius post, Michael Anissimov mentioned a number of pervasive errors in reasoning that are common to practically all human beings. Well, I know about that kind of thinking fallacies, but I didn't know all the "official" terms. In psychological research, a number of these fallacies have been given names, and been studied in some depth.
Situations in which people assess the frequency of a class or the probability of an event by the ease with which instances or occurrences can be brought to mind.
People inadvertently assume that readily-available instances, examples or images represent unbiased estimates of statistical probabilities.
E.g. if you've mainly been around a certain type of people, you easily get to believe they represent a typical cross-section of the population. At least, your estimates of various characteristics and beliefs will be biased towards the profile of the people you know. "To a hammer, everything is a nail".
When two events can occur separately or together, the conjunction, where they overlap, cannot be more likely than the likelihood of either of the two individual events. However, people forget this and ascribe a higher likelihood to combination events, erroneously associating quantity of events with quantity of probability.
Here's an example:
Bill is 34 years old. He is intelligent, but unimaginative, compulsive and generally lifeless. In school, he was strong in mathematics but weak in social studies and humanities.
Which statement is more probable:
A. Bill is an accountant that plays jazz for a hobby, or
B. Bill plays jazz for a hobby?
92% of people in a survey answered A. Which is completely wrong. See answer
Research has shown that people find it very difficult to decide what information is necessary in order to test the truth of an abstract logical reasoning problem. The Wason Selection Task is often used to examine this issue.
A typical experiment using the Wason Selection Task will present some rule, and ask subjects to see if the rule is being violated. Consider the rule: If a card has a D on one side, it has a 3 on the other side. Subjects are aware that on the particular set of cards, each one has a letter on one side and a number on the other side. Four cards are shown, such as those below:
Very few people can correctly pick the two cards to turn over to verify the rule. The correct cards are D and 7; most likely, you picked D and 3. Seeing what is on the reverse of the 7 card can lead to falsifying the rule if a D shows up. Seeing what is on the reverse of the 3 card cannot falsify the rule. It can confirm the rule, but not falsify it.
Various theories based on Amos Tversky's research, also related to availability bias, representativeness bias and anchoring.
Support Theory has an empirical base of results showing that different descriptions of the same event often produce different subjective probability estimates. It explains these results in terms of subjective evaluations of supporting evidence. [...]
According to the ‘framing effect’ peoples’ understanding of a problem is profoundly influenced by how the problem is presented.
For example, support for an option seems to increases the more that the option is broken down into smaller components. And naturally, if an option is particularly highlighted (anchored), people would tend to choose that over others, whether it is logical or probable or not.
Another interesting tidbit:
"This framework questioned the assumption of "homo oeconomicus", that is, of human beings motivated by self interest and capable of rational decision making behavior."
Masses of people are so easy to mislead (advertising, politics, media) that there's certainly no guarantee that they'll make rational decisions, which assumption is the basis for our economic system.
People tend to judge the probability of an event by finding a ‘comparable known’ event and assuming that the probabilities will be similar.
As a part of creating meaning from what we experience, we need to classify things. If something does not fit exactly into a known category, we will approximate with the nearest class available.
Overall, the primary fallacy is in assuming that similarity in one aspect leads to similarity in other aspects.
The gambler’s fallacy, the belief in runs of good and bad luck can be explained by the representativeness heuristic.
People will also ‘force’ statistical arrangements to represent their beliefs about them, for example a set of random numbers will be carefully mixed up so no similar numbers are near one another.
Examples:
If I meet someone with a laid back attitude and long hair, I might assume they are Californian, whereas someone who is very polite but rigid may be assumed to be English.
People will often assume that a random sequence in a lottery is more likely than a arithmetic sequence of numbers.
If I meet three people from a company and they are all aggressive, I will assume that the company has an aggressive culture and that most other people from that firm will also be aggressive.
There are a lot more theories and terms and models, of course. See, for example, this list of psychological theories, explained in simple terms.
Obviously, the human mind isn't overly suited for making logical decisions, or for correctly estimating the probability of events. It might seem a bit surprising that we even manage to keep ourselves alive and accomplish complicated technological feats. It explains at least why we often make decisions that don't serve us, and why we easily elect the wrong people to lead us. Of course it helps greatly if we can stay conscious of the various ways we are likely to fool ourselves, so we can avoid them, as much as possible, when we're trying to make important decisions.
And, of course, since we can even study it and talk about it, of course we can also be conscious of it and overcome it to some degree. Is human thinking so unreliable that we would have to develop a Friendly AI who can take over and run things for us so we don't destroy ourselves? Then how on earth can we be trusted to get that done right? It can get a bit circular. What makes the difference is our consciousness, our ability to be aware, also of our own shortcomings.
Related:
Stephen's Guide to Logical Fallacies.
More logical fallacies that are all too common:
http://www.datanation.com/fallacies/index.htm
Posted by: Philip Dhingra at May 10, 2004 09:44 AMAh, that's a brilliant list.
Posted by: Flemming Funch at May 10, 2004 11:11 AMYeah, this is a really interesting. I wasn't aware of most of these until Michael mentioned them. Since then, I've had a chance to learn more about two of these, Conjunction Fallacy and Wason Selection Task. There were online tests for both, and on Conjunction Fallacy I answered each question correctly. With Wasson Selection Task, I was confused about the questions, because it implied there was an absolutely correct answer, when it became obvious that it was about PROBABILITIES. As soon as I realized that, I was also able to answer all those questions correctly.
So, either I'm naturally immune to these specific fallacies of human reasoning, OR I have through a long life of rational training, LEARNED how to overcome some of these limitations. I think the latter is the more correct explanation.
Michael seemed to imply that it would be impossible for the human mind to overcome these limitations, but in this particular case with Conjunction Fallacy and Wason Selection I was able to. This immediately gave me some hope, as I now believe that once we become AWARE of our limitations, of our neurological programming, we can re-program our brains to overcome those limitations. This is what John Lilly's work was all about. I still love to come back to his quote, "In the Province of the Mind, there are no limits".
I look forward to getting more of Michaels input on this.
Hey Paul!
I'm really excited that you made this post on heuristics and biases. I should be more proper and say that the *inclination* to answer incorrectly on these problems, and fall subject to these biases, is a panhuman trait. Also, although you may have answered the one question correctly in the formalized problem, I'm sure you (and all of us, including me) probably don't do as well under conditions of cognitive load or when emotionally valent aspects of a given problem distract us.
It's funny, I was *just* working on an introductory paper to cognitive biases, which includes the five you have described so clearly in this post!
http://www.acceleratingfuture.com/michael/works/heuristicsandbiases.htm
It's currently in rough draft form but I hope to finish it up and link from my "writings" page soon.
I can't tell you how important I think it is that as many people as possible learn more about cognitive biases and formulate debiasing strategies for more rational (Bayes' rule-approximating) thinking. Yes, I think your performance on the Conjunction Fallacy Problem can be attributed to previous training in rational thought. Clearly I believe that humans can overcome some of our biases; but the ratio of biases we can overcome relative to the biases we *cannot* currently overcome is overwhelmingly in favor of the former.
But I must admit that there is a chain-reaction effect in the debiasing process. The more biases you start to transcend, the easier it is to detect these biases and eliminate them. That's the part of my paper I still haven't written yet - but the part I'm looking forward to the most - the "debiasing" section. I would attribute a good-sized chunk of my mental and philosophical progress in the past year to the process of formal debiasing.
You are absolutely correct that the Selection Task is about probabilities. Transferring from Aristotelian-style chunking to Bayesian probabilistic reasoning is one of the cornerstones of debiasing. However, I assure you that you are not immune to the majority of the many biases that have been academically studied (probably around 50). I'm still in the process of listing them and getting a better handle of the experimental work that has been done.
Try running your internal narrative through 20-50 distinct bebiasing filters on a continuous basis. It's not easy. In many cases there is an infinite regress of subpersonalities, each with their internal narratives, chattering about with their debiasing recommendations. This can result in cognitive overload and only a few debiasing filters can be applied quickly. But "a few" is not enough, and it can be hard/impossible to accidentally pre-design the necessary mental antiprograms ahead of time. The majority of these biases are invisible to introspection unless you know of their specific existence and something about their structure.
You are quick to assume that you are free from all the biases that have been studied academically, much less those that we have only vague inklings of. The necessary mental software is incredibly complicated and needs an incredibly high cognitive throughput to meet computational demands. Out of all the biases I can imagine transcending in theory, I have probably only escaped from 5% in practice.
Sure there are limits in the province of the mind. There are biological limits, neuronal limits, information-theoretic limits, physical limits, limits that follow from embryological constraints in brainbuilding, limits that follow from the nonforesightfulness of evolution, metabolic limits...
Anyway Paul, let me tell you again that I'm very impressed that you made this post and I strongly encourage you to continue researching heuristics and biases on the internet.
Posted by: Michael Anissimov at May 10, 2004 03:43 PMI also want to mention, incidentally, that your post has a very positive and uplifting message to it. I know Singularity activists who are extremely skeptical that Eliezer can succeed with Friendly AI alone because of the innate mental limitations of all Homo sapiens, including him. They would rather see a scientific process with more cross-checking. But making that possible in an environment of exponentially accelerating computing power requires very complicated plans that may or may not be possible. In any case, it would be nice if there were more extremely rational people about, and I'm very appreciative that you're contributing to making that happen.
Posted by: Michael Anissimov at May 10, 2004 03:56 PMHi Michael, thank you for the compliments, but they should be directed at Flemming who is the author of this post. I'm glad you like it, as Flemming is a very positive guy. I'm happy that you and he are becoming familiar with each others thinking. :)
Posted by: Paul Hughes at May 10, 2004 04:29 PMMichael,
Oh gosh no, I know I'm prone to all sorts of reasoning errors. I only wanted to point out that it was my *awareness* of these biases that helped my overcome a couple of them in this limited case. I actually took a test with over 15 questions, and got them all right. I have no doubt there are many other biases I may be proned to falling for. My point was that there is hope, that self-awareness is a powerful "undoing" process of illogical thinking. Gurdjieff taught people to cultivate an "internal observer" to continually monitor their own emotional states. From both my own work in this area, and many more who've praciticed this far longer than I, have gone a long way in overcoming many emotional limits.
As you argue, the wetware is harder to work with than hardware/software... but Flemming and I are not so sure. Whoever is right, I think it behoves us all to study all of these reasoning errors and biases as much as possible.
To fullfill the task of increasing the level of rationality in people, I think it would be fun to have an entire series of courses devoted to teaching people about these reasoning errors, and how to overcome them as much as possible. Kind of like a MENTAT School. Michael, perhaps you could become a professor at this new online school!
Posted by: Paul Hughes at May 10, 2004 04:34 PMWhoops, congratulations to Flemming in that case! I have read a little Gurdjieff and agree that his idea of "watching oneself continuously" is a good one. Such a school would be very interesting, maybe if I find someone to fund it that could get going. :)
Posted by: Michael Anissimov at May 11, 2004 04:50 AM