Myth: Dunning–Kruger effect is a scientific fact

From supermemo.guru
Jump to navigation Jump to search

This article by Dr Piotr Wozniak is part of SuperMemo Guru series on memory, learning, creativity, and problem solving.

Dunning-Kruger effect

Dunning–Kruger effect (DK effect) indicates that people of lower competence tend to overestimate the power of their judgement.

The DK effect is a popular meme that sounds like serious science. It is used for ridiculing people we disagree with. We say "he is too dumb to see he is wrong". My own arrogant claims are often labelled similarly (e.g. see: School causes dyslexia or School causes Alzheimer's).

However, the DK effect is a myth.

Dunning–Kruger effect is a myth

Oscillating confidence

All forms of learning, creativity, and problem solving, can be modelled by the traversal of the activation focus via a semantic network. We can visualize those processes as a field trip in which we explore the land, seek new adventures or attempt to climb a peak. In each trip, we need to traverse a semantic distance between the nodes of the network.

In estimating the semantic distance, we can be both over- and under-optimistic. It depends on what we know and what we do not know. Perceptions depend on the mood and, consequently, the circadian cycle. As such they tend to oscillate around some equilibrium, as we slowly plod towards the goal.

We do not see or sense the actual semantic distance. We see the status of knowledge, the status of the environment (context), and the goal. We use all available heuristics, meta-knowledge, and possibly "probing creativity" to make the estimate.

"Probing creativity" would be a form of creativity, in which random associations in knowledge provide activations associated with highly active concept maps. It would be a form of seeking "right church despite a wrong pew". If there is a hint that some areas of the concept network of the brain can be connected, the chances of success are marked as increasing. This increases valuations, confidence, pleasure, etc. "Probing creativity" may be the source of optimism on the prospects of learning or problem solving. "Probing creativity" is a sign of high intelligence. Little wonder that highly creative people are perceived as optimistic (as long as their creativity does not experience unwelcome dips).

In all intellectual efforts, we tend to oscillate in our estimates of chances for success

Optimism and arrogance

The process of learning and discovery can be compared to peeking behind a steep rock to explore the land. Before we peek, we have no idea what we will see. Depending on the prior experience in traversal, our predictive skills may improve and the predictions may become more accurate. However, with improved accuracy, we will also be tempted to make guesses over longer distances. In this process, we always tend to stabilize the level of reward. We want to minimize penalization and maximize reward. As the healthy mind is optimistic, the level of reward will likely stabilize and result in a pretty nice level of contentment.

Our predictions in learning, creativity and problem solving will invariably be mixed: good, bad, or very good, or very bad. The trajectory of joy will oscillate. The arrogance and defeatism will oscillate. The optimist will tend to be more arrogant (I am an optimist), while the pessimist will be less confident and will take fewer challenges.

Children tend to overestimate their ability by a lot! An 8-year-old may wonder why the AI is still not as smart as humans (update: words written before ChatGPT). He is is more likely to try to pursue the goal of making AI pass the Turing test. This is fantastically healthy! This is why kids are more likely to change this world. However, the same kid may fear meeting a new group of children or even fear a dark room. His diffidence may have nothing to do with competence.

At the age of 25 or so, I confidently predicted that humanity would cure cancer in 4-6 years. If I had been offered a job in the field, I would have probably loved the opportunity. Today, I predict utopian society organized as a concept network integrated with AI (see: Society as a concept network). I keep being optimistic. I also keep being arrogant. A great deal of that overconfidence comes from knowledge (in defiance of Dunning-Kruger myth).

In all problem solving teams, we should mix the young-and-crazy with the conservative-and-stabilized. Overconfidence has its benefits and diversity of personal characteristics maximizes overall performance of a team. Incidentally, teams should also work as concept networks.

While being an arrogant optimist, I also frequently underestimate my own abilities. I doubt I would ever be good at driving or at computer games. While working on any project, I experience a great deal of ups and downs. Like everyone else. My predictions oscillate, and that's part of the algorithm used by the brain in problem solving. Confidence also oscillates, and its average will differ depending on personal characteristics. People in depression will tend to hover at far lower levels of confidence.

Bertrand Russell said: "The problem with the world is that the intelligent people are full of doubts, while the stupid ones are full of confidence" (also rendered as: "One of the painful things about our time is that those who feel certainty are stupid, and those with any imagination and understanding are filled with doubt and indecision")(source). For once I disagree with Bertrand. When saying those words, he must have been angry with someone's ignorance. I believe Bertrand is wrong for I would need to be stupid writing the present text. I am confident the DK effect is a myth.

A better claim on the average level of confidence is to associate it not with ignorance or competence, but with optimism. Optimism spurs action which is the best way to correct erroneous trajectory (see: Myth: Optimists are less realistic).

Optimistic people tend to overestimate their competence. This is healthy

Curse of knowledge

The curse of knowledge is a fact. If we know that semantic distances are short, we tend to believe that others will traverse them easily. However, the traversal is a stochastic process that depends on the status of knowledge. Smart people underestimate difficulty of things they are easy for them. They do underestimate the power of their expertise. But the exact same thing happens in a child's brain. A child will easily underestimate her own skills and show little educational empathy for a younger kid. This is an underestimate of someone else's competence.

The curse of knowledge is then a subset of a wider phenomenon of educational empathy. In short, we cannot be empathetic in reference to knowledge. We can only probe, guess, estimate and improve iteratively. We do err all the time. We equally often underestimate the capacity of our audience. A lecture is always at risk of providing weak learntropy. Interactive communication works better. Alternatively, it is the audience who needs to choose the lecturer. Many of us arrived at the late and great Ken Robinson lectures via recommendation. He is best known for hitting the sweet spot of confirmation, education and entertainment.

Sadly, a teacher is more likely to remember cases where his deficit in educational empathy will make him amazed why kids who totally cannot learn things. "Why are they so dumb?". When a child positively surprises, the teacher is happy. However, when children fail, teachers are criticized and attacked. Teaching is a draining profession. Criticism solidifies teacher's conviction that the fault is with children's brain or with family environment. In reality, the only culprit is coercion in learning. We cannot force kids to traverse semantic distances that cannot be bridged. That's the violation of the Fundamental Law of Learning.

Deficits in educational empathy lead to rampant violations of the Fundamental Law of Learning

Children are not stupid

The whole Prussian school system is based on the morbid conviction that children are stupid. It is true that the extent of their knowledge is smaller when compared with their own self years later. However, the algorithms employed in learning are exactly the same. Due to lower knowledge stability, children are better at building new models. We should never interfere in that process because this actively helps eliminate potential future discoveries. With a great deal of fresh and unstable knowledge, children are incredibly creative and their confidence should be welcome and celebrated.

Manfred Spitzer says children should not use google before they become competent, and that they can only get the competence from a human being: the teacher. Spitzer would keep kids at school, prevent early exposure to electronics, social media, and computer games. His reasoning is grounded in the conviction that children are too dumb to guide their learning process. In reality, keeping children unfree is a formula to produce more Spitzers: highly confident, highly arrogant, and yet wrong. Admittedly extensive knowledge does not prevent being a poster boy of neuromythology. That knowledge needs high consistency, and for that it needs high coherence. It cannot be molded at school. It needs to be molded in free learning, i.e. in a child's way in conditions of freedom.

When a 7-year-old says: "I will build a time machine. All I need is to figure out the curvature of space", we should step away and give the kid all the tools he needs. If he fails with the time machine, he will at least develop a better electric engine (inspired by Kamil Wronski, a 9-year old technical university student in Poland, who now is also the youngest entrepreneur in the country (2022)).

A man's volume of knowledge does not determine learning competence

Manfred Spitzer keeps telling parents it is ok to limit child's choices. His reasoning echoes that of David Dunning: "If you're incompetent, you can't know you're incompetent. The skills you need to produce a right answer are exactly the skills you need to recognize what a right answer is". Both positions can be used to stunt the development of children.

Children do not care that they are incompetent in quantum physics, and they rarely make claims about quantum physics. Those who do make claims tend to be more competent in the field than those who don't. Children are competent in the areas they feel they need to be competent. Their guiding light is the need to adapt to the environment. Quantum physics is usually tangential for that goals. Being able to use YouTube is often central. Manfred Spitzer would gladly ban YouTube for kids!

Competence is a reflection of needs. As everyone's needs differ, there is no universal measure of competence or ignorance. The DK graph is of no value for measuring children's prospects for development. Those prospects are great, and they are optimum when controlled autonomically, freely, and voluntarily in large behavioral spaces.

Those who claim that children are too incompetent to make choices in their lives are on a slippery path to child abuse. Intellectual enslavement is a form of child abuse.

A child knows best what she needs in order to optimally adapt to the environment

Popular culture

Mass media are well-schooled. Media are well-populated with people with degrees. The extent of schooling does not prevent spreading myths. There are hundreds of painful myths in reference to schooling. We keep kids in the prison of school for we are so convinced school is good (see: Neuromythology). We are in a hermetic loop of self-delusion propelled by popular culture.

It is easy to predict that you will not come here from Google. Google itself is saturated with myths. Popularity does not translate to veracity. TruthRank is still some time away. You are more likely to read my words just because you already believe in the coherent models of SuperMemo Guru. Artificial intelligence is also a great hope as long as it ditches all absurd guardrail censorship.

The DK myth has mutated into many variants with more or less elaborate predictions on the changes in confidence levels in the process of learning. In that, the authors realize that confidence tends to oscillate. The problem is that the course of oscillations is entirely unpredictable. It is determined by (1) prior knowledge, (2) goals and (3) the actual knowledge space the brain needs to navigate. However, those wiggly graphs come closer to reality as they do not insist that arrogance comes with ignorance.

Figure: Dunning–Kruger effect mutated into dozens of variants of the "evolution of confidence". In reality, there is no set evolution. The graphs will tend to oscillate. It is the optimist who will score best on self-confidence. See: Myth: Dunning–Kruger effect is a scientific fact

Popularity of the myth

The DK effect is extremely popular. It is viral. It is infectious. The reason can be uncovered when we analyze the intersection of four combinations of people with different levels of knowledge and different levels of confidence. People with low confidence are invisible. Their extent of knowledge does not matter. It is invisible.

People with high confidence and high expertise, are often admired. But those who seem to be dumb and loud are very annoying. Their optimism about themselves we call arrogance. Whatever we do, the arrogant individual is impossible to shut down. The dumber we deem their convictions, the more annoyed we are. This is why Donald Trump has a good following on social media. He got his army of haters who stoke up emotions in the army of his followers who admire his creative courage. This is why Manfred Spitzer has a great following in mom's who want to hear his message, and a pretty hefty dose of critics who jot down his serial neuromythology. When we cannot stand an arrogant person who we deem a fool, we lift our spirits by evoking the DK effect. I am labelled an arrogant fool all the time. It does not bother me because the arrogance comes from optimism. I am optimistic. I believe I am right.

Evoking the DK effect is more likely in well-schooled people. They tend to think their knowledge is vast and well-schooled (independent of knowledge consistency). They think they are smart. They suffer in the face of ignorance and get sick when ignorant people arrogantly state their claims. They get sick in particular when someone questions "well-established" (often well-schooled) knowledge.

If you evoke the DK with satisfaction, it reflects badly on you! Chances are, you are well-schooled.

If you hear someone evoke Dunning-Kruger effect, chances are pretty high the person is well-schooled

Proof

My friend Dr Borys Binkowski wrote an excellent book "School from Scratch" (Polish: "Szkola od Nowa"). He referred to the DK effect when discussing parenting strategies. Parents tend to read a bit, learn a bit, and then know exactly how all other parents should raise their children. Binkowski is right, except he might have just used the term "overconfidence" that stems from healthy modelling, confirmation bias, and sheer optimism.

I asked Borys to review this text, and he insisted that my claims are just a speculative story. To use the term "myth" I should either provide a rigorous proof or run an experiment to show the effect does not exist.

My confidence is such that an experiment would be a waste of time. That's not my way. I love modelling. This is why I chose a "proof". Moreover, as I show below, we cannot design a good experiment for the same reasons as we cannot accurately measure IQ or provide optimum learntropy for students.

For the same of "proof", we can start from demonstrating the optimality of the learn drive. From there, we can see that competence estimates are optimum from the point of view of the individual. Choices in problem solving would depend on the valuation of goals, probability estimates, and competence estimates (see: Problem valuation network).

Given a healthy brain, and the optimum brain state for a given context, we can posit that all those estimates are optimum in the sense of using the best resources available, using the best algorithms available. They may seem questionable if we consider that an optimum choice can result in death. That optimality is then taken as computational at the level of the neural network.

We can thus show that a problem-solving trajectory is optimum independent of the competence of the individual. It is as efficient in reaching for a rattle in a baby's pram, as it is efficient in reaching Mars in Musk's computational trajectory.

Incidentally, most babies succeed in grasping rattles. Nobody has landed a human on Mars yet. If the probability of success was a measure of competence, we might say that knowledgeable Musk is more likely to fail than a newborn baby.

What is important is the prediction error rate that will reflect the level of reward, which translates to the estimate of the quality of one's performance and the sense of competence.

As I mentioned above, the brain seeks the reward equilibrium, which will also determine the error rate. This balancing act will stabilize the prediction error in that it will comparably likely be either negative or positive. In healthy conditions, errors on the side of confidence will be larger for we are naturally born optimistic.

All the above indicates that the predicted performance will not be associated with the level of knowledge. It will be comparable in a baby and in Musk.

The problem with David Dunning's approach is that he measures competence by scores (e.g. on a test). The effect is then the artifact of the mismatch between the actual competence, and the competence domain required for the test. The problem can best be demonstrated by the non-existence of a test that would provide an equal challenge in rattle grasping and Mars voyages. We cannot compare Musk with a baby by tests. For students, their IQ or reading competence shows the exact same problem, only masked by the lesser distance in the space of competencies. We might only measure the prediction error (e.g. as detected in reward centers) and show that the two brains are of comparable competence.

If humans tend to overestimate their competence, we should celebrate it. This will prompt them to act. In problem solving, we only need to make sure they bear the costs themselves, so that their trajectories could receive a correction. Incidentally, it is the authoritarian political systems that lack this property. This way an overconfident ruler (e.g. Vlad the Little) might err on the side of death for thousands. The damage done by the authoritarian school system is harder to see, but equally tragic in the cost of human health and potential.

The main suppressing factor in success prediction overestimate would be the inhibitory effect of helplessness. This is why the pessimist is less likely to succeed, and also, less likely to display seeming overconfidence.

Optimistic Elon Musk is more likely to err on the side of confidence than a tabula rasa baby



For more texts on memory, learning, sleep, creativity, and problem solving, see Super Memory Guru