On randomness, confidence, and why patterns appear where nothing is happening
Some patterns feel too strong to ignore — a streak of good decisions, a run of bad luck, a sense that something is “about to happen”. Sometimes it feels like you can just tell, not with certainty, but with enough confidence that it shapes what you do next. And occasionally, you are right, which is enough to make it feel real.
But there are two separate problems hiding here, and they reinforce each other in a way that makes intuition look far more reliable than it actually is.
Randomness does not look random
If you flip a fair coin 100 times, you will not get a neat alternation of heads and tails. You will get clusters: runs of five heads, then a break, then another cluster, some stretches that look unusually one-sided and others that look strangely balanced. None of this is surprising statistically, because it is exactly what randomness produces. What would actually be surprising is the opposite — a perfectly even, well-spaced sequence would suggest structure, not randomness.
The problem is that human intuition expects randomness to look mixed. When it instead produces clusters, we read those clusters as signals: a stock chart appears to have momentum, a player seems to be on a streak, a series of events feels too consistent to be accidental. But in many cases the underlying process has not changed at all and is still random; what has changed is simply the part of the sequence you happen to be looking at.
This is where two opposite mistakes become possible, and they are worth keeping distinct because they go wrong in different ways.
One mistake is believing the streak will continue because something has shifted — the coin feels hot, the player seems to be in a different gear. This is the hot hand fallacy, and the error is inferring that the underlying process has changed when it has not.
The other mistake is believing a reversal is due because the streak has gone on too long: five heads in a row, so tails must be coming. Statisticians call this the gambler’s fallacy. But notice that this error does not require believing anything has changed — you can fully accept that the coin is fair and still fall into it. The mistake is subtler: it is treating the coin as if it has a kind of memory, as if past outcomes create an obligation to balance out. They do not, because each flip is independent and the coin has no record of what came before.
Both mistakes come from the same raw material, which is a local pattern in a random sequence, but they go wrong differently. One assumes the process shifted; the other assumes the process is self-correcting, and neither of those things is true. If each coin flip is independent, neither the previous five heads nor the previous fifty tell you anything about the next one. Randomness naturally creates local patterns, and the longer the sequence, the more convincing those patterns become and the easier it is to build a story around them.
Confidence grows faster than evidence
Suppose you start forming impressions based on these patterns — you feel that a trend is real, or that a certain outcome is more likely, and you act on it or at least file it away. Over time, two things happen that it helps to keep separate.
The first is confirmation bias. You do not remember every time your intuition was uncertain or wrong, because those moments are unremarkable and they fade. What stands out are the moments when you felt confident, acted on that feeling, and turned out to be right. Your memory is not a neutral record of your predictions but a biased sample, weighted toward the hits.
The second is overconfidence. Because you are estimating how reliable your intuition is from that biased sample, your confidence ends up mis-calibrated — you are not drawing on all the times you were wrong but on a curated highlight reel, so the confidence you feel in a given moment reflects that skewed history rather than your actual track record.
These two things feed each other: confirmation bias corrupts the data you use to assess yourself, and overconfidence is, in part, the result of reasoning from that corrupted data.
Calibration vs accuracy
Statistically, this is the difference between accuracy and calibration. Accuracy is how often your predictions are correct; calibration is whether your confidence matches how often you are correct. A well-calibrated forecaster who says “I am 70% sure” should be right about 70% of the time, and human intuition is often not calibrated this way — confidence tends to drift upward faster than accuracy does, so you might feel 80% sure in situations where you are right only 55% of the time, or feel near certainty about predictions that are barely better than random.
This is not because you are irrational but because the inputs you are using to judge your own confidence are already distorted. A biased memory produces a biased sense of how often you are right, and miscalibrated confidence is downstream of that.
Why the two problems reinforce each other
Random processes produce clusters and apparent structure. You interpret those clusters as meaningful signals and form confident beliefs based on them. Confirmation bias then means you remember the confident beliefs that happen to be correct, and overconfidence means you trust that memory more than you should. This creates a feedback loop in which randomness generates patterns, patterns generate confidence, selective memory reinforces that confidence, and reinforced confidence makes the next pattern feel even more meaningful. Over time, it begins to feel like your intuition is tracking something real, when in reality it is tracking the interaction between noise and memory.
A simple example
Imagine trying to guess whether a fair coin will land heads or tails. If you guess randomly you will be correct about 50% of the time. Now imagine you start watching short sequences — three heads in a row, then two tails, then four heads — and it becomes tempting to believe that heads are running hot, or that the coin is due for a correction. Both feel like insight, but the process has not changed and each flip is still independent, so neither holds up. What has changed is the patterns you notice, the confidence you attach to them, and which outcomes you remember afterwards. Even if your accuracy stays close to 50%, your confidence may drift much higher.
Why this matters
The same structure that appears in coin-flipping games also appears in financial decisions, hiring judgments, research intuitions, and everyday predictions about people and outcomes. Any time you are reasoning under uncertainty with limited data, these effects are present. The danger is not being wrong, because randomness guarantees that sometimes; the danger is being systematically overconfident about why you are right.
What to take away
Two things are happening at once: randomness produces patterns that look meaningful, and confirmation bias combined with overconfidence amplifies the patterns that happen to work. Together they create a convincing illusion that your intuition is more reliable than it actually is. The uncomfortable part is that this illusion does not require you to be consistently wrong — it only requires you to be occasionally right and highly confident when you are, and that turns out to be more than enough for randomness to start feeling like insight.
Why Your Gut Feeling Feels Right (Even When It Isn’t) was originally published in DataDrivenInvestor on Medium, where people are continuing the conversation by highlighting and responding to this story.
