A professional basketball player retires having shot something like ten thousand free throws. Each attempt is the same fifteen feet, the same rim, the same ball, and the result is unambiguous. Their career percentage is known to a decimal place. If they shot 90 percent and a teammate shot 75, nobody doubts that the difference is skill.
A partner in a venture capital firm leads perhaps thirty investments over a career, each unlike the others. The returns are drawn from a distribution so skewed that one or two investments typically account for most of a fund's gains, and the average time from the decision to a resolved outcome is more than seven years. If their investments returned 15 percent a year and a colleague's returned 8, no reasonable observer can say from the outcomes alone that they are the better investor.
A founder's record is usually one number: what the company sold for, or that it did not sell at all. That number contains the state of the market in the year of the exit, the competitors who did or did not appear, a decade of economic ups and downs, and a few hundred decisions the founder made. If the company sold for a billion dollars, nothing in the figure says what share of it was the founder.
Chapter 1 named three sources of a record: ability, circumstance and chance. When does a record tell us anything about the first? The inference can fail in two distinct ways. The first is statistical: the record may contain too little signal, relative to its noise, to support any conclusion about the person. The second is narrative: the explanation we attach to a record may be shaped by the record itself, so that what looks like independent evidence of ability is just the result restated in flattering language. The two failures have different remedies; for the first, I draw on work I did with my colleague Linda Zhao.
Two Dials
How much a record can reveal about ability depends on two quantities. The first is how noisy each outcome is: how much a single result varies, for a given level of true ability, because of everything that is not ability. The second is how many independent outcomes the record contains. A free-throw shooter sits at one extreme on both. Each attempt is nearly noise-free, and there are thousands of them. The founder sits at the other extreme, with a single outcome drawn from a distribution so wide that the same founder, replayed in a different year, might have produced almost any result.
The arithmetic is short. Let's suppose each actor has a persistent ability that does not change over the period we observe, and that each outcome equals that ability plus noise. Take the spread of ability across actors and divide it by the spread of the noise in a single outcome; call the result the signal-to-noise ratio, SNR. Averaging N independent outcomes shrinks the noise by the square root of N. So the ratio of typical differences in ability to the noise left in an actor's average record is SNR times the square root of N. The statistical half of this chapter follows from that product, which I call the detection budget. The budget rises in proportion to SNR but only with the square root of N. Doubling the signal-to-noise ratio of each observation doubles the budget. Doubling the length of a career multiplies it by only the square root of two.
How big does the budget have to be? To declare with ordinary statistical confidence that an actor is better than average by some amount d, measured in units of the ability spread, the budget must exceed roughly 2.5 divided by d. The 2.5 is not my choice; it is the sum of the two numbers a statistician uses to detect an effect with 95 percent confidence and an 80 percent chance of finding it when it is there. (Anyone who prefers a stricter standard can raise it.) For a medium-sized difference, d of about one half, that means a budget of five. For a large difference, d of about 0.8, it means a budget of about three. Below those levels the record does not support the claim. The actor may well have the ability; the trouble is that an average actor with ordinary luck could have produced the same record.
Consider first a shooter with five thousand career free throws. The spread of true shooting ability across players is about 0.10 and the noise in a single attempt is about 0.43, so the signal-to-noise ratio is about 0.23; five thousand attempts give a square root of about 71, and the budget is about 16, well past the threshold of five. For a chief executive, the measured spread of true effect on annual firm returns is about 0.04 against year-to-year noise of about 0.18, a ratio of about 0.22; a tenure yields perhaps seven independent strategic decisions, with a square root of about 2.6, so the budget is about 0.6. Because the two ratios are nearly the same, the difference between a knowable domain and an unknowable one is almost entirely the number of outcomes a career can produce.
The first of two refinements is that detection and prediction are different questions. A record too weak to establish that an actor is good still shifts a forecast of their next result, and the shift has an exact size. A rational forecaster should keep a share of their observed deviation from the average equal to the budget squared, divided by one plus the budget squared. A budget of one keeps half; a budget of two keeps 80 percent; a free-throw shooter's budget of about sixteen keeps more than 99 percent. Nothing discontinuous happens at any threshold, so when I call a record "unknowable" in what follows, I mean that it barely moves a sensible starting estimate, not that it moves it by nothing.
The second refinement is that outcomes are rarely independent. A venture partner's thirty investments share the years in which they were made and the sectors they were in; a chief executive's forty quarters share one strategy and one economy. If any two outcomes have a correlation of rho, the usual correction says that N such outcomes are worth only N divided by one plus (N minus one) times rho independent ones, a number called the effective N. The formula has a ceiling: as N grows, the effective number of outcomes approaches one over rho. At a correlation of 0.1, no career, however long, yields more than ten independent observations.
The Map
Linda Zhao and I placed ten domains on a map whose axes are the signal-to-noise ratio and the effective number of outcomes a career can produce. We used published estimates of how much of the variation in outcomes comes from ability where they exist, and stated ranges where they do not. Three domains anchor the knowable corner: career free throws, with a budget around sixteen; tournament chess, where a serious player accumulates thousands of rated games, the Elo rating system is built to measure skill from those results, and the budget is above thirty; and high-frequency trading, where the effective N runs to millions and the budget is in the hundreds. Geopolitical forecasting sits on the boundary, and only because of the forecasting tournaments that Philip Tetlock's group ran. They imposed a structure (hundreds of resolvable questions scored against the truth) that pundits in their natural state never face.
The three domains in which the largest sums of money and the most prestige are allocated on the strength of records all sit in the unknowable region. The first is long-only active equity management, where per-year noise is far larger than the spread of true ability across managers, and a career contains perhaps thirty annual returns, giving a budget near two before correlation is counted and near one after. The second is venture capital, with a budget around 2.4 at twenty investments; the heavy right tail makes even that optimistic. The third is the non-founder chief executive of a large public company. A tenure of five to ten years contains on the order of five to fifteen independent strategic decisions. The measured spread in executives' effect on firm performance is small relative to the year-to-year noise in benchmark-adjusted returns, and the budget is around 0.6. Even an executive whose true effect is a full standard deviation and a half above the median cannot be reliably distinguished, from the record, from an ordinary executive who happened to run a firm in a good industry in a good decade. The economists Marianne Bertrand and Sendhil Mullainathan showed in 2001 that chief executives are paid for exactly such industry-wide luck.
The founder is the limiting case: with one outcome per actor, the variance observed across founders is the sum of the variance of ability and the variance of noise, and no data the domain generates can split the sum. So the record cannot pick out a skilled founder, or even say how much of the spread among founders is skill. In a study of venture-backed founders, Paul Gompers and his colleagues found that founders with a prior success were more likely to succeed again, roughly 30 percent against 18 percent for first-timers. That tells us something about serial founders as a group and nothing about which individual's success was skill. Chapter 4 returns to that finding, since skill is not the only mechanism behind it.
But surely the top of the distribution is different? The median fund manager may be indistinguishable from an index, but the manager who beat the market for fifteen years must be good. Two results say the opposite: the top is where inference from outcomes is least trustworthy.
The first is selection: the budgets on the map are computed for a single actor named in advance. A manager who comes to an evaluator's attention because of an extreme record has been selected from a population of thousands, and the probability that the best record among thousands was produced by luck is not small. The finance economists Eugene Fama and Kenneth French applied this logic to the universe of US equity mutual funds. They found that the spread of returns across funds looked very much like what chance alone would produce, with evidence of skill sufficient to cover costs confined to a thin tail. Laurent Barras and his colleagues, using methods for controlling false discoveries, reached a similar conclusion. The record that got the manager into the room is the record you must discount most.
The second result is stranger. Jerker Denrell and Chengwei Liu showed that in domains where outcomes are noisy and heavy-tailed (a few extreme results account for much of the total), the most extreme performances are disproportionately produced by high-variance, unreliable processes rather than by high ability. A process that produces a consistent result rarely produces the very best result in any given year. A process that swings wildly does so more often and also produces the very worst. In such domains the top of the distribution should be actively discounted, not merely treated as uninformative. The hedge fund that returned 60 percent last year is more likely to be the one that will lose 40 this year than the one with the best process.
The next chapter develops a further complication: even where ability is real and detectable, the apparent effect of an exceptional actor includes the advantages that their prior outcomes generated. Morten Sorensen, using a structural model of how venture capital firms and startups match, estimated that about two thirds of the persistence in top-quartile returns reflects the selection of better deals by better-known firms rather than better decisions once the deals are in hand. The better deals come because of the earlier results, and the earlier results may or may not have been skill.
The Halo
The second way the inference fails has nothing to do with sample size. It would happen with perfect measurement and infinite data, because the problem is in the explanations rather than in the outcomes.
Phil Rosenzweig made the case in The Halo Effect in 2007. His central observation is that our descriptions of a company's leadership, culture, strategy and customer focus are not independent of its results. When results are good, observers describe the culture as cohesive and the leader as visionary. When the same company's results turn, the same culture becomes insular and the same leader arrogant. Rosenzweig's examples include companies whose practices were praised in the business press in one year and blamed the next, with nothing about the practices having changed. The problem is not that the descriptions are false but that they are not evidence. A variable that was measured by looking at the outcome cannot explain the outcome. Yet most of what passes for research on why companies succeed, including surveys of managers who already know how their firm is doing, measures its explanatory variables that way.
Of the eight further delusions Rosenzweig names, three bear directly on this book. Connecting the winning dots: selecting a sample of successful companies, finding what they have in common, and calling it a cause, without asking how many failed companies had the same traits. The delusion of lasting success: treating a decade of outperformance as proof of a formula, when the base rate of regression is high. And the delusion of absolute performance: judging a company by its results rather than by its results relative to what rivals did in the same conditions. (A company can do everything right and lose to a rival who did it better.)
Rosenzweig's principal example is uncomfortable for me. It is Good to Great, by the business writer Jim Collins, whose best-known image is the flywheel; chapter 5 of this book is built around flywheels. Collins selected eleven companies whose stock had beaten the market for fifteen years after a transition, compared them with companies that had not made the leap, and reported the traits the eleven shared. Two of the eleven, Fannie Mae and Circuit City, failed within a decade of the book's publication. Collins's method could not have caught that. He identified the traits by looking at companies already known to have succeeded and chose the comparison firms to be similar in everything except the result. Whatever the eleven shared, his sample cannot say whether it caused the results, how many firms with the same traits failed, or whether the traits were themselves descriptions of success.
Strategy scholars raised the same objection against the resource-based view six years before Rosenzweig raised it against the business press. Richard Priem and John Butler argued in 2001 that if a valuable resource is defined as one that improves performance, then the claim that valuable resources improve performance is empty. In practice, they argued, the resources of successful firms get identified by their success. The alpha asset worksheet in the appendix is exposed to exactly this. Fill it in for a company you already know has done well, and every resource looks rare and necessary.
Three habits protect the worksheet and this book from the halo. State the mechanism before the result: say how the resource lowers cost or raises willingness to pay, in terms that could be checked without knowing whether the firm succeeded. Score rarity by what a rival would have to spend to get the resource. And look for the comparison the halo hides: the firm with the same resource that did not succeed, or the firm without it that did. The same habits apply to every story in this book, including the ones about my own companies. Where I cannot name the mechanism, point to an observation that does not depend on the result, and say what else might explain the outcome, the story is decoration.
None of this means that management has no effect. Two findings are the kind of evidence the halo cannot contaminate. When Nicholas Bloom and his colleagues randomly assigned management consulting to Indian textile plants, the treated plants became about a sixth more productive in the first year. The plants were chosen at random, so nobody's view of which plants would succeed could color the comparison. When Morten Bennedsen and his colleagues studied what happened to firms whose chief executive was hospitalized, the firms' profitability and investment moved. So the person at the top has an effect, though the study says nothing about which executives are better. Management makes a difference; which manager made the difference, in any particular case, is the question the record usually cannot answer.
Researchers fall for a welcome result too. In late 2024 a working paper from a doctoral student at MIT reported a study of artificial intelligence in a materials-science laboratory. The tool nearly doubled the output of the best researchers while the weakest third gained almost nothing, a result that supported, precisely, the thesis that AI rewards judgment at the top. It was widely circulated and admired until May 16, 2025, when MIT's economics department announced that it had no confidence in the provenance, reliability or validity of the data, or in the veracity of the research, and asked that the paper be withdrawn. I return to the question of what AI does to advantage in chapter 6, with the evidence that did survive, which is weaker and points partly the other way.
A population is different: across many actors in the same domain, the total number of outcomes is far larger than any one career contains, and the population can support inferences that no individual record can. The spread between return on capital and its cost persists across thousands of firms; management practice moves productivity across dozens of plants; serial founders succeed more often than first-timers across thousands of ventures. Each of these is knowable at the level of the population and unknowable at the level of the case.
Medicine has used that asymmetry for seventy years. The Framingham Heart Study followed thousands of residents of one Massachusetts town from 1948, recording blood pressure, cholesterol, smoking and the rest at regular visits, and waiting for the sparse, unambiguous cardiovascular events. Doctors then apply the relationship estimated across the population to a new patient, using their observable features alone. Nobody waits for the patient's own heart attacks to accumulate before estimating their risk. The same approach works in the unknowable region of the map. A board that cannot learn from three years of results whether its chief executive is good can still ask whether they have adopted the practices that, across hundreds of executives and firms, predict better results. A limited partner who cannot tell skill from deal flow in two fund records can still compare the firm's process with the process features that predict returns across the population of funds. The map sets the weight to put on the record versus the population. Where the budget is high, trust the record; where it is low, trust the population and treat the record as a small adjustment.
The medical comparison also shows the limits of the method. Features validated across a population are, by construction, generic. A practice can be validated across hundreds of firms only if it means roughly the same thing in each of them. Such studies therefore concentrate on monitoring, targets and incentives rather than on the idiosyncratic strategic choices that strategy scholars care most about. An idiosyncratic choice is an observation without a population. The method rescues evaluation of the transferable part of ability and leaves the idiosyncratic part where the map put it. And population features can be gamed, as any metric can. The method does not escape that problem; it moves the target from an outcome that is equally gameable and far less informative.
The domain can also be changed. Although forecasting pundits are unknowable, forecasters in Tetlock's tournaments were detectable: the ones selected into the top two percent after one year stayed there in the following two, defying the regression that luck predicts. The forecasters were the same; the structure, many resolvable questions scored honestly, is what moved the domain across the boundary. Any organization that wants to know who is good can often do better by asking what structure would make the question answerable than by asking it of the records it already has.
An N of One
I have founded several companies, and I am a professor. Both records are the kind this chapter says are unknowable from the outside. Two companies I co-founded survived and grew. Others did not. My early academic papers were cited more than most of my classmates' papers. Put any of this through the map and the budget is well below three. Anyone judging me by it should shrink the record most of the way toward the average. In the chapters that follow, where my account of a decision cannot be separated from my knowledge of how it turned out, I say so.
That leaves the question chapter 1 raised with the salesperson's territory. Suppose the record cannot tell us whether the venture partner, the executive, or the founder earned their early success. The early success still happened. It brought them deals, recruits, capital and invitations. Are those advantages now, and do they belong to the person or to the position? Because those questions concern the present rather than the past, the record can help with them. The next chapter takes them up.
Notes
which I call the detection budget The random-effects model, the detection budget SNR x sqrt(N), the threshold 2.5/d, the reliability weight (the budget squared, divided by one plus the budget squared), and the equicorrelation correction are developed in Karl T. Ulrich and Linda Zhao, "Lucky or Good? Outcome Noise, Effective Sample Size, and the Attribution of Skill," working paper, The Wharton School, 2026 (draft of September 28, 2026, prepared for Judgment and Decision Making). The domain calibrations summarized in The Map, including the sources for each placement, are reported there.
are paid for exactly such industry-wide luck Marianne Bertrand and Sendhil Mullainathan, "Are CEOs Rewarded for Luck? The Ones Without Principals Are," Quarterly Journal of Economics 116, no. 3 (2001): 901–932.
roughly 30 percent against 18 percent for first-timers Paul Gompers, Anna Kovner, Josh Lerner, and David Scharfstein, "Performance Persistence in Entrepreneurship," Journal of Financial Economics 96, no. 1 (2010): 18–32. The figures are success rates for venture-backed founders with a prior venture-backed success versus first-time founders in their sample.
looked very much like what chance alone would produce Eugene F. Fama and Kenneth R. French, "Luck versus Skill in the Cross-Section of Mutual Fund Returns," Journal of Finance 65, no. 5 (2010): 1915–1947.
methods for controlling false discoveries Laurent Barras, Olivier Scaillet, and Russ Wermers, "False Discoveries in Mutual Fund Performance: Measuring Luck in Estimated Alphas," Journal of Finance 65, no. 1 (2010): 179–216.
the most extreme performances are disproportionately produced by high-variance, unreliable processes Jerker Denrell and Chengwei Liu, "Top Performers Are Not the Most Impressive When Extreme Performance Indicates Unreliability," Proceedings of the National Academy of Sciences 109, no. 24 (2012): 9331–9336.
about two thirds of the persistence in top-quartile returns Morten Sorensen, "How Smart Is Smart Money? A Two-Sided Matching Model of Venture Capital," Journal of Finance 62, no. 6 (2007): 2725–2762. The allocation between sorting and influence is a structural estimate and depends on the model's matching assumptions.
Phil Rosenzweig made the case in The Halo Effect in 2007 Phil Rosenzweig, The Halo Effect ... and the Eight Other Business Delusions That Deceive Managers (New York: Free Press, 2007). The halo effect is delusion one; connecting the winning dots, lasting success, and absolute performance are among the eight others.
Richard Priem and John Butler argued in 2001 Richard L. Priem and John E. Butler, "Is the Resource-Based 'View' a Useful Perspective for Strategic Management Research?" Academy of Management Review 26, no. 1 (2001): 22–40.
the firms' profitability and investment moved Morten Bennedsen, Francisco Pérez-González, and Daniel Wolfenzon, "Do CEOs Matter? Evidence from Hospitalization Events," Journal of Finance 75, no. 4 (2020): 1877–1911.
MIT's economics department announced MIT Department of Economics, "Assuring an accurate research record", May 16, 2025. The statement says MIT "has no confidence in the provenance, reliability or validity of the data" and "no confidence in the veracity of the research" in the paper "Artificial Intelligence, Scientific Discovery, and Product Innovation," and that it asked arXiv to mark the paper withdrawn and contacted the journal to which it had been submitted. The statement does not name the author.