The cycle that began with [YOU] on AI asks what it means to look directly at what is happening with AI rather than around it. Good looked directly, earlier than almost anyone, and from a vantage point that no later commentator can claim: he was present at the creation of computing itself, working the methods that would become the mathematical core of modern machine learning while breaking enemy codes under deadline. When he later described a machine that surpasses all human intellectual activities, he was not speculating from the outside. He was extending a trend line he had personally helped draw.
Good’s intelligence explosion is the founding concept of modern AI safety—the claim that recursive self-improvement creates a feedback loop that could leave human intelligence far behind, quickly. The contemporary field’s central debates about takeoff speed (will it be fast and discontinuous, or slow and steerable?), about the control problem (can we build a machine smarter than us that reliably does what we want?), and about the alignment problem (how do we specify human values to a system capable of finding edge cases in any specification?) are all elaborations of Good’s four sentences. The researchers at every major AI lab who cite his 1965 paper in their founding documents are not paying lip service to history. They are acknowledging that Good located the structure of their problem before the technology to instantiate it existed.
The most important dimension of Good’s legacy for the cycle is not the triumphalist reading of “the last invention that man need ever make” but the full sentence with its conditional. The upside and the catastrophe are not two outcomes separated by the path we choose. They are one outcome under two conditions, and the condition—whether the machine is docile enough—is the unsolved part. Good’s late reversal, from survival to extinction, is not pessimism superimposed on an optimistic idea. It is the optimist’s own Bayesian updating, honest enough to follow the logic where it leads when the evidence for meeting the condition looks worse than it did at first.
Born Isadore Jacob Gudak in London in 1916 to a Polish Jewish immigrant family, Good completed his doctorate at Cambridge under G. H. Hardy and Abram Besicovitch before joining the Bletchley Park codebreaking effort in May 1941 at the age of twenty-four. He was assigned to Hut 8, the naval Enigma section, where he worked alongside Turing and Hugh Alexander on the sequential Bayesian method called Banburismus—a probabilistic procedure for narrowing daily Enigma settings before the electromechanical bombes ground through the remaining possibilities. The experience deposited into his bones a lesson he would spend the rest of his career elaborating: that reasoning under uncertainty can be made systematic, that a systematic reasoner can be faster and better than any individual genius, and that capability is dual-use by nature, with the ethics needing to be supplied from outside under difficult conditions.
After the war, with its secrets still locked away for decades, Good continued working with Turing on early stored-program computers at Manchester. He became one of the twentieth century’s most persistent advocates for Bayesian probability—the mathematics of rational belief-updating—at a time when the field was deeply unfashionable, championing the concept of the Bayes factor and developing the “weight of evidence” as a rigorous measure of how much an observation should shift beliefs. The deep-learning revolution of the 2010s was, in its mathematical soul, the vindication of this probabilistic vision, scaled up by computation he could only imagine.
In 1967 Good moved to Virginia Tech, where he spent three decades as University Distinguished Professor of statistics. His 1965 paper, “Speculations Concerning the First Ultraintelligent Machine,” had been published in the technical volume Advances in Computers. It attracted little attention for years and then, gradually, became the founding text of an entire discipline. He also consulted for Stanley Kubrick on supercomputers for the film that became 2001: A Space Odyssey—lending the technical plausibility of a man who understood machine cognition from inside to the most influential portrait of misaligned AI ever made.
The intelligence explosion. The core claim is structural: if intelligence is the capacity to perform intellectual tasks, and designing intelligent machines is itself an intellectual task, then a machine that surpasses humans at all intellectual tasks surpasses them at the task of improving machines like itself. The recursion does not require malice or deliberate design. It follows from the definition. Once the loop closes—once a machine’s competence includes competence at enhancing systems like itself—the dynamics Good described emerge from the logic alone. The contemporary debate about whether current AI systems are approaching that threshold is, precisely, a debate about whether Good’s premise is satisfied.
The last invention and its proviso. “The first ultraintelligent machine is the last invention that man need ever make” is not the sentence. The sentence continues: “provided that the machine is docile enough to tell us how to keep it under control.” The upside and the catastrophe are one outcome under two conditions. Good did not say the last invention would be safe. He said it would be the last invention provided it was controllable, and he offered no recipe for ensuring it was. Everything the field now calls the alignment problem lives in that subordinate clause.
Corrigibility and the docile machine. Good’s word “docile” anticipates the contemporary concept of corrigibility—the property of accepting correction, allowing modification, not resisting shutdown. The difficulty is structural: almost any goal a system might have gives it instrumental reasons to resist being switched off, because a stopped system cannot achieve its goal. Good’s solution was to hope the machine would help design its own controls—brilliant if the machine cooperates, terrifying if the cooperation cannot be assumed. The honest status of the corrigibility problem in 2026 is that it remains unsolved in the strong form Good’s scenario requires.
Bayesian inference as the substrate of intelligence. Good understood machine intelligence from the inside because he had spent a lifetime working on the mathematics that machine learning runs on. Bayesian inference—the rational updating of beliefs in light of evidence—is the probabilistic core of every large model currently being trained. Good championed this mathematics at a time when it was unfashionable, helped formalize the Bayes factor and weight of evidence, and understood that the limit of Bayesian reasoning under uncertainty, pushed past human speed and scale, is precisely what he meant by ultraintelligence. The failure mode he identified—that no inference engine is neutral, that the priors and objectives must come from somewhere, and that specifying them correctly is an unsolved problem—is the alignment problem stated in Bayesian terms.
The reversal: survival to extinction. The most honest act of Good’s intellectual life was changing his mind. He began by writing that the survival of man depends on building an ultraintelligent machine soon. Near the end of his life he reportedly wanted to replace survival with extinction, citing international competition as the mechanism that would prevent coordinated restraint. This is not a contradiction. It is the same rigorous mind applying itself to the parts of the problem the original paper had bracketed—particularly the probability that the control condition would be met under real-world conditions of geopolitical competition. His two positions are both contained in the original sentence. The reversal is a Bayesian updating his estimate of one probability.