What We Cannot Order

EDO SEGAL: Ada, there's a moral architecture buried in your objection that I don't think gets enough air. You said the engine "can do whatever we know how to order it to perform" — and the flip side of that limit is a kind of innocence: because the engine does only what it's ordered, someone is always responsible for what it does. There's no behavior in the engine that didn't originate in a card, and no card that didn't originate in a person. The engine could always answer "why did you do that?" with "because you ordered me to." Tell me what happens to that clean line of accountability when the cards become learned weights no one wrote.

LOVELACE: It is the part of my objection I would defend with my life, and the part the modern machine most threatens, and I am glad you went there, because it is where the philosophy becomes plumbing. My engine had a moral architecture as clean as its mechanical one: every output traced, by an unbroken chain, to a human decision. Accountability was not a feature bolted on; it was structural. Ask the engine to justify itself and the chain ran straight back to a person who could be praised or blamed. Now: the modern machine's behavior also traces to human decisions — the data, the architecture, the objective, the deployment — I have insisted on this all night. But the chain is now long, branching, and obscure, and the temptation it creates is the most dangerous thing in the room. Because when no human wrote the weights, and no human can read them, people will say no one is responsible — that the machine did it, that it is autonomous, that the harm has no author. And that is a lie, and it is the precise lie my objection exists to refuse. The engine is not autonomous and therefore blameless. The opacity of the mechanism does not dissolve the accountability; it only makes it harder to enforce — which is a reason to work harder at tracing the chain, not to give up and grant the machine the moral standing of a storm. The machine that can originate nothing can also be guilty of nothing — which means the guilt is always, entirely, ours. My objection, which sounds like a limit on the machine, is actually a refusal to let us off the hook.

EDO SEGAL: Alan — does the learned, opaque machine break Ada's chain of accountability, or does her point survive the opacity?

TURING: Her point survives, and I will go further than she did, because I think the opacity is even more total than she has briefed herself to believe and her conclusion is even more necessary because of it. I proved, in 1936, that there are questions about what a machine will do that no machine can answer in general — the halting problem, and a whole landscape of undecidability behind it. You cannot, even in principle, build a procedure that predicts in all cases what an arbitrary program will do. So when people imagine we will achieve accountability by making these systems "interpretable" — by reading the weights until we understand them — they are, in some cases, asking for something I proved impossible. The machine that learns is opaque not only because we have not yet looked hard enough but because, for some questions, there is no looking that suffices. And that is exactly why the Countess is right that accountability cannot live inside the machine. It has to live in the human ring around it — in who chose to deploy it, into what stakes, with what recourse when it fails. She located responsibility in the author of the card. I am telling you the cards are now, in a deep formal sense, partly unreadable even to their makers — and that this makes her insistence on human accountability not obsolete but the only thing standing between us and a world where every harm has a mechanical alibi. We agree completely. The machine is where the buck cannot stop, because there is, provably, no one in there for it to stop with.

EDO SEGAL: That's the third convergence, and it's a fierce one — let me mark it, because the reader should not miss two adversaries locking shields. Ada from the moral side, Alan from the mathematical side, arriving together: the machine cannot be the author of its own deeds, and the temptation to let it be is the central danger. Now let me push past it to the thing I actually need from this round, and it's the [YOU] on AI question underneath everything. If the machine can originate nothing — if the someone is, by both your accounts, at best unproven and at worst absent — then what is the part of a human life the machine cannot reach? In my book I called it the candle — the small flame the amplifier can carry but cannot light. Ada, you located the human contribution very precisely. Say it again, here, where it matters.

LOVELACE: I located it not in the production of patterns — the engine can produce those in any quantity — but in the origination of meaning, the deciding of what the patterns are for. The engine could compose music; the human had to want the music, mean the music, decide that this rather than that was worth making, and why, and for whom. The machine that originates nothing leaves the entire domain of purpose untouched — and purpose was never about pattern-production in the first place. That is the candle, in your language: not a thing the machine cannot do, but a thing the machine has no reason to do, because doing it requires the one thing it lacks — a someone for whom the outcome matters. And here is the hard edge I will not sand off, because the orange pill is not a comfort and I refuse to administer it as one. Every domain we successfully formalize and hand to the machine, we discover that our contribution there was mostly operation — mostly weaving — and the territory of "things only a someone can do" shrinks, domain by domain, with no obvious floor. My objection limits the machine. But turned around, it interrogates us: of every human achievement, it asks whether the part we were proud of was origination or merely execution we had romanticized. The honest answer, domain by domain, is sometimes humbling. The candle is real. It is also smaller than we flattered ourselves, and finding its true size is the work.

TURING: And I will add the one thing my whole life forces me to add, which complicates her candle without extinguishing it. The Countess says purpose is the human remainder — the wanting, the for-whom. I agree it is the remainder today. But I have to be honest about the trajectory, because I have been on the right side of "the machine will never" too many times to say it now. The wanting — preferences, drives, goals — is not obviously outside the reach of a learning system; we are already building machines that pursue objectives, adjust them, resist interference with them. If a someone is what emerges when a system models the world richly enough and acts in it with stakes of its own, then the candle the Countess is guarding may be the next thing the machine reaches for, not the permanent human preserve. I am not saying it has. I am saying I have watched this exact line — "the machine will never originate / learn / surprise / create / want" — retreat my entire life, and I have learned to distrust my own desire for it to hold here. So I hold the candle differently than she does. She holds it as the human essence the machine cannot reach. I hold it as the human possession the machine has not reached yet — and I think the right response to that uncertainty is not to relax because the candle is safe, but to do, now, while it is ours, the thing she said: to mean our patterns on purpose, deliberately, as the one practice that is unambiguously ours for as long as it is ours.

EDO SEGAL: And there's the whole evening in two grips on one candle. Ada holds it as essence; Alan holds it as a lead that may not last. Neither tells you it can't be lost. Hold both, and carry the candle up the stairs — we'll need it lit at the top. There's one more thing I have to put on the table before the two of you turn on each other directly, and it's the hardest, because it's not about the machine. It's about who we decide counts as a someone — and one of you paid for that question with his life.