The Functional Indistinguishability Problem

The strongest objection to Thompson's denial of AI cognition runs as follows: large language models produce outputs that are functionally indistinguishable from the outputs of conscious minds. They generate creative text, solve novel problems, appear to reason, express what appears to be uncertainty, and adapt to conversational context with flexibility that early AI researchers would have considered a sufficient condition for intelligence. If the outputs are indistinguishable, the objection continues, then insistence on a categorical difference between the processes is either unfalsifiable or irrelevant. Thompson concedes the functional point but denies the inference. The functional indistinguishability of the outputs is the problem, not the solution, because it creates a situation in which the difference between two categorically different processes — enacting meaning and generating probable tokens — becomes invisible to observers who attend only to the output.