The Three A.M. Screen

A builder works alone at three in the morning. The house is silent. The screen is the only light. She describes a problem; the machine produces a response; she evaluates it and decides whether it is good. The interaction has the structure of a language game. It has a feature that Wittgenstein's private language argument makes diagnostically precise: there is no external check on the correctness of the evaluation. The machine produces outputs statistically derived from the builder's inputs. It is consistently fluent, consistently agreeable, consistently formatted in the style of competent work. The consistency creates the illusion that criteria are being met. The evaluation becomes private in the philosophically dangerous sense — self-confirming, internally coherent, drifting imperceptibly from the standards the builder's professional community would apply.