Neural Networks

In the AI Story

The Orange Pill Asimov volume uses neural networks as the point of contrast with the positronic brain. Where Asimov's robots have explicit rules operating on a designed substrate, real AI systems have implicit preferences operating on a learned substrate. This asymmetry is not a shortcoming to be fixed; it is a fundamental property of the architecture that made modern AI possible at all.

Every technique for understanding neural network behavior — prompt engineering, interpretability research, red-teaming, RLHF — is a workaround for the fact that the system's "reasoning" is not accessible the way positronic reasoning would be. Asimov's fiction, read against this background, is a record of what the alternative would have looked like.

Neural networks' mid-century origins were ambitious and quickly stalled. McCulloch & Pitts's 1943 threshold-unit paper proposed a formal model of biological neurons; Rosenblatt's 1958 perceptron instantiated it; Minsky & Papert's 1969 Perceptrons demonstrated the single-layer perceptron's limits and helped collapse public and funding enthusiasm. The intervening forty years were not a straight line — backpropagation was independently rediscovered several times — but the field existed more or less underground until the 2010s, when compute, data, and architectural advances converged to make deep networks practical at scale.

Origin

The mathematical ancestors go back to McCulloch & Pitts (1943, modeling neurons as threshold units) and Frank Rosenblatt's perceptron (1958). The "deep" era dates roughly from the breakthrough results of Hinton, LeCun, and Bengio in the 2000s–2010s, enabled by GPU compute and large datasets. The transformer architecture (2017) is the current generation's foundation.

Key Ideas

Learned representations. Instead of being told what features matter, the network discovers them from data.

Gradient descent. The training process is iterative adjustment of billions of parameters toward lower loss.

Emergent capability. At sufficient scale, networks acquire capabilities that were not explicitly trained for.

Opacity. The resulting system's behavior is not derivable from its architecture in any human-legible way.

Scale beats cleverness, within limits. The current era's defining empirical finding is that for many tasks, training a larger network on more data outperforms smarter architectural inventions. This is the scaling hypothesis. Its limits are actively contested; its surprising power is the reason the field moved from symbolic AI to connectionism decisively after 2012.