CONCEPT
In-Context Learning
The emergent ability of a language model to learn from examples placed in its prompt — no gradient updates, no fine-tuning. The most surprising capability next-token prediction produced, and the engine of modern prompt engineering.
In-context learning is the phenomenon by which a pretrained language model, given a handful of examples in its prompt, generalizes to new inputs of the same pattern — without any weight updates. A model that has never been trained on a specific translation task can, given three English–French pairs in the prompt, produce reasonable French translations of a fourth English sentence. The behavior looks like learning because the model's output adjusts to the examples; it is not learning in the parameter-updating sense because nothing in the model changes. It is, mechanically, a consequence of how the attention layers route information during inference. It is, operationally, one of the most-used and least-understood properties of contemporary LLMs.
In The You On AI Field Guide
In-context learning was introduced as a named phenomenon by Brown et al.'s Language Models are Few-Shot Learners (GPT-3, 2020), though it had been observed in smaller models earlier. The paper's headline finding was that a large enough model