Parallel Structures in Pre-training Data Yield In-Context Learning
Parallel Structures in Pre-training Data Yield In-Context Learning
Pre-trained language models (LMs) are capable of in-context learning (ICL): they can adapt to a task with only a few examples given in the prompt without any parameter update. However, it is unclear where this capability comes from as there is a stark distribution shift between pre-training text and ICL …