In-context Learning vs. Instruction Tuning: The Case of Small and Multilingual Language Models

Instruction following is a critical ability for Large Language Models to be used directly by humans. This often requires supervised fine-tuning on curated instruction datasets, sometimes complemented with an alignment step. However, in multilingual scenarios, obtaining high-quality data for these stages remains challenging, motivating the exploration of In-Context Learning (ICL) as a possible alternative. In this work, we study whether ICL can serve as a substitute for Instruction Tuning in multilingual language models, while also examining how the comparison changes with model scale. Our results indicate that a gap remains between ICL and Instruction Tuning, motivating further research to reduce it.