FineInstructions: Scaling Synthetic Instructions to Pre-Training Scale
IntermediateAjay Patel, Colin Raffel et al.Jan 29arXiv
Large language models usually learn by guessing the next word, then get a tiny bit of instruction-following practice; this paper flips that by turning massive web documents into instruction-and-answer pairs at huge scale.
#FineInstructions#synthetic instruction–answer pairs#instruction-tuning pre-training