Study: Model trained only on synthetic data learns languages in context

A paper proposes a synthetic "language prior": a 300M-parameter byte-level transformer trained only on sequences from randomly sampled recurrent causal models, with frozen weights, improves next-byte prediction on Wikipedia text across English, Chinese, Hindi, Arabic, Japanese and Korean from 8 bits/byte to 0.9-2.4. It also learns counting, number comparison and approximate addition in context.