FineWeb-Edu

Experimenting with hypersurface-constrained dynamic weight updating

A researcher shared a small language model architecture experiment that generates weight deltas from learned hypersurfaces, looping a single decoder block with dynamically updated weights to cut training parameters and VRAM. Pre-training used a 10B-token FineWeb-Edu sample with frozen GPT-2 embeddings, NoPE, and sequence length 1024.