Sebastian Raschka @rasbt

Whoa, great and even excessive Marin Mochi (535B-A23B) checkpoint drop (in a good way!) Comes with great tech report btw with insights into why / how components were picked (for training stability and GPU throughput). Short fact sheet: 1) Latent MoE with an extra RMSNorm on the latent that made latent MoE actually work (better) 2) Classic 3:1 sliding-window/global gated GQA, partial RoPE (local) and NoPE (global) 3) Adding short convolutions on keys, attention output, and MoE output 4) Exclusive self-attention (XSA) -- I guess the end-of-the year architecture roundup this year won't be boring 😅 [引用 @dlwh]: We’re releasing all 57 intermediate checkpoints from Marin “Mochi” 535B-A23B pretraining, including 10 from long-context extension. 18.2T tokens, about 61 TB of weights. https://t.co/MmfZeQdB1V Blog: https://t.co/kNCJ6PoNW5 https://t.co/rVApf8wIaH

X15 h ago
Read original ↗