Issue 2026-11-10 · Voices
Percy Liang @percyliang
Why was Marin 535B-A23B the size it was? Why 8 of 384 experts? Why GatedNorm, sconv, XSA? Why linear learning rate? All of these factors were driven by tradeoffs between timelines, compute efficiency, MoE router stability, etc. Read about the rationale of each design decision: [引用 @classiclarryd]: Sharing the thought process behind the architecture for Marin 535B-A23B, which finished pretraining on 18T tokens last week: https://t.co/ZdeshNHhcy Earlier this year the Marin project had 6 full-time employees at Open Athena, a mix of AI veterans and people newer to the space, like me. I learned a lot from the team as an IC. Since then we got our B200 GPU stack running in a month, scaled up 300…
Read original ↗Topics:Percy Liang