Percy Liang is a computer science professor at Stanford University who works on machine learning and language models and is involved in the Marin open model project. Recent posts cover synthetic data scaling laws, an expert-rated study of AI-proposed research questions, Marin 535B-A23B pretraining progress and a 25T-token open dataset pipeline, a new NanoGPT training record, and priced-guidance LLM forecasting of papers.
Scaling laws depend on a metric that improves gradually with scale in order to extrapolate to a large target scale. We are interested in the accuracy metric, but you might get zero accuracy for small models. You can't extrapolate from a bunch of zeros. Key i…
Marin 535B-A23B now on 65K contexts... Counting down...
Synthetic data is used all the time now for LM pretraining...but can we carefully quantify the gains in compute efficiency and data efficiency? [Quoting @AhmedSQRD]: Synthetic rewrites are now standard to improve data quality and when data is limited. But how…
Recent weeks have made it clear that AI excels at solving research problems. How good is it at proposing *interesting* research problems? Thanks to the ratings of 29 experts across math, stats, CS, physics, we found 1/3 were interesting, and we fully expect t…
The final stretch... [Quoting @dlwh]: Marin's 535b is now >80% through pretraining! Here’s loss vs. our scaling-ladder forecasts. The sharp drop comes with the 4K→16K context switch, so we're kind of cheating the forecast. (This was planned.) We'll re-eval…
Most classes teaching the algorithms, but a lot of ML is about the empirical intuitions that develop from experience. This class systemizes this process. Assessment is next experiment prediction. [Quoting @tatsu_hashimoto]: Deep learning is often taught as a…
What is the probability that Astra today can generate/forecast a particular new paper past its cutoff date? Naive sampling is intractable (tail probability is astronomically small). The key idea of priced guidance is that you can lower bound this probability…
Given how many people have tried their hand at NanoGPT, I really didn't expect a 40% reduction in a single go! [Quoting @classiclarryd]: New historic NanoGPT record at 39.9s (-27.7s) from @DevenPzak , obliterating the prior record of 67.6s! This record introd…
Marin believes strongly in the power of the open community. There are so many datasets that have been produced, living on Hugging Face. We have built scalable infrastructure that ingests this data, performs deduplication, decontamination, mixing, producing 25…
Glad to see another live training run! Here's how Marin 535B-A23B is doing today: Anyone else want to share? [Quoting @_LuoFuli]: Nearly half a year of silence. We spent it studying one problem: how far RL can scale. MiMo-V2.6 is in the middle of its RL run…
The A.I. Daily