The heavy reliance on eval datasets showcased here and also evidenced by the frontier lab's high payments for such data cast doubt on the claims that AI will become self-improving soon
The frontier labs want you to believe that their models are magically gaining intelligence that generalizes to new areas with each new release. But it’s an illusion: the models are post-trained on a HUGE array of hand-curated datasets for specific tasks, with good enough coverage that it feels general.
I’ve gotten recruiting emails to be a human grader of Claude coded projects. 6 month contract at $150/hour 40hrs/week. Not insanely expansive but also not cheap.
I always wonder what you lose when you start maxing out some specific benchmark. It is really hard to do this in a balanced way and I would love if everybody that did these kind of efforts would take the time to also list the before/afters of a number of other benchmarks that they did not use in their training.
Good point! First of all, none of the benchmark tasks were used in training, though the training data was designed to improve capabilities on tasks like TB4.1. If you consider TB4.1 the main target, the blog actually shows before/after results on several other benchmarks, with no regressions and even some improvements.
Yes, I've seen that, but I wonder what is missing because I find it hard to believe that you can just do some RL and then end up with a model that is better across the board. I would find it far more believable if you outperformed on the benchmark that was your goal (assuming you selected training data to give you an edge there) and underperformed on things that you were not optimizing for. But since it all moves in one direction (better) I feel there must be some kind of unknown at play here. Even so, task specific AI is definitely the future, you don't hire a plumber to do your bookkeeping either...
I agree with you and I think, personally, that the heavy post-training like here makes the model more capable in coding but less capable for certain unbenchmarkable tasks, like prose or interpreting prompts correctly. I've been noticing that small models have been gradually being pushed towards programming and are slowly becoming _worse_ at some non-programming tasks, not better; e.g. new Deepseek models appear to be much worse at interpreting metaphors and overall feel more autistic than previous ones.
That mirrors my experience exactly. I keep a couple of older models (and much larger ones as well) for when I want to do stuff that is unrelated to computer programming because the models that are really good at that tend to fare less well at other stuff. I suspect that there is a boundary where maximizing benchmarks for coding at the expense of general knowledge is going to give you a model that is really fast and good but 'narrow' and that this may also lead to loss of performance when applied to coding tasks that are less well represented in the training set (such as, for instance, DSP code). This is a variation on 'the bitter lesson', short term you can make gains like this but long term leveraging compute and data will give you something better. I'm really happy I didn't waste half a life on engineered AI because I would be really quite miserable if I had.
Why do you find it hard to believe that you can do some RL and improve across the board (albeit less strongly on other semi-related tasks than your objective)? That's the idea behind generalization: it's not perfect but you get a little bit of cross-task generalization, so if you just keep spamming more RL environments you can hill-climb to AGI. That's what the labs are doing!
If you carefully curate the data, plus the original model isn't saturated for RL post-training, it's possible to improve targeted benchmarks without regression on other metrics. This experiment is done with terminal bench 4.1 as a target, and the regression on other benchmarks has been carefully controlled through data + infra design.
Ok, so the obvious question then is: this seems (I'm not saying it is, it just looks like a small thing compared to training up a model from scratch) a relatively trivial undertaking for the authors of the original model, if there was this much left in the tank why didn't they do something similar? Or is there some step here that only this team can perform? Or is it simply a matter of economics?
Rather than waiting, ask your assistant to quantize for you. It could be not the best quantization possible but it’s relatively quick, and I had an overall positive experience.
My bad, I haven’t really looked yet (and haven’t asked any assistants), thought it’s BF16 safetensors as usual, and for those I used to quantize to ggufs with just a couple commands.
It isn't if you train the quantized model with a quantized K/V and cache. That's my approach and usually takes around a day for 100k reasoning traces (including abliteration).
Also this is on consumer hardware, meaning 96GB CPU RAM and 48GB GPU VRAM, using an optimized Vulkan concurrent shader pipeline. So costs for the machine are around 3-5k depending on RAM crisis.
How did you guys set up the RL topology? Were you guys doing individual RL nodes (4xh200 rollout/4xh200 train) or did you have dedicated trainer and rollout nodes?
3.8 27B is an interesting choice. on the face of it, doing RL on it feels like refrying twice-cooked pork then stuffing it in the microwave for good measure. the fact that even one benchmark (tb4.1) increased more than marginally is kind of miraculous. would be interesting to see where "miles" would take eg. 3.8 flash next, which is already impressive despite being an under-trained architectural preview.
I look forward to testing this against my private eval set to determine if it's benchmaxxing or actual improvement based on just 1200 tasks. downloading it now
You'd probably end up spending more on pcie h200s. They're not that much cheaper than 8x sxm nodes and they only have 4 way nvlink. At the scale these guys were training at you'd probably want at a minimum an 8x nvlinked node for the trainer, even though you could technically fit the full training and optimizer states on a 4x h200 node.
Savings would likely be larger with magic beans such as floobarbaz, no?
But seriously. If there were a cheaper and reliable way, I'm sure the authors would have mentioned it instead of H200 SXM modules. They're trying to brag about cheap training.
Maybe you wanted to use an opportunity to shill tranium.
The heavy reliance on eval datasets showcased here and also evidenced by the frontier lab's high payments for such data cast doubt on the claims that AI will become self-improving soon
The frontier labs want you to believe that their models are magically gaining intelligence that generalizes to new areas with each new release. But it’s an illusion: the models are post-trained on a HUGE array of hand-curated datasets for specific tasks, with good enough coverage that it feels general.
The frontier labs are using all their customers' logs to train. It's the best training set there is.
Time to do all business in cockney ryming slang!
How is that different from you?
Humans generalize much more quickly than models? We don't require millions of curated examples to learn a concept?
Because there is four billion years of evolutionary pre-training.
Instead of pre-training, I'd say it's more like a better architecture, training regime, higher fidelity data, and perhaps better initial "weights".
All of that stuff was still trained by evolution, though.
Curious, where did you see the numbers on how much frontier labs are paying for eval data?
https://getlatka.com/companies/mercor
I’ve gotten recruiting emails to be a human grader of Claude coded projects. 6 month contract at $150/hour 40hrs/week. Not insanely expansive but also not cheap.
>doubt on the claims that AI will become self-improving soon
It already happened. https://github.com/karpathy/autoresearch
I always wonder what you lose when you start maxing out some specific benchmark. It is really hard to do this in a balanced way and I would love if everybody that did these kind of efforts would take the time to also list the before/afters of a number of other benchmarks that they did not use in their training.
Good point! First of all, none of the benchmark tasks were used in training, though the training data was designed to improve capabilities on tasks like TB4.1. If you consider TB4.1 the main target, the blog actually shows before/after results on several other benchmarks, with no regressions and even some improvements.
Yes, I've seen that, but I wonder what is missing because I find it hard to believe that you can just do some RL and then end up with a model that is better across the board. I would find it far more believable if you outperformed on the benchmark that was your goal (assuming you selected training data to give you an edge there) and underperformed on things that you were not optimizing for. But since it all moves in one direction (better) I feel there must be some kind of unknown at play here. Even so, task specific AI is definitely the future, you don't hire a plumber to do your bookkeeping either...
I agree with you and I think, personally, that the heavy post-training like here makes the model more capable in coding but less capable for certain unbenchmarkable tasks, like prose or interpreting prompts correctly. I've been noticing that small models have been gradually being pushed towards programming and are slowly becoming _worse_ at some non-programming tasks, not better; e.g. new Deepseek models appear to be much worse at interpreting metaphors and overall feel more autistic than previous ones.
That mirrors my experience exactly. I keep a couple of older models (and much larger ones as well) for when I want to do stuff that is unrelated to computer programming because the models that are really good at that tend to fare less well at other stuff. I suspect that there is a boundary where maximizing benchmarks for coding at the expense of general knowledge is going to give you a model that is really fast and good but 'narrow' and that this may also lead to loss of performance when applied to coding tasks that are less well represented in the training set (such as, for instance, DSP code). This is a variation on 'the bitter lesson', short term you can make gains like this but long term leveraging compute and data will give you something better. I'm really happy I didn't waste half a life on engineered AI because I would be really quite miserable if I had.
http://www.incompleteideas.net/IncIdeas/BitterLesson.html
Why do you find it hard to believe that you can do some RL and improve across the board (albeit less strongly on other semi-related tasks than your objective)? That's the idea behind generalization: it's not perfect but you get a little bit of cross-task generalization, so if you just keep spamming more RL environments you can hill-climb to AGI. That's what the labs are doing!
You can hill climb but you will (or at least, I think you will!) lose generality.
And no, that won't get you to AGI, that is a completely different goal.
If you carefully curate the data, plus the original model isn't saturated for RL post-training, it's possible to improve targeted benchmarks without regression on other metrics. This experiment is done with terminal bench 4.1 as a target, and the regression on other benchmarks has been carefully controlled through data + infra design.
Ok, so the obvious question then is: this seems (I'm not saying it is, it just looks like a small thing compared to training up a model from scratch) a relatively trivial undertaking for the authors of the original model, if there was this much left in the tank why didn't they do something similar? Or is there some step here that only this team can perform? Or is it simply a matter of economics?
Check the article. It’s the first graphic that has other benchmarks.
Yes, they all go up. I am wondering what was lost.
Sabi-noema also looks like a close competitor in the same space. Built from qwen3.8 and made better.
https://huggingface.co/sabicap/sabi-noema
Also HF model here: https://huggingface.co/RadixArk/DaisyRose-27B
Snap, no gguf to download, will wait for one to appear to test, any improvement over an already good model is good news.
"18 days on 144 H200 GPUs" insane.. didn't thought so much hardware would be needed to train Qwen.
Rather than waiting, ask your assistant to quantize for you. It could be not the best quantization possible but it’s relatively quick, and I had an overall positive experience.
It is already quantized. You need to convert to GGUF format though. Did your assistant explain this wrong?
My bad, I haven’t really looked yet (and haven’t asked any assistants), thought it’s BF16 safetensors as usual, and for those I used to quantize to ggufs with just a couple commands.
It isn't if you train the quantized model with a quantized K/V and cache. That's my approach and usually takes around a day for 100k reasoning traces (including abliteration).
Also this is on consumer hardware, meaning 96GB CPU RAM and 48GB GPU VRAM, using an optimized Vulkan concurrent shader pipeline. So costs for the machine are around 3-5k depending on RAM crisis.
[1] https://github.com/cookiengineer/qwen-reasoning-trainer
How did you guys set up the RL topology? Were you guys doing individual RL nodes (4xh200 rollout/4xh200 train) or did you have dedicated trainer and rollout nodes?
Dedicated rollout and trainer nodes, with RDMA transfers between them.
3.8 27B is an interesting choice. on the face of it, doing RL on it feels like refrying twice-cooked pork then stuffing it in the microwave for good measure. the fact that even one benchmark (tb4.1) increased more than marginally is kind of miraculous. would be interesting to see where "miles" would take eg. 3.8 flash next, which is already impressive despite being an under-trained architectural preview.
I look forward to testing this against my private eval set to determine if it's benchmaxxing or actual improvement based on just 1200 tasks. downloading it now
Is it post-training or bench-maxxing?
FWIW that is about $331,000 on low-onboarding-grief clouds (e.g. vast.ai) right now using the SXM variants they used.
You might be able to shave that down using a different flavor of H200 (like PCIe) but it would take more hours, so benefits are unclear.
You'd probably end up spending more on pcie h200s. They're not that much cheaper than 8x sxm nodes and they only have 4 way nvlink. At the scale these guys were training at you'd probably want at a minimum an 8x nvlinked node for the trainer, even though you could technically fit the full training and optimizer states on a 4x h200 node.
Savings would likely be larger with alternative accelerators such as trainium, no?
Savings would likely be larger with magic beans such as floobarbaz, no?
But seriously. If there were a cheaper and reliable way, I'm sure the authors would have mentioned it instead of H200 SXM modules. They're trying to brag about cheap training.
Maybe you wanted to use an opportunity to shill tranium.
> Maybe you wanted to use an opportunity to shill tranium.
This was unnecessary and against HN guidelines to engage in good faith.
No, and we get way too many low-effort "how does this compare to $FOO" comments that serve only to improve brand recognition of $FOO.
Once you realize people are doing it you see it everywhere.