I always get annoyed when people misinterpret Occam’s razor. It’s not that the simplest is more likely to be correct, it’s that you should prefer it, because it’s simple.
It’s just like the Hopper quote. She said it’s better to ask for forgiveness during the fog of war, doing something you thought was right, not to do something you knew they were going to say no to and now you are trying to get away with something.
There are also various metaphysical theories that posit that the universe is algorithmically generated in some sense or the other, and from many of those theories it follows that simplicity is a fundamental feature of reality, which yields an even stronger version of Occam’s Razor.
It's partly because of disagreement about what "complexity" and "simplicity" actually are. Many simple statements are in fact backed by massively complex, unstated assumptions. In attempting to deal with assumptions, scientists necessarily end up having to deal with the complexity involved in that. But the problem is if you don't engage with that, how do you know what is really more conformant with Occam's razor, as opposed to just satisfying what is readily expressed in common language?
Likewise in software development a C developer will say abstractions are not simple, a java programmer will argue that dealing with low levels details is not simple. They're both kinda right but will resort to framings which back their world view.
That's not true. It's pretty clear that she meant "do something you knew they were going to say no to and now you are trying to get away with something."
> So I want to tell something to all the young people here on many many occasions you'll find it is much easier to apologize than it is to get permission.
You do it then when somebody comes after you and say are you supposed to do that, "oh gee I didn't know I wasn't supposed
to do that" ...
so just remember it's frequently much easier to apologize than it is to get permission do it
She goes on further, explaining how to deceive your superiors to manipulate them to get what you want.
But I still don’t think that means eat all the cookies in the cookie jar and then apologize after because nobody would have given permission. That’s still about doing what you believe to be right. She even frames the fallout as “where you supposed to do that?” and not “you shouldn’t have done that”.
Not sure about Hopper, as I recall biographers of Lawrence of Arabia certainly made it seem like he was using the fog of war to do things he knew his superiors may object to.
Regardless, even if its misinterpreted it still has a kernal of truth and separate utility than your version, that is: the people in the field closest to the action have an operational awareness that may result in better decisions in times of urgency.
Ah, over-the-top larger-than-life LLM-isms, they are really funny when you see them in a company blog, but they are vomitive when it's your coworker copy-pasting it and insisting you on reading it.
These models are trained on human language, which belongs to us, we shouldn’t surrender it to them. Keep the em-dashes. IMO don’t overuse negative parallelisms though, they were always bad and lazy.
Compression in this modern day and age is so slop.
Yes, I'm familiar with keystone results such as Solomonoff induction. It's a direct counterexample to compression - your intensional algorithm can completely outrun reality. I can literally specify a huge mega-algorithm that just searches over all possible Turing machines and evaluates them, and it's an optimal compressor. It's completely vacuous though. You can always hide the "heavy work" in your mappings and descriptions. It's ironic that a kolomogorov complexity minimizer is so loaded that it's vacuous.
This is pretty much why I roll my eyes at this point at all the compression is intelligence memes.
I wonder when intervention and causality will hit the mainstream. These tools were designed specifically to counteract purely predictive theories. But your average compression dude will hold tight to their paradigms and slogans, not realize their internal contradictions (that their own field has brought up), and then whenever a new paradigm suddenly becomes visible and mainstream, they'll latch onto that. It's not principled at all.
And to be clear - I do think intelligence is some amount of compression, and I am well aware of formal results such as the arithmetic decoding theoretical and empricial result. Just annoyed. It's literally no different than the whole Bayesianism meme. If you're not actually practicing that type of intelligence as a basis, then you don't get to go around beating the drum about how it's the ultimate reality. You're just spouting dogma to feel like part of an in-group.
I don't take issue with that. Attempting memorized pattern generalization through holdout / validation strategies is a big part of ML that you would not typically see with econometrics / psychometrics / possibly sabermetrics / most other -metrics. Philosophically the explain versus predict divide. https://www.stat.berkeley.edu/~aldous/157/Papers/shmueli.pdf
That's a bit pedantic no. Memorization in ML refers to the model having the wrong level of capacity such that it's too hard to optimise it such that it doesn't memorize the _training examples_ themselves.
I always get annoyed when people misinterpret Occam’s razor. It’s not that the simplest is more likely to be correct, it’s that you should prefer it, because it’s simple.
It’s just like the Hopper quote. She said it’s better to ask for forgiveness during the fog of war, doing something you thought was right, not to do something you knew they were going to say no to and now you are trying to get away with something.
I think you should get less annoyed.
> It’s not that the simplest is more likely to be correct, it’s that you should prefer it, because it’s simple.
I don't know what Occam meant, but if you accept the formalism of PAC learning, it is more likely to be correct
https://web.archive.org/web/20170428225156/http://www.cse.bu...
https://web.archive.org/web/20130412062821/http://cs.ecs.bay...
None of your links work for me.
Ah! from my very dated and messy bibtex file comments. Wait, let me search for them on archive.org.
Fixed.
There are also various metaphysical theories that posit that the universe is algorithmically generated in some sense or the other, and from many of those theories it follows that simplicity is a fundamental feature of reality, which yields an even stronger version of Occam’s Razor.
Marcus Hutter formalized this in his AIXI work.
And sadly, in academia, complexity (opposite of Occam's razor) is what gets you published.
It's partly because of disagreement about what "complexity" and "simplicity" actually are. Many simple statements are in fact backed by massively complex, unstated assumptions. In attempting to deal with assumptions, scientists necessarily end up having to deal with the complexity involved in that. But the problem is if you don't engage with that, how do you know what is really more conformant with Occam's razor, as opposed to just satisfying what is readily expressed in common language?
Likewise in software development a C developer will say abstractions are not simple, a java programmer will argue that dealing with low levels details is not simple. They're both kinda right but will resort to framings which back their world view.
That's not true. It's pretty clear that she meant "do something you knew they were going to say no to and now you are trying to get away with something."
https://youtu.be/wHdHCoeUbU4?t=861s
> So I want to tell something to all the young people here on many many occasions you'll find it is much easier to apologize than it is to get permission. You do it then when somebody comes after you and say are you supposed to do that, "oh gee I didn't know I wasn't supposed to do that" ... so just remember it's frequently much easier to apologize than it is to get permission do it
She goes on further, explaining how to deceive your superiors to manipulate them to get what you want.
But I still don’t think that means eat all the cookies in the cookie jar and then apologize after because nobody would have given permission. That’s still about doing what you believe to be right. She even frames the fallout as “where you supposed to do that?” and not “you shouldn’t have done that”.
Damn no wonder she got a supercomputer named after her
>It’s just like the Hopper quote.
Not sure about Hopper, as I recall biographers of Lawrence of Arabia certainly made it seem like he was using the fog of war to do things he knew his superiors may object to.
Regardless, even if its misinterpreted it still has a kernal of truth and separate utility than your version, that is: the people in the field closest to the action have an operational awareness that may result in better decisions in times of urgency.
Even tech giants are putting out articles seemingly fully written by Claude.
The animated graphic labeled "Occam's razor, formalized" is bizarre. Is that really visualizing "Occam's razor, formalized"?
Ah, over-the-top larger-than-life LLM-isms, they are really funny when you see them in a company blog, but they are vomitive when it's your coworker copy-pasting it and insisting you on reading it.
I was expecting Occam wearing a suit.
Nobody wants to work anymore!
Time to first detected slop in this article is <1s. Claudisms per paragraph is also very high.
Is it too much to ask from people to read their own article anymore?
If anyone read this at all, they would have had the ick, and would have fired off a prompt to get rid of the most popular AI slop tells...
What I really dislike is having to edit my own non-LLM assisted writing to make sure I'm not accidentally confused with AI.
I caught myself writing "And that matters because..." in a HN comment but had to edit myself. Also miss uising emdashes.
These models are trained on human language, which belongs to us, we shouldn’t surrender it to them. Keep the em-dashes. IMO don’t overuse negative parallelisms though, they were always bad and lazy.
An arms race on style would be interesting. Essentially a real life GAN.
> Why don't machine learning research agents overfit?
they do.
Compression in this modern day and age is so slop.
Yes, I'm familiar with keystone results such as Solomonoff induction. It's a direct counterexample to compression - your intensional algorithm can completely outrun reality. I can literally specify a huge mega-algorithm that just searches over all possible Turing machines and evaluates them, and it's an optimal compressor. It's completely vacuous though. You can always hide the "heavy work" in your mappings and descriptions. It's ironic that a kolomogorov complexity minimizer is so loaded that it's vacuous.
This is pretty much why I roll my eyes at this point at all the compression is intelligence memes.
I wonder when intervention and causality will hit the mainstream. These tools were designed specifically to counteract purely predictive theories. But your average compression dude will hold tight to their paradigms and slogans, not realize their internal contradictions (that their own field has brought up), and then whenever a new paradigm suddenly becomes visible and mainstream, they'll latch onto that. It's not principled at all.
And to be clear - I do think intelligence is some amount of compression, and I am well aware of formal results such as the arithmetic decoding theoretical and empricial result. Just annoyed. It's literally no different than the whole Bayesianism meme. If you're not actually practicing that type of intelligence as a basis, then you don't get to go around beating the drum about how it's the ultimate reality. You're just spouting dogma to feel like part of an in-group.
They tend not to overfit ... when there are way more data points than parameters.
Or when there are way more parameters than data points
https://en.wikipedia.org/wiki/Double_descent
arXiv link: https://arxiv.org/abs/2606.11045
The methodology is partially based on https://www.offconvex.org/2021/04/07/ripvanwinkle/ , for those thinking this sounded familiar.
Wherein Claude gives an honest assessment that it genuinely does not overfit. I also had Grok telling me that it isn't quantized.
Do the submitters really not notice that this is AI slop? Do they like this? It is a complete pain to read.
No point in reading anything AI related anymore. It's all slop.
We need to retvrn to rss feeds
what would returning to rss feeds achieve?
[delayed]
> Machine learning, at its core, is about generalization, not memorization.
Well they memorize the patterns.
memorization doesnt mean rote learning.
I don't take issue with that. Attempting memorized pattern generalization through holdout / validation strategies is a big part of ML that you would not typically see with econometrics / psychometrics / possibly sabermetrics / most other -metrics. Philosophically the explain versus predict divide. https://www.stat.berkeley.edu/~aldous/157/Papers/shmueli.pdf
That's a bit pedantic no. Memorization in ML refers to the model having the wrong level of capacity such that it's too hard to optimise it such that it doesn't memorize the _training examples_ themselves.