Metacognition in our brains is of course part of cognition and it’s constrained by the time needed for a response and your energy budget (eg. it’s lower if you’re tired)
It seems humans subconsciously adjust these constraints constantly in real time too.
There’s just a ton of machinery before we get close to something we can call intelligence..
But in the meantime, we should be grounding a model’s responses by giving it context and letting it research with tools; and having secondary out of band processes that go over what the model emitted and enriching the UI with “warning: possibly untrue/nuanced” underlines, call-outs, sources, and outright killing some responses from ever making it to a user. Constrained output could also let the model self-enrich with sources; Anthropic and OpenAI both have API modes and UIs that do this.
IDK if the author's 'metacognition' needs to be a feature of the LLM itself.
I could imagine a harness that 1) reads LLM output 2) uses a research sub-agent to attempt to verify any factual claims 2) rephrase the main agent's output such that it conveys uncertainty if the factual claim cannot be independently verified
Unproductive tangent: Why do we call it "hallucinationing" instead of "bullshitting" when that is so clearly what it is?
If I'm talking to a guy that says, "I have a really fast metabolism, that's why I can eat whatever I want", he's not hallucinating - he's full of shit.
bullshitting is an intent; hallucination is a misrepresentation of something.
I agree that the way models are trained, their basic presentation is bullshitting, but models dont have intent. Nor, do I think, they can be trained to have intent. They have a k value that generates non-determinism for them, that leads to the bullshitting.
But the intent part is what I don't think bullshitting properly describes. LLMs don't have intent.
I do think you could refer to their affect as narcissism. Every time I see them bullshit something, it's often because it thinks there's no way what _its doing_ is wrong; like when it tries to run pgsql using default settings (despite being told the credentials), it reports it can't connect because the database isn't running, then it goes and tries to run it. Sometimes it gets back correctly, sometimes not.
But it's not bullshitting because there's no intent it's essentially just the type of narcissism that everything else is wrong.
Metacognition in our brains is of course part of cognition and it’s constrained by the time needed for a response and your energy budget (eg. it’s lower if you’re tired)
It seems humans subconsciously adjust these constraints constantly in real time too.
There’s just a ton of machinery before we get close to something we can call intelligence..
But in the meantime, we should be grounding a model’s responses by giving it context and letting it research with tools; and having secondary out of band processes that go over what the model emitted and enriching the UI with “warning: possibly untrue/nuanced” underlines, call-outs, sources, and outright killing some responses from ever making it to a user. Constrained output could also let the model self-enrich with sources; Anthropic and OpenAI both have API modes and UIs that do this.
Related: https://github.com/space-bacon/SRT
This repository empirically proves computational semiotics.
The only “metacognitive” (2nd order) and metapragmatic (3rd order) model I’m aware of.
IDK if the author's 'metacognition' needs to be a feature of the LLM itself.
I could imagine a harness that 1) reads LLM output 2) uses a research sub-agent to attempt to verify any factual claims 2) rephrase the main agent's output such that it conveys uncertainty if the factual claim cannot be independently verified
It does need to be and doing so is possible at 0 cost to CE. The SRT does just that. It actually improves the CE.
Unproductive tangent: Why do we call it "hallucinationing" instead of "bullshitting" when that is so clearly what it is?
If I'm talking to a guy that says, "I have a really fast metabolism, that's why I can eat whatever I want", he's not hallucinating - he's full of shit.
Avoiding crass language may be one reason.
Just think, one of the hullabaloos of today is because our ancestors were two Victorian to put "sex" on the birth certificate
bullshitting is an intent; hallucination is a misrepresentation of something.
I agree that the way models are trained, their basic presentation is bullshitting, but models dont have intent. Nor, do I think, they can be trained to have intent. They have a k value that generates non-determinism for them, that leads to the bullshitting.
But the intent part is what I don't think bullshitting properly describes. LLMs don't have intent.
I do think you could refer to their affect as narcissism. Every time I see them bullshit something, it's often because it thinks there's no way what _its doing_ is wrong; like when it tries to run pgsql using default settings (despite being told the credentials), it reports it can't connect because the database isn't running, then it goes and tries to run it. Sometimes it gets back correctly, sometimes not.
But it's not bullshitting because there's no intent it's essentially just the type of narcissism that everything else is wrong.