One thing that evals are super important from the get go are where the harness+model inference is part of the product, e.g. if you are doing voice ai, building out a test harness to test the system is a non trivial first step.
If you switch from claude to codex(or really any other frontier lab) after a spending a little bit of time with how claude writes, you will immediately smell claude's writing from a mile away.
This is pretty basic stuff, just build a bunch of tools with any openAI compatible api and you'll get good. I think stuff like this is kinda overcomplicating simple problems. "Framework free", bruh its just jsonRPC for the most part.
This was probably made to show off for an interview or something, hence why they're throwing throwing around the term "FDE" so much. Its meant to impress a very specific type of person (gullible managers).
Actually yeah, that's kind of the original intent — not trying to go deep on any topic, just trying to get someone unfamiliar to a starting point fast. Once you've got that base, you're free to go deep on your own.
Glad to hear they are prioritizing evaluation right from the start. Usually people just throw together a rag pipeline on the knee and then judge the metrics by eye, skimming three responses in the terminal
Why does applied AI intentionally exclude a framework/harness around AI? The job is to harness the power of AI, and a harness is a critical part of that.
One thing that evals are super important from the get go are where the harness+model inference is part of the product, e.g. if you are doing voice ai, building out a test harness to test the system is a non trivial first step.
Man, I could have used something like this last year.
Upon review, it appears that fumbling my way through the dark with Claude and Google led to the same place, in nearly all cases.
However, this is all written by Claude — it has too many em-dashes to not be, does it not? So, maybe that's why we ended up in the same places.
Does anyone know of any other resources in this vein?
If I get told about a swappable seam one more time I am going to lose it.
But were they load-bearing?
If you switch from claude to codex(or really any other frontier lab) after a spending a little bit of time with how claude writes, you will immediately smell claude's writing from a mile away.
This is pretty basic stuff, just build a bunch of tools with any openAI compatible api and you'll get good. I think stuff like this is kinda overcomplicating simple problems. "Framework free", bruh its just jsonRPC for the most part.
This was probably made to show off for an interview or something, hence why they're throwing throwing around the term "FDE" so much. Its meant to impress a very specific type of person (gullible managers).
Actually yeah, that's kind of the original intent — not trying to go deep on any topic, just trying to get someone unfamiliar to a starting point fast. Once you've got that base, you're free to go deep on your own.
Glad to hear they are prioritizing evaluation right from the start. Usually people just throw together a rag pipeline on the knee and then judge the metrics by eye, skimming three responses in the terminal
Why does applied AI intentionally exclude a framework/harness around AI? The job is to harness the power of AI, and a harness is a critical part of that.