This is a neat little trick, but I wonder if you could do substantially the same thing by just prompting/LoRA finetuning the model to produce a single-token output ("yes" or "no"). This only requires a single model forward pass, you can use the same KV caching strategy for shared parts of the prompt, and isotonic regression should work just as well to calibrate the output logits. I guess if you use this method and probe on an internal layer you can skip all the remaining layers, which could be a nice inference speedup.
This is a neat little trick, but I wonder if you could do substantially the same thing by just prompting/LoRA finetuning the model to produce a single-token output ("yes" or "no"). This only requires a single model forward pass, you can use the same KV caching strategy for shared parts of the prompt, and isotonic regression should work just as well to calibrate the output logits. I guess if you use this method and probe on an internal layer you can skip all the remaining layers, which could be a nice inference speedup.
[dead]