Absolutely nothing changes about how I interview. I care whether you are “smart and gets things done”.
“Tell me about the project that you are most proud of?” And then dig in and asking them about their challenges decision making processes, and gauge the level of scope, impact and ambiguity they know how to work at.
“I see you have been working for $x years. Knowing what you know now, what would you do differently?”
“Say you are in a meeting with myself, the CEO and other senior developers who have been at the company for awhile and we all agree on an idea that in your experience you know is a bad idea, what would you do?”
Follow up question: “What would you do if after we listened to you, we decided to go in another direction?”
“Tell me about a time when you had unclear requirements , how did you handle it?” - gets back to ambiguity. There is a lot of that with startups.
We're building a CLOB-based DEX and this problem hits close, we've interviewed a lot of smart contract developers where AI makes it nearly impossible to assess real understanding. What we've shifted to: giving candidates a live matching engine with a subtle bug in the order execution logic and asking them to find and fix it. Prompting Claude gives you something that looks right but breaks under edge cases, partial fills, price-time priority violations, self-trade prevention. The candidates who actually understand the mechanics catch it. The ones steering AI without real knowledge submit a fix that breaks three other things. Your "AI steering" framing is exactly right. The real skill now is knowing when the output is plausible but wrong and that only comes from genuine domain knowledge.
> 1. Does testing a candidate's ability to "steer" and debug AI-generated code make more sense to you than traditional algorithms?
Testing the candidate's ability to "steer" agents seems to be like testing their ability to know the Java API or to recite SOLID by heart.
> 2. How are you currently preventing these "prompt-only" developers from slipping through your own interview loops?
We don't ask anymore leetcode. We keep the usual systems design interview in which usage of AI is not needed (or at least we don't allow it because in this kind of interview we are more interested in seeing how the candidate thinks and so on)
We have a new stage in our job interview, though: generic Q/A about the fundamental of software engineering/computer science. Again, we don't care anymore how candidates produce code. We care about what they know, and what they don't know. What's the scope of their knowledge, and when do they need to rely on AI to come up with an answer. Silly (non-real) example: "Can you write a program that detects if another program halts?". The people we want are the ones who would say something about the Halting Problem but also perhaps be practical and perhaps ask more questions about such a program requirements.
You get the point: we look for people with a good breadth of knowledge, who can communicate well and know their shit. Whether they can use tool x or y (including LLMs), comes for granted for such people
This is a fantastic perspective, thank you. You hit the nail on the head: the ultimate goal is testing fundamental engineering breadth and systems thinking, not tool usage.
I should definitely clarify my use of the word steering — I completely agree that testing prompt engineering is just the new API memorization, which is useless.
By steering, I mean putting them in a situation where the AI generates a plausible but architecturally flawed solution, and seeing if they have the fundamental knowledge to spot the BS, understand the scope of the problem, and fix it.
Basically, an automated way to test the exact critical thinking you mentioned.
I love your approach of dropping LeetCode for fundamentals Q/A and Systems Design. But out of curiosity, how do you scale that at the top of the funnel? Doing deep, manual 1-on-1 assessments gives the best signal by far, but doesn't that burn a massive amount of your senior engineers' time?
We binned any form of coding questions or take home task. Our preference now is to focus on architectural concepts and ability to apply those to a problem.
So, as happened last week, if I’m interviewing for an Elixir dev I’m going to be interested in your knowledge of the BEAM and how it’s features can be used to solve common architectural problems.
As a hiring manager here is what I do when I do interviews:
1. I don't do anything with code. No whiteboard. I usually don't even ask code related questions. Code literacy is a silent prerequisite. You wouldn't waste a lawyer's time asking if they could read books.
2. I talk to people. Soft skills are more important than tech skills. I want to see how they converse, their confidence, and their level of comfort.
3. I do ask technical questions and I ask them fast. It takes AI time to generate an answer. I am looking at speed of response. If the candidate needs to pause to think about a creative answer that's fine, because I am watching for that too. When they pause are they reading something from a screen or are they using their body language to convey they are using their imagination. I also pivot on the fly to other subjects between questions.
4. When I ask technical questions I try to make them as open ended as possible so that there is no single right answer. I want to know what the candidate would do in a given scenario. If I circle back on the same question 20 minutes later will I get the same answer?
5. I can usually make a determination about a candidate within 7-9 minutes of talking with them, but I schedule an hour to really make it more of a conversation and dive a bit deeper.
6. I don't really care about ASD in my current line of work, because in enterprise API management you will spend the majority of your time performing requirements discovery/analysis as opposed to writing code. So, it really is about the soft skills. If I were to go back to JavaScript I would absolutely look for ASD in the interview. Here is what I would look for:
6a. Frequency of first person pronouns. Does the candidate like to talk about themselves or is it more about a product or skill?
6b. Can the candidate measure things? Ask them to demonstrate some manner of technical performance analysis with numbers.
6c. Ask questions to inform of a bias. Does the candidate prefer objectivity when building something from scratch or must everything, even the most trivial of things, be familiar and comforting.
6d. On a scale between reckless and safety where the candidate land and how do they control for it?
As the candidate answers my questions I am really only just listening enough to know where to pivot for the next question in rudely short time. Most of what I am paying attention to is where the candidate's eyes go between talking to me versus answering a challenging question. I am also looking for the level of relaxation in the candidate's body and the stress in their voice.
Absolutely nothing changes about how I interview. I care whether you are “smart and gets things done”.
“Tell me about the project that you are most proud of?” And then dig in and asking them about their challenges decision making processes, and gauge the level of scope, impact and ambiguity they know how to work at.
“I see you have been working for $x years. Knowing what you know now, what would you do differently?”
“Say you are in a meeting with myself, the CEO and other senior developers who have been at the company for awhile and we all agree on an idea that in your experience you know is a bad idea, what would you do?”
Follow up question: “What would you do if after we listened to you, we decided to go in another direction?”
“Tell me about a time when you had unclear requirements , how did you handle it?” - gets back to ambiguity. There is a lot of that with startups.
We're building a CLOB-based DEX and this problem hits close, we've interviewed a lot of smart contract developers where AI makes it nearly impossible to assess real understanding. What we've shifted to: giving candidates a live matching engine with a subtle bug in the order execution logic and asking them to find and fix it. Prompting Claude gives you something that looks right but breaks under edge cases, partial fills, price-time priority violations, self-trade prevention. The candidates who actually understand the mechanics catch it. The ones steering AI without real knowledge submit a fix that breaks three other things. Your "AI steering" framing is exactly right. The real skill now is knowing when the output is plausible but wrong and that only comes from genuine domain knowledge.
> 1. Does testing a candidate's ability to "steer" and debug AI-generated code make more sense to you than traditional algorithms?
Testing the candidate's ability to "steer" agents seems to be like testing their ability to know the Java API or to recite SOLID by heart.
> 2. How are you currently preventing these "prompt-only" developers from slipping through your own interview loops?
We don't ask anymore leetcode. We keep the usual systems design interview in which usage of AI is not needed (or at least we don't allow it because in this kind of interview we are more interested in seeing how the candidate thinks and so on)
We have a new stage in our job interview, though: generic Q/A about the fundamental of software engineering/computer science. Again, we don't care anymore how candidates produce code. We care about what they know, and what they don't know. What's the scope of their knowledge, and when do they need to rely on AI to come up with an answer. Silly (non-real) example: "Can you write a program that detects if another program halts?". The people we want are the ones who would say something about the Halting Problem but also perhaps be practical and perhaps ask more questions about such a program requirements.
You get the point: we look for people with a good breadth of knowledge, who can communicate well and know their shit. Whether they can use tool x or y (including LLMs), comes for granted for such people
This is a fantastic perspective, thank you. You hit the nail on the head: the ultimate goal is testing fundamental engineering breadth and systems thinking, not tool usage.
I should definitely clarify my use of the word steering — I completely agree that testing prompt engineering is just the new API memorization, which is useless.
By steering, I mean putting them in a situation where the AI generates a plausible but architecturally flawed solution, and seeing if they have the fundamental knowledge to spot the BS, understand the scope of the problem, and fix it.
Basically, an automated way to test the exact critical thinking you mentioned.
I love your approach of dropping LeetCode for fundamentals Q/A and Systems Design. But out of curiosity, how do you scale that at the top of the funnel? Doing deep, manual 1-on-1 assessments gives the best signal by far, but doesn't that burn a massive amount of your senior engineers' time?
We binned any form of coding questions or take home task. Our preference now is to focus on architectural concepts and ability to apply those to a problem.
So, as happened last week, if I’m interviewing for an Elixir dev I’m going to be interested in your knowledge of the BEAM and how it’s features can be used to solve common architectural problems.
As a hiring manager here is what I do when I do interviews:
1. I don't do anything with code. No whiteboard. I usually don't even ask code related questions. Code literacy is a silent prerequisite. You wouldn't waste a lawyer's time asking if they could read books.
2. I talk to people. Soft skills are more important than tech skills. I want to see how they converse, their confidence, and their level of comfort.
3. I do ask technical questions and I ask them fast. It takes AI time to generate an answer. I am looking at speed of response. If the candidate needs to pause to think about a creative answer that's fine, because I am watching for that too. When they pause are they reading something from a screen or are they using their body language to convey they are using their imagination. I also pivot on the fly to other subjects between questions.
4. When I ask technical questions I try to make them as open ended as possible so that there is no single right answer. I want to know what the candidate would do in a given scenario. If I circle back on the same question 20 minutes later will I get the same answer?
5. I can usually make a determination about a candidate within 7-9 minutes of talking with them, but I schedule an hour to really make it more of a conversation and dive a bit deeper.
6. I don't really care about ASD in my current line of work, because in enterprise API management you will spend the majority of your time performing requirements discovery/analysis as opposed to writing code. So, it really is about the soft skills. If I were to go back to JavaScript I would absolutely look for ASD in the interview. Here is what I would look for:
6a. Frequency of first person pronouns. Does the candidate like to talk about themselves or is it more about a product or skill?
6b. Can the candidate measure things? Ask them to demonstrate some manner of technical performance analysis with numbers.
6c. Ask questions to inform of a bias. Does the candidate prefer objectivity when building something from scratch or must everything, even the most trivial of things, be familiar and comforting.
6d. On a scale between reckless and safety where the candidate land and how do they control for it?
As the candidate answers my questions I am really only just listening enough to know where to pivot for the next question in rudely short time. Most of what I am paying attention to is where the candidate's eyes go between talking to me versus answering a challenging question. I am also looking for the level of relaxation in the candidate's body and the stress in their voice.