You do not need to know what a vector database is to hire someone who does. You need to know what production looks like, and whether the person across the table has been there. Every question below is answerable in plain language, and every good answer has a shape you can recognize without a technical background. We use versions of these in our own vetting. They work because they ask about consequences, and consequences are hard to fake.
The seven questions
1. "Walk me through one AI system you shipped that real people used. What broke?"
A good answer names a specific system, a specific group of users, and at least one thing that went wrong after launch: a wrong answer that reached a customer, a cost that spiked, a source that went stale. The candidate describes what they did about it. A weak answer describes an impressive demo, a hackathon or a personal project with no users, or uses "we" so consistently that you cannot tell what they personally did. Follow up with "what was your part in that" until you can.
2. "How did you know it was working?"
A good answer mentions an evaluation set, a labeled sample, a dashboard, a number that was tracked over time. It sounds like measurement. A weak answer is "we tested it and it looked good" or "the users liked it". Anyone who shipped an AI feature without a way to measure quality either got lucky or does not know whether it worked. Both are risks.
3. "What did it cost to run, and how did that change your design?"
A good answer has a rough number and a story: they switched to a cheaper model for simple requests, added caching, shortened the context, or pushed back on a feature because the economics did not work at scale. A weak answer shrugs, or has never seen the bill. Cost is where AI projects meet finance, and an engineer who has never met finance will meet them on your project.
4. "Tell me about a time the model was confidently wrong. What did the user see?"
A good answer shows the candidate designed for failure: a confidence threshold, a "I could not find that" response, a handoff to a person, a way for users to flag bad answers. A weak answer treats wrong answers as rare or as the model's fault. They are neither. They are the normal operating condition, and the design has to assume them.
5. "How did you keep the system from showing someone data they should not see?"
A good answer talks about permissions travelling with the data and being checked at retrieval time, about testing it, and about what they did when the source system's permissions were messy, because they always are. A weak answer says the data was "all internal anyway" or that security was someone else's team. In a regulated business, this question alone can be disqualifying.
6. "What did you do when a business user disagreed with what the system should do?"
A good answer describes sitting with the user, watching how the work is actually done, and changing the product, or persuading the user with evidence and keeping the relationship either way. A weak answer frames users as the obstacle. The best AI engineers we know spend a surprising amount of time with the people who use what they build. It is the defining trait of the forward-deployed model, and it cannot be taught in the first month.
7. "If you joined us Monday, what would you want to see by Friday?"
A good answer asks for the data, the existing evaluation set if any, the users, and the thing that is currently broken. It wants to ship something small in the first two weeks. A weak answer wants to re-platform, pick a new framework, or spend the first month "understanding the landscape". Curiosity is good. Deferral dressed as diligence is not.
The two red flags
Framework fluency without production scars. Some candidates can name every tool released in the last eighteen months and describe none of the consequences of using them. The tell is that every story ends at "and then it worked". Real systems do not end there. Ask what broke, and keep asking.
Contempt for the business. Listen for how the candidate talks about users, stakeholders and the people who set requirements. Mild frustration is human. A pattern of "they did not understand" is a forecast of how they will talk about your team, and AI work fails more often on the business side than on the model side.
A simple scoring sheet
Score each of the seven questions from 0 to 2: zero for a weak answer, one for a plausible answer without specifics, two for a specific answer with a consequence and a personal role in it. Anyone below nine has not been to production. Anyone at twelve or above is worth a practical exercise. And regardless of the total, either red flag is a stop.
If you would rather not run this process yourself, it is roughly what our Production Gate does at greater depth, with a live build in the candidate's discipline and references checked against the stories. Either way, the principle is the same: hire for what has shipped, not for what has been read.