Your job posting says "agentic AI engineer." You're getting 200+ applications. Your ATS shows dozens of candidates with "AI Engineer" in their title. Your internal screeners are overwhelmed, and nobody on your team has actually shipped a production agentic system before.
This is the situation every AI hiring manager faces right now. The talent category is new, the terminology is inconsistent, and the people who know how to evaluate it are rare. This checklist is designed to change that. Use it to cut through the noise and find candidates who've actually done the work.
Why Standard AI Hiring Criteria Fail
Most companies evaluate AI engineers the same way they evaluate any engineer: languages, frameworks, years of experience, notable companies, degree path. For conventional software roles, this works. For agentic engineering, it falls apart entirely.
The problem is that "AI engineer" now covers two completely different disciplines:
- ML engineering: Training models, optimizing inference pipelines, deploying GPU clusters. This is what most screeners are calibrated for.
- Agentic engineering: Building AI systems that take actions autonomously, call tools, maintain state across sessions, and operate in production without constant oversight. This is what your job posting actually needs.
A senior ML engineer who built recommendation systems at scale for five years might be completely lost trying to design a multi-agent pipeline with reliable tool invocation. Meanwhile, a mid-level backend engineer who's been obsessively building with Claude Code, LangGraph, and MCP for 18 months will deliver far more value on your agentic product. Your hiring criteria need to account for this split.
The Evaluation Checklist
Run every candidate through this checklist before making an offer. Each item filters for something specific.
- MCP fluency: Can the candidate describe Model Context Protocol in concrete terms? Do they know the difference between an MCP server and a tool definition? Have they built or integrated with one? MCP isn't a feature request on a roadmap somewhere, it's becoming the standard for how AI systems talk to external tools. Engineers who understand it architect better.
- Tool-calling architecture: Ask them to walk through how they'd design a system where an AI agent calls your internal API reliably. Strong candidates talk about retry logic, output validation, graceful degradation, and monitoring. Weak candidates hand-wave it. Tool calling is the highest-leverage skill in agentic engineering. Everything else breaks if this doesn't work.
- Context window management: Have they encountered context overflow in a real production system? What was it like? How did they handle it? What patterns do they use to keep context lean? Context management is the most common failure mode in agentic systems. Candidates who've never hit the ceiling can't design around it.
- Multi-agent orchestration: Can they describe a scenario where they'd use multiple agents vs. a single agent with better prompting? Do they understand handoff patterns, shared state management, and the distributed systems challenges that come with multi-agent architectures? The best agentic engineers think about this at the design stage, not as an afterthought.
- Failure mode debugging: Show them a log snippet from a broken agentic system and ask them to diagnose it. Engineers who've operated in production immediately recognize the patterns: context overflow, tool hallucination, prompt drift, infinite loops. Engineers who haven't see it as a black box. This is the single most revealing evaluation technique available.
- Production shipped: Not a weekend hackathon. Not a tutorial project. A production system that serves real users. Ask them to describe what broke in the first week. If they can't answer specifically, the system probably didn't exist or wasn't actually in production.
- Claude Code as development environment: Ask how they use Claude Code day-to-day. Do they think of it as a pair programmer, an architectural sounding board, a code reviewer? Candidates who've truly internalized it as a core part of their workflow iterate faster and architect differently. This shows up in everything they produce.
- "AI Engineer" from a company whose AI product was actually ML pipelines 90% of the market. Filter these carefully.
- LangChain listed as a skill without production context LangChain is a learning tool. In production, it creates as many problems as it solves. Ambiguous usage is a red flag.
- No GitHub or portfolio evidence of agentic work Real agentic engineers build constantly. The absence of any public artifacts means either they don't work on side projects or they're protecting IP that doesn't exist.
- Generic "LLM fine-tuning" without downstream agentic application Fine-tuning is rarely the right answer for agentic systems. Engineers who reach for fine-tuning by default haven't internalized the agentic design space.
- TensorFlow/PyTorch as primary AI skills These skills matter for ML infrastructure. For agentic engineering, they're nearly irrelevant. A candidate whose AI skills are entirely PyTorch-based is a conventional ML engineer trying to catch a wave.
Interview Questions for Agentic Systems
These questions are designed to be answered in 10-15 minutes. Use them in a live technical screen or async recorded response.
Walk me through how you'd build an agent that searches your internal knowledge base and writes a summary for a customer support ticket. What's the architecture?
Look for: tool design clarity, how they handle the search-to-summary handoff, whether they think about partial failures (what if search returns nothing?), and whether they know when this is a multi-agent problem vs. a single-agent problem.
Your agent loop keeps running longer than expected and the model starts producing weird outputs. Walk me through your debugging approach.
Strong candidates: ask about token counts, context state, tool output quality, whether there's a runaway loop. Weak candidates: suggest adding more instructions or restarting.
What does "context window management" mean to you in practice? Give me an example of something you did in production to keep an agent's context lean.
This is a diagnostic question. Candidates who have actually worked with long-running agents have a specific answer. Candidates with theory but no production experience go vague fast.
Why do traditional staffing firms have trouble finding agentic AI engineers?
This isn't a trivia question. It's a signal test. Candidates who understand the problem deeply can articulate why the supply and evaluation signals are misaligned. Candidates who just want the job give you a generic "AI is new" answer.
If a candidate can't answer three of these four questions specifically, they're not ready for a production agentic role. Move on.
Why Traditional Staffing Firms Can't Help
Toptal, Turing, and the major staffing platforms were designed for a different era. They screen by credentials: top-tier companies, elite schools, verified previous work. For conventional software roles, this produces good signal. For agentic engineering, it's almost useless.
The agentic engineering talent pool doesn't self-select into traditional staffing funnels. The best engineers are building things on their own, contributing to open-source agentic frameworks, shipping products. They're not filling out onboarding questionnaires on staffing platforms at the same rate as engineers with conventional career paths.
Beyond supply, the evaluation problem is structural. A screener on a traditional staffing platform knows how to assess React proficiency or backend architecture. They don't know how to evaluate MCP fluency or tool-calling architecture because those skills are too new to have generated a shared rubric. You're better off using your own checklist than relying on their screening process.
The fastest path to a qualified agentic engineer right now is a targeted sourcing process. Before you source, know exactly what you are evaluating — our Claude Code vs Cursor vs Copilot guide explains the full spectrum of AI coding tools and what agentic fluency actually means. that reaches directly into AI-native communities, followed by an evaluation built specifically for the discipline. That's what Minimalistech does. If you're trying to build an agentic team and hitting a wall on sourcing or screening, we can help.
Building an agentic AI team? Let's talk.
Minimalistech sources and vets senior agentic engineers in 3-5 days. We evaluate against criteria that actually predict whether they'll ship.
Tell us what you need