How to Assess AI Capabilities in Your Next Hire (Without Being Technical)

Every candidate now lists “AI proficient” on their resume. Half of your applicant pool claims they’re “familiar with ChatGPT.” But when you dig into what that actually means, you get wildly different answers—from someone who occasionally asks AI to polish an email, to someone who’s rebuilt entire workflows and cut their turnaround time by 60%.

For alternative investment firms, this gap matters. Hire someone who can genuinely leverage AI, and you’ve added a productivity multiplier to your team. Hire someone who’s just fluent in buzzwords, and you’re paying for a learning curve—or worse, introducing risk when they deploy AI without judgment in investor communications or compliance work.

Most funds don’t have technical interview infrastructure. You’re not going to start administering coding tests or AI proficiency exams. However, you can identify genuine capability in a standard behavioral interview if you know what to listen for.

Here’s the framework we’ve used to separate genuine AI competency from resume filler.

Question 1: “Walk me through a recent work task where you used AI. What was the result?”

This isn’t a theoretical question. You’re not asking what they could do with AI or what they’ve heard about it. You’re asking for a specific example from their actual work.

What you’re listening for: Specificity. A strong answer includes the problem context, their prompt strategy, how they validated the output, and the measurable outcome. You want to hear iteration—how they refined their approach when the first pass wasn’t quite right. And you want numbers: time saved, quality improvement, error reduction.

Red flags: The candidate provides vague answers, such as “I use ChatGPT for emails” or “I ask it for research help.” There is no mention of refinement or multiple attempts. They can’t articulate what changed as a result of using AI—just that they used it.

Question 2: “Tell me about a time AI gave you a wrong or incomplete answer. How did you handle it?”

This question distinguishes between those who use AI as a tool and those who treat it as magic.

What you’re listening for: Evidence of critical thinking and verification habits. You want to hear that they caught an error, understood why it happened, and adjusted their approach. This reveals their mental model of AI’s limitations, and whether they have the judgment to know when output needs human review.

Red flags: Comments like “That’s never happened to me,” or worse, “I always trust the output,” suggest either limited use or dangerous over-reliance. A blank stare can also be telling, indicating they haven’t used AI enough to encounter its failure modes.

Question 3: “If you joined us tomorrow, what’s one process you’d explore using AI to improve?”

This question tests initiative, process thinking, and, critically, whether the candidate understands where AI adds value versus where human judgment is non-negotiable.

What you’re listening for: They should identify a specific friction point in fund operations, articulate how AI could assist (not replace), and acknowledge what still requires human oversight. Strong candidates think in terms of workflow augmentation, not role elimination.

Red flags: Suggesting you automate investor relations or compliance processes without significant caveats. Proposing that AI replace roles rather than make existing roles more effective. Anything that suggests they haven’t thought about the judgment calls and relationship dynamics that make alternative investment operations work.

Why This Framework Works

It’s behavior-based, not technical. These questions fit naturally into the interview structures you already use; there’s no need for technical assessors or separate evaluation rounds.

It reveals applied experience, not theoretical knowledge. You’re not testing whether someone took an AI course or can explain how large language models work. You’re testing whether they’ve actually used these tools to solve real problems.

It uncovers risk management instincts. The second question, in particular, tells you whether this person will create liability or catch it before it becomes your problem.

And it works across all non-technical roles: operations, investor relations, talent acquisition, finance, and marketing. Anywhere someone’s daily work involves communication, analysis, or process management, this framework reveals whether they can utilize AI to amplify that work—or whether they’re merely familiar with the buzzwords.

The Bottom Line

The gap between AI curiosity and AI capability is a hiring challenge currently facing alternative investment firms. Every candidate has experimented with ChatGPT. Far fewer have developed the judgment to deploy it effectively in a regulated, high-stakes environment.

Firms that get this distinction right will build productivity advantages while their competitors hire “AI-familiar” candidates who can’t actually execute. This isn’t about technical assessments or AI expertise. It’s about operational judgment—the same judgment you evaluate for in every other aspect of hiring.

The difference is knowing what questions to ask.

Building a team that can leverage AI without introducing risk? Arootah’s advisors assist alternative investment firms in designing interview frameworks and competency models for the AI era. Sign up for The Talent Transformation Edge to receive early access to our upcoming AI Hiring Pivot webinar and practical frameworks, such as this one, delivered directly to your inbox.