Applied AI Engineer, Evals

About the role

Every Kairon agent passes three gates before it touches real customer data: offline evals, shadow mode and a canary. The quality of those gates decides whether our customers can trust what they ship. You’ll work on the systems that make them rigorous, fast and hard to fool.

You’ll join the Evals team, which sits inside Research and works closely with our runtime and product teams. The work is a mix of applied research and solid engineering. Ideas only count once they run on real customer suites.

What you’ll do

• Build model-based graders that agree with careful human reviewers, and measure how often they do.
• Design fault injection for tool calls, such as timeouts, partial records and stale data, so suites test recovery, not just the happy path.
• Develop methods to grow small hand-picked suites into large, representative ones without bias creeping in.
• Create metrics that predict production incidents better than a single pass rate.
• Publish what you learn on our blog and in customer-facing guides.

What you’ll bring

• 4+ years of software engineering, with at least two working on ML systems in production.
• Hands-on experience building evaluations for large language models.
• Strong Python skills and sound statistical instincts.
• The habit of reading raw outputs, not just dashboards.
• Clear writing. Our eval decisions are made in short docs with evidence.

Nice to have

• Research experience in evaluation, reliability or human-AI agreement.
• Background in testing infrastructure or QA at scale.
• Experience with agentic systems that use tools.

Benefits

Competitive salary and meaningful equity, 30 paid days off, full health cover, 20 weeks of parental leave, a $2,000 yearly learning budget and a dedicated compute budget for experiments. Read about our values and team.