Job Description
AI Evaluation Engineer
Location: Pune, India
Seniority: Mid-level (3-6 years)
Purpose: Execute comprehensive evaluations of FNZ's AI agents across the six-pillar framework, working as a generalist while developing specialist expertise in 1-2 pillars.
Key Responsibilities:
Design and conduct evaluations covering Task Performance, Safety, Efficiency, Groundedness, Robustness, and Suitability
Create "golden sets" of test examples representing expert judgment on desired agent behaviour
Develop evaluation rubrics and scoring criteria aligned to FNZ Evaluation Framework principles
Build comprehensive test suites covering happy paths, edge cases, and adversarial inputs
Evaluate multi-step agentic workflows: planning, tool selection, execution, error handling
Assess agent groundedness: verify outputs against knowledge bases, detect hallucinations
Document findings with clear evidence; collaborate with development teams on remediation
Contribute to building automated evaluation platform and CI/CD integration
Skills and Experience:
3-6 years in software testing, QA engineering, AI/ML development, or data science
Hands-on test automation skills, experience with ML frameworks highly valuable
Practical experience evaluating LLM applications, RAG systems, or AI agents
Understanding of prompt engineering, retrieval-augmented generation, and agent architectures
Analytical mindset to decompose complex agent behaviours and identify failure modes
Strong documentation and presentation skill
About FNZ
FNZ is committed to opening up wealth so that everyone, everywhere can invest in their future on their terms. We know the foundation to do that already exists in the wealth management industry, but complexity holds firms back.
We created wealth’s growth platform to help. We provide a global, end-to-end wealth management platform that integrates modern technology with business and investment operations. All in a regulated financial institution.
We partner with the world’s leading financial institutions, with over US$2.4 trillion in assets on platform (AoP).
Together with our clients, we empower nearly 30 million people across all wealth segments to invest in their future.