Senior AI Eval & Safety Engineer
Discovered MENA
Role:Senior AI Evaluation & Safety Engineer
Location: Abu Dhabi, UAE
Contract Duration: 12 Months
Employment Type: Contract
Industry: Financial Services / Banking
About the Opportunity
We are partnering with a leading financial institution in the UAE that is investing heavily in Artificial Intelligence and Generative AI technologies.
We are looking for an experienced Senior AI Evaluation & Safety Engineer to join on a 12-month contract, supporting the development, evaluation and safe deployment of enterprise-scale AI applications.
Key Responsibilities
LLM Evaluation & Testing
Design, develop and maintain automated evaluation frameworks for LLM-powered applications.
Evaluate AI systems for accuracy, relevance, faithfulness, hallucinations, consistency and reliability.
Develop golden datasets, evaluation benchmarks and automated testing pipelines.
Implement LLM-as-a-Judge methodologies and automated scoring mechanisms.
Define evaluation metrics, acceptance criteria and quality standards for production AI applications.
Integrate evaluation and regression testing into AI development workflows.
RAG & Agentic AI Evaluation
Develop evaluation frameworks for Retrieval-Augmented Generation (RAG) applications.
Assess retrieval accuracy, context relevance, answer faithfulness and response quality.
Evaluate AI agents, including tool calling, multi-step workflows, task completion and reliability.
Identify failure scenarios and recommend improvements to AI application performance.
Collaborate with AI engineering teams to optimise system accuracy and reliability.
AI Safety, Security & Red Teaming
Design and execute AI safety tests to identify vulnerabilities and unsafe model behaviour.
Conduct adversarial testing, including prompt injection, jailbreak attempts and manipulation of AI responses.
Evaluate risks relating to sensitive data exposure, confidential information leakage and unauthorised actions.
Develop and implement AI guardrails, safety controls and validation mechanisms.
Support AI red teaming exercises and security assessments.
Ensure AI applications align with responsible AI and enterprise security standards.
Evaluation Frameworks & Automation
Build scalable evaluation pipelines using Python and relevant LLM evaluation frameworks.
Work with tools such as RAGAS, DeepEval, LangSmith, TruLens, promptfoo or equivalent technologies.
Develop automated testing suites to evaluate AI applications across different models and configurations.
Integrate evaluation processes into CI/CD pipelines.
Monitor production AI applications for performance degradation, safety issues and unexpected behaviour.
AI Governance & Risk Management
Support the implementation of responsible AI standards and evaluation controls.
Ensure AI applications meet internal security, privacy and regulatory requirements.
Document evaluation methodologies, test results, identified risks and remediation actions.
Collaborate with cybersecurity, risk, compliance and governance teams.
Contribute to the development of AI evaluation and safety best practices.
Required Skills & Experience
Bachelor's or Master's degree in Computer Science, Artificial Intelligence, Machine Learning, Software Engineering or a related field.
Strong professional experience in AI/ML engineering, applied AI or software engineering.
Advanced Python programming skills, including experience developing automated evaluation and testing frameworks.
Hands-on experience building or evaluating production Generative AI applications.
Strong understanding of LLMs, RAG architectures, embeddings, vector databases and agentic AI systems.
Practical experience implementing LLM evaluation metrics, including hallucination detection, faithfulness, relevance and response quality.
Experience with evaluation frameworks such as RAGAS, DeepEval, LangSmith, TruLens or equivalent.
Experience with AI safety testing, adversarial testing, prompt injection detection or LLM red teaming.
Understanding of LLM security vulnerabilities, guardrails and responsible AI principles.
Experience integrating automated testing and evaluation into engineering workflows.
Strong analytical, debugging and problem-solving skills.
Desirable Experience
Previous experience within banking, financial services, fintech or another highly regulated industry.
Experience with Azure OpenAI, Azure AI Foundry, AWS Bedrock or Google Cloud AI services.
Knowledge of LangChain, LangGraph, LlamaIndex or similar GenAI frameworks.
Familiarity with NIST AI RMF and OWASP Top 10 for LLM Applications.
Experience with LLMOps, model monitoring and AI observability.
Understanding of data privacy, information security and regulatory compliance.
Experience evaluating enterprise AI agents and multi-agent systems.
We are looking for a technically strong AI Engineer who has experience beyond simply building LLM applications and can demonstrate practical expertise in evaluating, testing and securing AI systems for production environments.
The ideal candidate will have personally developed automated evaluation frameworks, implemented AI safety testing and worked closely with engineering teams to improve the performance, reliability and security of Generative AI applications.
Experience delivering AI solutions within enterprise or regulated environments would be particularly valuable.
Contract Details
Contract Duration: 12 months
Location: Abu Dhabi, UAE
Industry: Banking / Financial Services
Working Arrangements: Onsite
This is an opportunity to work on advanced enterprise AI initiatives within a major financial institution, contributing to the development of secure, reliable and responsible Generative AI solutions.
