Website Simon James IT Ltd
Specialising in niche IT roles, our aim is to power the careers of IT professionals and provide customers with a professional, tailored, and personal service.
Principal Test Engineer
AI Core Services (Microsoft Azure / .Net)
We’re seeking a highly skilled Principal Test Engineer to lead the validation of cloud-native AI services and APIs. This role focuses on designing automated testing frameworks and AI-specific evaluation tools to ensure enterprise-grade reliability, security, and ethical compliance.
About the Role
As a Principal Test Engineer, you will be responsible for leading the validation efforts for Azure-based services, ensuring they meet the highest standards of quality and compliance.
Responsibilities
- Lead functional, performance, and security testing of Azure-based services
- Build automated test suites for APIs, service components, and AI pipelines
- Develop tools to assess AI outputs for accuracy, consistency, and safety
- Collaborate with engineering and data science teams to define quality metrics and observability
- Ensure services meet AI regulatory standards (e.g., NIST AI RMF, EU AI Act) with traceability and fairness
- Conduct threat modelling and validate exposed APIs for security
- Provide early feedback to improve design and reduce defects
- Mentor junior testers and foster a culture of innovation
Qualifications
- Proven experience in testing cloud-native applications using Microsoft/.NET technologies
- Strong background in automated testing frameworks and QA leadership
- Deep understanding of AI systems and regulatory compliance
Required Skills
- AI evaluation: LLM scoring, hallucination detection, prompt engineering, RAG, model orchestration
- AI safety: bias detection, explainability, ethical AI practices
- Cloud & DevOps: Azure DevOps, CI/CD, containerized environments
- Testing: API validation, performance profiling, OWASP security testing
- Agile collaboration across global R&D teams
- Experience with advanced AI testing methodologies