Architect, ML Engineering
- Icertis
- 6 - 10 years
- Pune
- 3 days ago
- Email to a friend
- Report this job
Job Description
Key Responsibilities
AI Assurance & Strategy
- Define and own the end-to-end AI assurance strategy for AI products and copilots, including validation, benchmarking, and release sign-off criteria.
- Establish quality gates, metrics, and acceptance thresholds for LLM-based systems.
- Design assurance approaches covering accuracy, robustness, safety, regression, and reliability.
AI Evaluation & Benchmarking
- Design and execute AI evaluation (eval) frameworks for LLMs, prompts, agents, and copilots.
- Define benchmarking methodologies using curated datasets, synthetic data, and real-world scenarios.
- Analyze evaluation results to identify failure modes, drift, and improvement opportunities.
- Partner with Product, Legal, and AI teams on benchmarking, compliance, and defensibility of AI behavior.
Hands-on Engineering & Automation
- Build and maintain automation frameworks, scripts, and tools for AI validation and benchmarking.
- Implement repeatable pipelines for regression testing of AI features and releases.
- Contribute code to support evals, test harnesses, dataset management, and reporting/dashboarding etc.
- Enable scalable validation across multiple SKUs and releases.
AI Test Design & Execution
- Design test strategies for complex, high-risk AI features and workflows.
- Perform hands-on validation for critical AI capabilities and customer-impacting scenarios.
- Validate fixes and improvements to prevent regressions in production AI systems.
Collaboration & Enablement
- Collaborate with Applied AI Engineers and Data Scientists during design, development, and release.
- Support Product Managers in defining AI-specific acceptance criteria and release readiness.
- Work with Legal and Compliance teams on benchmarking, auditability, and assurance evidence .
- Mentor QA and validation engineers on AI-specific testing, evals, and best practices.
Documentation & Reporting
- Maintain clear documentation of validation approaches, eval methods, benchmarks, and outcomes.
- Prepare assurance summaries and release-readiness reports for stakeholders.
- Continuously improve standards, tools, and processes based on learnings and industry trends.
Key Responsibilities
AI Assurance & Strategy
- Define and own the end-to-end AI assurance strategy for AI products and copilots, including validation, benchmarking, and release sign-off criteria.
- Establish quality gates, metrics, and acceptance thresholds for LLM-based systems.
- Design assurance approaches covering accuracy, robustness, safety, regression, and reliability.
AI Evaluation & Benchmarking
- Design and execute AI evaluation (eval) frameworks for LLMs, prompts, agents, and copilots.
- Define benchmarking methodologies using curated datasets, synthetic data, and real-world scenarios.
- Analyze evaluation results to identify failure modes, drift, and improvement opportunities.
- Partner with Product, Legal, and AI teams on benchmarking, compliance, and defensibility of AI behavior.
Hands-on Engineering & Automation
- Build and maintain automation frameworks, scripts, and tools for AI validation and benchmarking.
- Implement repeatable pipelines for regression testing of AI features and releases.
- Contribute code to support evals, test harnesses, dataset management, and reporting/dashboarding etc.
- Enable scalable validation across multiple SKUs and releases.
AI Test Design & Execution
- Design test strategies for complex, high-risk AI features and workflows.
- Perform hands-on validation for critical AI capabilities and customer-impacting scenarios.
- Validate fixes and improvements to prevent regressions in production AI systems.
Collaboration & Enablement
- Collaborate with Applied AI Engineers and Data Scientists during design, development, and release.
- Support Product Managers in defining AI-specific acceptance criteria and release readiness.
- Work with Legal and Compliance teams on benchmarking, auditability, and assurance evidence .
- Mentor QA and validation engineers on AI-specific testing, evals, and best practices.
Documentation & Reporting
- Maintain clear documentation of validation approaches, eval methods, benchmarks, and outcomes.
- Prepare assurance summaries and release-readiness reports for stakeholders.
- Continuously improve standards, tools, and processes based on learnings and industry trends.
Required Qualifications
- Bachelor s degree in Computer Science , Engineering, Data Science, or a related field.
- 6 10 years of experience in software quality, AI engineering, validation, or related roles.
- Strong understanding of AI/ML concepts , especially LLMs and AI copilots .
- Proven experience with AI validation, benchmarking, or evaluation frameworks .
- Hands-on experience writing scripts or code for testing, automation, or evaluation.
- Strong foundation in software quality principles and test strategy.
- Ability to work in fast-paced, agile product environments.
- Excellent analytical, documentation, and communication skills.
Preferred Qualifications
- Experience with LLM eval frameworks , prompt testing, or agent validation.
- Familiarity with automation tools, CI/CD integration, and test orchestration.
- Experience creating and managing sample or synthetic datasets for AI testing.
- Exposure to AI assurance, responsible AI, or compliance-oriented validation.
- Prior experience in architect-level or cross-team ownership roles.
Job Classification
Industry: IT Services & ConsultingFunctional Area / Department: Engineering - Software & QARole Category: Software DevelopmentRole: Technical ArchitectEmployement Type: Full timeContact Details:
Company: IcertisLocation(s): Pune+ View Contact
Keyskills: Computer science Automation Assurance Test strategy Contract management Analytical Test design Agile Regression testing software quality
Fraud Alert to job seekers!
₹ Not Disclosed
Similar positions
Icertis
This is a manufacturing, Trading and Retail Sales Company.\r\nProducts: Door, Chokhat, Plywood, Board etc.
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in