Your browser does not support javascript! Please enable it, otherwise web will not work for you.

Architect, ML Engineering @ Icertis

Home > Software Development

 Architect, ML Engineering

Job Description

Key Responsibilities
AI Assurance & Strategy
  • Define and own the end-to-end AI assurance strategy for AI products and copilots, including validation, benchmarking, and release sign-off criteria.
  • Establish quality gates, metrics, and acceptance thresholds for LLM-based systems.
  • Design assurance approaches covering accuracy, robustness, safety, regression, and reliability.
AI Evaluation & Benchmarking
  • Design and execute AI evaluation (eval) frameworks for LLMs, prompts, agents, and copilots.
  • Define benchmarking methodologies using curated datasets, synthetic data, and real-world scenarios.
  • Analyze evaluation results to identify failure modes, drift, and improvement opportunities.
  • Partner with Product, Legal, and AI teams on benchmarking, compliance, and defensibility of AI behavior.
Hands-on Engineering & Automation
  • Build and maintain automation frameworks, scripts, and tools for AI validation and benchmarking.
  • Implement repeatable pipelines for regression testing of AI features and releases.
  • Contribute code to support evals, test harnesses, dataset management, and reporting/dashboarding etc.
  • Enable scalable validation across multiple SKUs and releases.
AI Test Design & Execution
  • Design test strategies for complex, high-risk AI features and workflows.
  • Perform hands-on validation for critical AI capabilities and customer-impacting scenarios.
  • Validate fixes and improvements to prevent regressions in production AI systems.
Collaboration & Enablement
  • Collaborate with Applied AI Engineers and Data Scientists during design, development, and release.
  • Support Product Managers in defining AI-specific acceptance criteria and release readiness.
  • Work with Legal and Compliance teams on benchmarking, auditability, and assurance evidence .
  • Mentor QA and validation engineers on AI-specific testing, evals, and best practices.
Documentation & Reporting
  • Maintain clear documentation of validation approaches, eval methods, benchmarks, and outcomes.
  • Prepare assurance summaries and release-readiness reports for stakeholders.
  • Continuously improve standards, tools, and processes based on learnings and industry trends.
Key Responsibilities
AI Assurance & Strategy
  • Define and own the end-to-end AI assurance strategy for AI products and copilots, including validation, benchmarking, and release sign-off criteria.
  • Establish quality gates, metrics, and acceptance thresholds for LLM-based systems.
  • Design assurance approaches covering accuracy, robustness, safety, regression, and reliability.
AI Evaluation & Benchmarking
  • Design and execute AI evaluation (eval) frameworks for LLMs, prompts, agents, and copilots.
  • Define benchmarking methodologies using curated datasets, synthetic data, and real-world scenarios.
  • Analyze evaluation results to identify failure modes, drift, and improvement opportunities.
  • Partner with Product, Legal, and AI teams on benchmarking, compliance, and defensibility of AI behavior.
Hands-on Engineering & Automation
  • Build and maintain automation frameworks, scripts, and tools for AI validation and benchmarking.
  • Implement repeatable pipelines for regression testing of AI features and releases.
  • Contribute code to support evals, test harnesses, dataset management, and reporting/dashboarding etc.
  • Enable scalable validation across multiple SKUs and releases.
AI Test Design & Execution
  • Design test strategies for complex, high-risk AI features and workflows.
  • Perform hands-on validation for critical AI capabilities and customer-impacting scenarios.
  • Validate fixes and improvements to prevent regressions in production AI systems.
Collaboration & Enablement
  • Collaborate with Applied AI Engineers and Data Scientists during design, development, and release.
  • Support Product Managers in defining AI-specific acceptance criteria and release readiness.
  • Work with Legal and Compliance teams on benchmarking, auditability, and assurance evidence .
  • Mentor QA and validation engineers on AI-specific testing, evals, and best practices.
Documentation & Reporting
  • Maintain clear documentation of validation approaches, eval methods, benchmarks, and outcomes.
  • Prepare assurance summaries and release-readiness reports for stakeholders.
  • Continuously improve standards, tools, and processes based on learnings and industry trends.
Required Qualifications
  • Bachelor s degree in Computer Science , Engineering, Data Science, or a related field.
  • 6 10 years of experience in software quality, AI engineering, validation, or related roles.
  • Strong understanding of AI/ML concepts , especially LLMs and AI copilots .
  • Proven experience with AI validation, benchmarking, or evaluation frameworks .
  • Hands-on experience writing scripts or code for testing, automation, or evaluation.
  • Strong foundation in software quality principles and test strategy.
  • Ability to work in fast-paced, agile product environments.
  • Excellent analytical, documentation, and communication skills.
Preferred Qualifications
  • Experience with LLM eval frameworks , prompt testing, or agent validation.
  • Familiarity with automation tools, CI/CD integration, and test orchestration.
  • Experience creating and managing sample or synthetic datasets for AI testing.
  • Exposure to AI assurance, responsible AI, or compliance-oriented validation.
  • Prior experience in architect-level or cross-team ownership roles.

Job Classification

Industry: IT Services & Consulting
Functional Area / Department: Engineering - Software & QA
Role Category: Software Development
Role: Technical Architect
Employement Type: Full time

Contact Details:

Company: Icertis
Location(s): Pune

+ View Contactajax loader


Keyskills:   Computer science Automation Assurance Test strategy Contract management Analytical Test design Agile Regression testing software quality

 Fraud Alert to job seekers!

₹ Not Disclosed

Similar positions

Senior Analyst 1 Software Engineering

  • DXC Technology
  • 2 - 5 years
  • Noida, Gurugram
  • 3 days ago
₹ Not Disclosed

Senior Analyst 1 Software Engineering

  • DXC Technology
  • 2 - 5 years
  • Noida, Gurugram
  • 3 days ago
₹ Not Disclosed

Sr Analyst II Software Engineering

  • DXC Technology
  • 1 - 4 years
  • Noida, Gurugram
  • 3 days ago
₹ Not Disclosed

Sr Analyst I Software Engineering

  • DXC Technology
  • 2 - 5 years
  • Noida, Gurugram
  • 3 days ago
₹ Not Disclosed

Icertis

This is a manufacturing, Trading and Retail Sales Company.\r\nProducts: Door, Chokhat, Plywood, Board etc.