Generative Ai Testing & Prompt Engineer (ai Evaluation, Bma)
- Synechron
- 3 - 5 years
- Bengaluru
- 6 months ago
- Email to a friend
- Report this job
Job Description
Job Summary
Synechron is seeking a proactive and skilled Generative AI Testing & Prompt Engineer to join our innovative technology team. In this role, you will design and execute prompt strategies, develop automation workflows, and evaluate AI outputs to ensure high-quality, unbiased, and reliable generative models. Your expertise will help improve AI performance, safety, and ethical standards, enabling the organization to leverage cutting-edge AI capabilities effectively and responsibly.
Software Requirements
- Required:
- Python (including libraries such as NumPy, Pandas, and testing frameworks)
- Bash or shell scripting for automation workflows
- Version control tools (Git)
- AI evaluation metrics tools and frameworks for measuring quality, bias, and safety
- Preferred:
- Experience with AI/ML frameworks such as TensorFlow or PyTorch
- CI/CD automation tools for seamless testing and deployment workflows
- Data analysis tools and platforms for metrics development
Overall Responsibilities
- Develop and refine prompts to evaluate the capabilities, accuracy, creativity, and safety of Generative AI models, including NLP and multimodal models.
- Design comprehensive testing strategies across diverse scenarios using prompt variations to assess performance, bias, and ethical considerations.
- Automate testing workflows to streamline data collection, prompt execution, output analysis, and reporting.
- Analyze and interpret AI outputs, providing insights for improvements and bias mitigation.
- Collaborate closely with AI data scientists and developers to tune prompt techniques and enhance model behavior through automation and feedback.
- Document prompt scripts, testing procedures, automation workflows, and test results for transparency and reproducibility.
- Continuously refine prompt engineering and testing methodologies based on latest AI advancements and industry best practices.
- Stay informed about new trends, tools, and ethical considerations related to generative AI, prompt engineering, and automation.
Technical Skills (By Category)
- Programming Languages:
- Required: Python (advanced scripting and automation capabilities)
- Preferred: Bash, shell scripting, or other automation languages
- Databases/Data Management:
- Basic understanding of data storage and querying for managing testing datasets
- Experience with data labeling and quality control processes
- Cloud Technologies:
- Preferred: Basic familiarity with cloud platforms (AWS, Azure, GCP) for AI deployment and testing
- Frameworks and Libraries:
- Required: Generative AI evaluation tools, scripting frameworks for automation, prompt management tools
- Preferred: TensorFlow, PyTorch, or similar AI/ML libraries
- Development Tools and Methodologies:
- Required: Version control (Git), scripting for automation, reporting tools
- Preferred: CI/CD pipelines for automated testing and deployment in cloud or on-prem environments
- Security Protocols:
- Awareness of data privacy, security standards, and bias mitigation techniques
Experience Requirements
- 3-5 years of professional experience in testing generative AI models, including prompt writing and optimization.
- Proven experience designing prompt strategies for NLP and multimodal models.
- Hands-on experience automating evaluation workflows for AI outputs, bias detection, and safety assessment.
- Familiarity with model evaluation metrics, including relevance, coherence, bias, and safety.
- Industry experience in AI development, testing, or related research preferred; alternative experience with related automation and data analysis roles may be considered.
Day-to-Day Activities
- Develop, test, and optimize user prompts for AI model evaluations across multiple scenarios.
- Automate processes for prompt execution, data collection, and output assessment.
- Monitor and analyze AI model responses for quality, relevance, bias, and ethical compliance.
- Collaborate with data scientists, AI developers, and product teams to implement prompt tuning and automation improvements.
- Document testing strategies, automation scripts, and test outcomes thoroughly.
- Provide actionable feedback to improve model behavior and address bias or safety issues.
- Review industry trends and incorporate new testing tools or methodologies to advance testing capabilities.
Qualifications
- Bachelors degree in Computer Science, Data Science, AI, or related field.
- Relevant certifications in AI, data analysis, or automation practices are a plus.
- Demonstrated commitment to continuous learning of AI ethics, bias mitigation, and prompt engineering.
Professional Competencies
- Critical thinking and analytical skills to interpret complex AI outputs and identify issues.
- Problem-solving abilities to develop effective prompts, troubleshoot automation workflows, and improve testing strategies.
- Strong communication skills for documenting processes and collaborating with multidisciplinary teams.
- Ability to adapt rapidly to evolving AI technologies and industry standards.
- Interpersonal skills for stakeholder engagement and feedback provision.
- Effective time management to handle multiple test scenarios and documentation tasks efficiently.
Job Classification
Industry: IT Services & ConsultingFunctional Area / Department: Engineering - Software & QARole Category: Quality Assurance and TestingRole: Automation Test EngineerEmployement Type: Full timeContact Details:
Company: SynechronLocation(s): Bengaluru
Keyskills: continuous integration python version control ci/cd tensorflow data management test scenarios software testing automation testing microsoft azure dbms numpy artificial intelligence pandas git gcp pytorch bash shell scripting aws ci cd pipeline
Similar positions
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in