Azure Data Engineer (Lead)
- Infogain
- 9 - 12 years
- Bengaluru
- 2 months ago
- Email to a friend
- Report this job
Job Description
Key Responsibilities
1. Data Quality Framework Design & Leadership
- Define and implement enterprise-wide data quality frameworks and governance standards.
- Architect automated DQ pipelines using Databricks (Delta Lake) , PySpark , and Ataccama ONE .
- Design DQ monitoring architecture profiling, lineage integration, and alerting mechanisms.
- Establish KPIs and DQ scorecards to measure and communicate data trust metrics across domains.
2. Advanced Data Quality Development & Automation
- Build and optimize complex validation, reconciliation, and anomaly detection workflows using PySpark and Python.
- Implement rule-based and ML-based DQ checks, leveraging Ataccama workflows and open-source frameworks.
- Integrate DQ rules into CI/CD and orchestration platforms (Airflow, ADF, or Databricks Workflows).
- Partner with data engineers to embed DQ checks into ingestion and transformation pipelines.
3. Root Cause Analysis & Continuous Improvement
- Lead root-cause investigations for recurring DQ issues and drive long-term remediation solutions.
- Create and enforce best practices for rule versioning, DQ exception handling, and reporting.
- Own the playbook for DQ incident response and continuous optimization.
4. Stakeholder Management & Governance
- Act as the primary liaison between business data owners, IT, and governance teams.
- Translate business DQ requirements into technical implementation strategies.
- Drive executive-level reporting on DQ KPIs, SLAs, and issue trends.
- Contribute to metadata management, lineage documentation, and master data alignment.
5. Mentorship & Leadership
- Guide junior analysts and data engineers in developing robust DQ solutions.
- Lead cross-functional squads to implement new data quality capabilities or upgrades.
- Contribute to capability uplift training peers on DQ best practices, tools, and technologies.
Core Technical Skills
Category
Tools / Skills
Data Engineering & Quality
Databricks (Delta Lake), PySpark, SQL, Python
DQ Platforms
Ataccama ONE / Studio (rule authoring, workflow automation, profiling)
Orchestration & CI/CD
Apache Airflow, Azure Data Factory, Databricks Workflows, GitHub Actions
Data Warehouses
Databricks Lakehouse
Cloud & Infrastructure
Azure (preferred), AWS, or GCP; familiarity with Terraform or IaC concepts
Version Control / CI-CD
Git, GitHub Actions, Azure DevOps
Metadata & Governance
Collibra, Alation, Ataccama Catalog, OpenLineage
Monitoring & Observability
Grafana, Datadog, Prometheus for DQ metrics and alerts
Qualifications & Experience
- Bachelor s or Master s in Computer Science, Information Systems, Statistics, or related field.
- 9 12 years of experience in data quality, data engineering, or governance-focused roles.
- Proven experience designing and deploying enterprise DQ frameworks and automated checks.
- Strong expertise in Databricks , PySpark , and Ataccama for data profiling and rule execution.
- Advanced proficiency in SQL and Python for large-scale data analysis and validation.
- Solid understanding of data models, lineage, reconciliation, and governance frameworks
- Experience integrating DQ checks into CI/CD pipelines and orchestrated data flows.
Soft Skills & Leadership Attributes
- Strong analytical thinking and systems-level problem solving.
- Excellent communication and presentation skills for senior stakeholders.
- Ability to balance detail orientation with strategic vision.
- Influencer with a proactive, ownership-driven mindset.
- Comfortable leading cross-functional teams in fast-paced, cloud-native environments.
Preferred / Nice to Have
- Experience in financial, manufacturing, or large enterprise data environments.
- Familiarity with MDM , reference data , and data stewardship processes.
- Exposure to machine learning-driven anomaly detection or predictive data quality .
- Certifications: Databricks, Ataccama, or Cloud Data Engineering certifications (Azure/AWS).
Success Indicators
- Increased DQ rule coverage and automation across key data domains.
- Reduced manual DQ exceptions and faster remediation cycle times.
- Measurable improvement in data trust metrics and reporting accuracy.
- High stakeholder satisfaction with data availability and reliability.
- 8-11 Years
- Primary Skill: Data Engineering
- Sub Skill(s): Data Engineering
- Additional Skill(s): Python, databricks, SQL, Azure Data Factory
Job Classification
Industry: IT Services & ConsultingFunctional Area / Department: Engineering - Software & QARole Category: Software DevelopmentRole: Data EngineerEmployement Type: Full timeContact Details:
Company: InfogainLocation(s): Bengaluru+ View Contact
Keyskills: Computer science Telecom Data analysis Automation Managed services Analytical Reconciliation Healthcare Data quality Monitoring
Fraud Alert to job seekers!
₹ Not Disclosed
Similar positions
Infogain
A Silicon-Valley headquartered company, Infogain is a global business oriented IT consulting provider of front-end, customer-facing technologies, processes and applications, leading to a more efficient and streamlined customer experience. We want our clients€™ interactions with their cus...
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in