Azure Data Engineer
- Infogain
- 10 - 14 years
- Noida, Gurugram
- 4 months ago
- Email to a friend
- Report this job
Job Description
- Lead design and execution of Dataproc Databricks PySpark migration roadmap.
- Define modernization strategy , including data ingestion, transformation, orchestration, and governance.
- Architect scalable Delta Lake and Unity Catalog -based solutions.
- Manage and guide teams on code conversion, dependency mapping, and data validation.
- Collaborate with platform, infra, and DevOps teams to optimize compute costs and performance.
- Own the automation & GenAI acceleration layer , integrating code parsers, lineage tools, and validation utilities.
- Conduct performance benchmarking, cost optimization, and platform tuning (Photon, Auto-scaling, Delta Caching).
- Mentor senior and mid-level developers, ensuring quality standards, documentation, and delivery timelines.
- Languages: Python, PySpark, SQL
- Platforms: Databricks (Jobs, Workflows, Delta Live Tables, Unity Catalog), GCP Dataproc
- Data Tools: Hadoop, Hive, Pig, Spark (RDD & DataFrame APIs), Delta Lake
- Cloud & Integration: GCS, BigQuery, Pub/Sub, Cloud Composer, Airflow
- Automation: GenAI-powered migration tools, custom Python utilities for code conversion
- Version Control & DevOps: Git, Terraform, Jenkins, CI/CD pipelines
- Other: Performance tuning, cost optimization, and lineage tracking with Unity Catalog
- 10-14 years of data engineering experience with at least 3 years leading Databricks or Spark modernization programs.
- Proven success in migration or replatforming projects from Hadoop or Dataproc to Databricks.
- Exposure to AI/GenAI in code transformation or data engineering automation .
- Strong stakeholder management and technical leadership skills.
- 11-12 Years
Job Classification
Industry: IT Services & ConsultingFunctional Area / Department: Engineering - Software & QARole Category: Software DevelopmentRole: Data EngineerEmployement Type: Full timeContact Details:
Company: InfogainLocation(s): Noida, Gurugram
Keyskills: hive python technical leadership data validation performance tuning airflow pyspark apache pig data engineering artificial intelligence sql dataproc data bricks automation apache git stakeholder management spark gcp data ingestion hadoop bigquery
Similar positions
Infogain
A global digital engineering company delivering technology solutions that accelerate business outcomes. It specializes in cloud, data, AI, and experience-led transformation for enterprises across industries.
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in