Azure Data Engineer
- Infogain
- 5 - 8 years
- Noida, Gurugram
- 2 months ago
- Email to a friend
- Report this job
Job Description
- Analyze existing Hadoop, Pig, and Spark scripts from Dataproc and refactor them into Databricks-native PySpark.
- Implement data ingestion and transformation pipelines using Delta Lake best practices.
- Apply conversion rules and templates for automated code migration and testing.
- Conduct data validation between legacy and migrated environments (schema, count, and data-level checks).
- Collaborate on developing AI-driven tools for code conversion, dependency extraction, and error remediation.
- Ensure best practices for code versioning, error handling, and performance optimization.
- Participate in UAT, troubleshooting, and post-migration validation activities.
- Core: Python, PySpark, SQL
- Databricks: Delta Lake, Unity Catalog, Databricks Workflows, MLflow (basic understanding)
- GCP: Dataproc, BigQuery, GCS, Composer/Airflow, Cloud Functions
- Data Engineering: Hadoop, Hive, Pig, Spark SQL
- Automation: Experience with migration utilities or AI-assisted code transformation tools
- CI/CD: Git, Jenkins, Terraform (preferred)
- Validation: Data comparison utilities (Delta-to-Delta, DataFrame diffing, schema validation)
- 5-8 years in data engineering or big data application development.
- Hands-on experience migrating Spark or Hadoop workloads to Databricks.
- Familiarity with Delta architecture , data quality frameworks , and GCP cloud integration .
- Exposure to GenAI-based tools for automation or code refactoring is a plus.
- 6-8 Years
Job Classification
Industry: IT Services & ConsultingFunctional Area / Department: Engineering - Software & QARole Category: Software DevelopmentRole: Data EngineerEmployement Type: Full timeContact Details:
Company: InfogainLocation(s): Noida, Gurugram
Keyskills: hive continuous integration python scala pyspark microsoft azure apache pig code versioning tools machine learning data engineering application development sql dataproc gen git automation spark gcp jenkins terraform hadoop bigquery sqoop big data
Similar positions
Infogain
A Silicon-Valley headquartered company, Infogain is a global business oriented IT consulting provider of front-end, customer-facing technologies, processes and applications, leading to a more efficient and streamlined customer experience. We want our clients€™ interactions with their cus...
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in