Data Engineering Consultant
- Optum
- 7 - 12 years
- Noida, Gurugram
- 5 months ago
- Email to a friend
- Report this job
Job Description
Primary Responsibilities:
- Ingest data from multiple on-prem and cloud data sources using various tools & capabilities in Azure
- Design and develop Azure Databricks processes using PySpark/Spark-SQL
- Design and develop orchestration jobs using ADF, Databricks Workflow
- Analyzing data engineering processes being developed and act as an SME to troubleshoot performance issues and suggest solutions to improve
- Building test framework for the Databricks notebook jobs for automated testing before code deployment
- Design and build POCs to validate new ideas, tools, and architectures in Azure
- Continuously explore new Azure services and capabilities; assess their applicability to business needs
- Prepare case studies and technical write-ups to showcase successful implementations and lessons learned
- Work closely with clients, business stakeholders, and internal teams to gather requirements and translate them into technical solutions using best practices and appropriate architecture
- Contribute to full lifecycle project implementations, from design and development to deployment and monitoring
- Ensure solutions adhere to security, compliance, and governance standards
- Monitor and optimize data pipelines and cloud resources for cost and performance efficiency
- Identifies solutions to non-standard requests and problems
- Mentor and support existing on-prem developers for cloud environment
- Comply with the terms and conditions of the employment contract, company policies and procedures, and any and all directives (such as, but not limited to, transfer and/or re-assignment to different work locations, change in teams and/or work shifts, policies in regards to flexibility of work benefits and/or work environment, alternative work arrangements, and other decisions that may arise due to the changing business environment). The Company may adopt, vary or rescind these policies and directives in its absolute discretion and without any limitation (implied or otherwise) on its ability to do so
Required Qualifications:
- Undergraduate degree or equivalent experience
- 7+ years of overall experience in Data & Analytics engineering
- 5+ years of experience working with Azure, Databricks, and ADF, Data Lake
- 5+ years of experience working with data platform or product using PySpark and Spark-SQL
- Solid experience with CICD tools such as Jenkins, GitHub, Github Actions, Maven etc.
- In-depth understanding of Azure architecture & ability to come up with efficient design & solutions
- Highly proficient in Python and SQL
- Proven excellent communication skills
- Key Skill: Azure Data Engineer - Azure Databricks, Azure Data factory, Python/Pyspark, Terraform
Job Classification
Industry: RetailFunctional Area / Department: Engineering - Software & QARole Category: Software DevelopmentRole: Data EngineerEmployement Type: Full timeContact Details:
Company: OptumLocation(s): Noida, Gurugram
Keyskills: sql ci tools spark python data bricks pyspark ci/cd data warehousing azure data factory docker java oracle adf linux jenkins etl big data data lake azure databricks github ssas maven microsoft azure data engineering sql server terraform aws ssis
Similar positions
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in