Hiring For Senior Azure Data Engineer
- Vidushi Infotech
- 10 - 15 years
- Pune
- 6 months ago
- Email to a friend
- Report this job
Job Description
Job Summary:
We are seeking an experienced Senior Data Engineer to join our data engineering team. The ideal candidate will be responsible for designing, building, and maintaining robust data pipelines to support enterprise data solutions. The role focuses on data ingestion into Azure Data Lake (ADLS Gen2), data transformations in Databricks, and end-to-end pipeline automation to enable scalable and high-quality data workflows across the organization.
Key Responsibilities:
Data Ingestion & Pipeline Development
- Design and develop scalable, secure, and efficient data pipelines using Azure Data Factory, PySpark, or Databricks notebooks.
- Ingest structured and unstructured data from multiple sources (SQL, APIs, flat files, on-premise systems) into Azure Data Lake (ADLS Gen2).
- Ensure robust partitioning, metadata management, and schema evolution in the ingestion process.
Data Transformation & Curation
- Perform advanced data wrangling and transformation in Azure Databricks (PySpark/SQL).
- Develop and maintain reusable code modules and transformation logic using Delta Lake architecture (Bronze/Silver/Gold layers).
- Optimize pipelines for performance, cost-efficiency, and maintainability.
Testing, Validation & Monitoring
- Implement unit testing, data validation, and pipeline monitoring to ensure data quality and reliability.
- Collaborate with QA and data governance teams to enforce data integrity, lineage, and compliance.
Deployment & Automation
- Manage CI/CD pipelines for data workflows using Azure DevOps or Git-based repos.
- Develop modular pipelines that support version control, rollback, and scheduled orchestration.
Collaboration & Stakeholder Support
- Work closely with data architects, analysts, and business stakeholders to translate data needs into technical requirements.
- Provide documentation and training for data consumers and ensure ongoing support for data products.
Required Qualifications:
- 10+ years of experience in data engineering, big data development, or ETL/ELT pipeline design.
- Strong hands-on experience with:
- Azure Data Lake (ADLS Gen2)
- Azure Databricks (PySpark, Delta Lake)
- Azure Data Factory / Synapse Pipelines
- Strong SQL and Python skills for data processing and transformation.
- Experience with CI/CD practices and tools like Azure DevOps.
- Deep understanding of data architecture patterns, including medallion architecture (Bronze/Silver/Gold).
- Familiarity with data governance, metadata management, and privacy standards (e.g., GDPR, HIPAA).
Preferred Qualifications:
- Azure Data Engineer Associate certification (DP-203)
- Databricks Certified Data Engineer Associate or Professional
- Experience with REST APIs and JSON ingestion
- Familiarity with Power BI or other reporting tools (for data validation and testing)
- Exposure to real-time data ingestion using Event Hubs, Kafka, or Stream Analytics
Soft Skills:
- Excellent communication and stakeholder management skills
- Ability to work independently and mentor junior developers
- Strong problem-solving and analytical thinking
Job Classification
Industry: IT Services & ConsultingFunctional Area / Department: Engineering - Software & QARole Category: Software DevelopmentRole: Data EngineerEmployement Type: Full timeContact Details:
Company: Vidushi InfotechLocation(s): Pune
Keyskills: SQL Azure Data Engineering Python Data Ingestion Data Transformation Pyspark pipeline development Automation Azure Databricks Deployment Data Bricks Azure Data Factory Azure Synapse Data Curation Data Lake Spark ETL Testing
Similar positions
Vidushi Infotech
Vidushi Infotech Software Solutions Provider Pvt. Ltd. Vidushi Infotech is a Pune based CMMI Level 3 organization, specializing across various arenas of the IT and the digital marketing industry. Vidushi Infotech has mastered the craft of customized web development, development of distributed an...
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in