Azure Data Engineer
- Dexian India
- 6 - 8 years
- Hyderabad
- 5 months ago
- Email to a friend
- Report this job
Job Description
Role & responsibilities
- We are seeking a hands-on Data Engineer to develop, optimize, and maintain automated data pipelines supporting data governance and analytics initiatives. This role will focus on building production-ready workflows for ingestion, transformation, quality checks, lineage capture, access auditing, cost usage analysis, retention tracking, and metadata integration, primarily using Azure Databricks, Azure Data Lake, and Microsoft Purview.
- Experience: 4+ years in data engineering, with strong Azure and Databricks experience
- Key Responsibilities
- Pipeline Development Design, build, and deploy robust ETL/ELT pipelines in Databricks (PySpark, SQL, Delta Lake) to ingest, transform, and curate governance and operational metadata from multiple sources landed in Databricks.
- Granular Data Quality Capture Implement profiling logic to capture issue-level metadata (source table, column, timestamp, severity, rule type) to support drill-down from dashboards into specific records and enable targeted remediation.
- Governance Metrics Automation Develop data pipelines to generate metrics for dashboards covering data quality, lineage, job monitoring, access & permissions, query cost, usage & consumption, retention & lifecycle, policy enforcement, sensitive data mapping, and governance KPIs.
- Microsoft Purview Integration Automate asset onboarding, metadata enrichment, classification tagging, and lineage extraction for integration into governance reporting.
- Data Retention & Policy Enforcement Implement logic for retention tracking and policy compliance monitoring (masking, RLS, exceptions).
- Job & Query Monitoring Build pipelines to track job performance, SLA adherence, and query costs for cost and performance optimization.
- Metadata Storage & Optimization Maintain curated Delta tables for governance metrics, structured for efficient dashboard consumption.
- Testing & Troubleshooting Monitor pipeline execution, optimize performance, and resolve issues quickly.
- Collaboration Work closely with the lead engineer, QA, and reporting teams to validate metrics and resolve data quality issues.
- Security & Compliance Ensure all pipelines meet organizational governance, privacy, and security standards.
- Required Qualifications
- Bachelors degree in Computer Science, Engineering, Information Systems, or related field
- 4+ years of hands-on data engineering experience, with Azure Databricks and Azure Data Lake
- Proficiency in PySpark, SQL, and ETL/ELT pipeline design
- Demonstrated experience building granular data quality checks and integrating governance logic into pipelines
- Working knowledge of Microsoft Purview for metadata management, lineage capture, and classification
- Experience with Azure Data Factory or equivalent orchestration tools
- Understanding of data modeling, metadata structures, and data cataloging concepts
- Strong debugging, performance tuning, and problem-solving skills
- Ability to document pipeline logic and collaborate with cross-functional teams
Preferred candidate profile
- Preferred Qualifications
- Microsoft certification in Azure Data Engineering
- Experience in governance-heavy or regulated environments (e.g., finance, healthcare, hospitality)
- Exposure to Power BI or other BI tools as a data source consumer
- Familiarity with DevOps/CI-CD for data pipelines in Azure
Experience integrating both cloud and on-premises data sources into Azure
Job Classification
Industry: IT Services & ConsultingFunctional Area / Department: OtherRole Category: OtherRole: OtherEmployement Type: Full timeContact Details:
Company: Dexian IndiaLocation(s): Hyderabad+ View Contact
Keyskills: Data Lake Azure Data Factory Pyspark Azure Databricks Data Warehousing Data Governance SQL Python
Fraud Alert to job seekers!
₹ 15-20 Lacs P.A
Similar positions
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in