Data Engineer - Data Lake
- Wissen Infotech
- 7 - 10 years
- Bengaluru
- 5 months ago
- Email to a friend
- Report this job
Job Description
Data Engineer Data Lakes
Exp: 7+ Years
Location :Bangalore
Notice:Immediate
Key Experience & Expertise
- Extensive hands-on experience in Partitioning Delta Lake Tables to achieve optimized query performance and storage efficiency.
- Proficient in reading, interpreting, and reporting Delta Lake Transaction Logs to ensure data reliability and consistency.
- Strong expertise in transforming streaming data and building Delta Live Tables (DLT) pipelines for real-time analytics.
- Proven ability to design and implement Slowly Changing Dimensions (SCDs) using PySpark for scalable data warehousing solutions.
- Skilled in configuring and leveraging Auto Optimize to enhance Delta Lake performance.
- Practical knowledge of Directed Acyclic Graphs (DAGs) for workflow orchestration (e.g., Airflow, Databricks Workflows).
- Proficient in developing and managing Materialized Views for accelerating queries and optimizing performance.
- In-depth understanding of Medallion Architecture (Bronze, Silver, Gold layers) to structure data pipelines in a Lakehouse environment.
- Strong focus on performance tuning and PySpark best practices, including:
- Efficient implementation of Window Functions
- Optimized Merge operations for handling large-scale datasets
Job Classification
Industry: IT Services & ConsultingFunctional Area / Department: Data Science & AnalyticsRole Category: Data Science & Machine LearningRole: Data EngineerEmployement Type: Full timeContact Details:
Company: Wissen InfotechLocation(s): Bengaluru+ View Contact
Keyskills: Pyspark Delta Lake Transaction Logs Data Engineer Delta Lake Data Bricks Delta Live Tables Directed Acyclic Graphs
Fraud Alert to job seekers!
₹ Not Disclosed
Similar positions
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in