Unity Catalog Migration - DB
- Puresoftware
- 8 - 13 years
- Hyderabad
- 2 months ago
- Email to a friend
- Report this job
Job Description
This role focuses on migrating existing data environments from Apache Hive Metastore to Databricks Unity Catalog, leveraging Scala for data transformations and pipeline adjustments.
A seasoned senior candidate with 8+ years of relevant experience with strong expertise in Databricks, Scala, Spark with Azure cloud environment.
The typical responsibilities include;
- Experience with large-scale data migrations.
- Good knowledge and implementation experience in data lineage and auditing tools.
- Assessment Planning: Analyze the current Hive Metastore environment, including data models, pipelines, and access controls, to define a comprehensive migration strategy to Unity Catalog.
- Unity Catalog Setup: Configure and manage Unity Catalog metastores, external locations, and credentials within Databricks workspaces.
- Metadata Migration: Develop and execute Scala-based scripts and Databricks notebooks to migrate Hive Metastore tables, views, and associated metadata to Unity Catalog. This may involve using Unity Catalogs upgrade wizard or custom solutions for complex scenarios.
- Data Governance Security: Implement and enforce Unity Catalogs centralized access controls (ACLs, grants) to ensure secure data access and compliance.
- Pipeline Modernization: Refactor existing Scala/Spark data pipelines to integrate seamlessly with Unity Catalog, updating table references and ensuring data integrity during and after migration.
- Testing Validation: Conduct thorough testing to validate data consistency, performance, and access control policies in the Unity Catalog environment.
- Documentation: Create comprehensive documentation for the migration process, including architecture diagrams, migration scripts, and operational procedures.
- Collaboration: Work closely with data architects, data scientists, and other engineering teams to ensure a smooth transition and adoption of Unity Catalog.
- Required Skills Qualifications:
- Expertise in Scala: Strong proficiency in Scala for data manipulation, Spark development, and building robust data pipelines.
- Databricks Platform: In-depth knowledge of Databricks, including Spark, Delta Lake, and Databricks notebooks.
- Unity Catalog: Hands-on experience with Unity Catalog setup, configuration, and migration strategies.
- Hive Metastore: Solid understanding of Hive Metastore concepts and its integration with data processing frameworks.
- Cloud Platforms: Experience with cloud platforms (e.g., Azure, AWS, GCP) and their data storage services (e.g., ADLS, S3, GCS).
- Data Governance: Familiarity with data governance principles, access control mechanisms, and data security best practices.
- Problem-Solving: Excellent analytical and problem-solving skills to address complex migration challenges.
Job Classification
Industry: IT Services & ConsultingFunctional Area / Department: Engineering - Software & QARole Category: Software DevelopmentRole: Data Platform EngineerEmployement Type: Full timeContact Details:
Company: PuresoftwareLocation(s): Hyderabad
Keyskills: Access control hive metadata data security spark Analytical SCALA Cloud data governance Apache
Similar positions
Puresoftware
PureSoftware delivers IT services and digital solutions across industries including telecom, healthcare, and financial services. The company specializes in software engineering, product development, and digital transformation. Its career portal reflects roles in DevOps, QA, cloud technologies, and e...
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in