Openings For Gcp Data Engineer / Lead _immediate Joiners
- Infogain
- 7 - 12 years
- Pune
- 14 days ago
- Email to a friend
- Report this job
Job Description
Job Description:Core Skills
Required Skills & Experience
610 years of experience in data engineering or analytics, with 3+ years hands-on on GCP.
Strong experience with PySpark, Dataproc, GCS, BigQuery, and JDBC ingestion.
Proven experience migrating SAS workloads to PySpark or SQL-based systems.
Hands-on knowledge of GCP Medallion architecture (Bronze/Silver/Gold).
Understanding of Dataplex, IAM, policy tags, and secure data handling.
Experience with CI/CD (Cloud Build/GitHub Actions) and workflow orchestration (Cloud Composer/Airflow).
Strong problem-solving ability, debugging skills, and ability to guide teams through technical challenges.Preferred Skills
Experience with Vertex AI, ML Ops, and ML pipeline deployment.
Knowledge of Delta/Iceberg/Hudi table formats on GCS.
Exposure to real-time ingestion (Pub/Sub, Dataflow).
Google Cloud Professional Data Engineer or Cloud Architect certification.Soft Skills
Strong leadership and mentoring capabilities.
Excellent communication skills to support developers, architects, and business teams.
Ability to manage multiple priorities, resolve conflicts, and maintain steady progress under pressure.
Key Responsibilities
1. Hands-on Technical Leadership
Work closely with development teams on a daily basis to guide solution design, troubleshoot issues, and resolve technical blockers.
Enforce engineering best practices, coding standards, and architectural guidelines across all data pipelines and workloads.
Perform design and code reviews, ensuring quality, scalability, and reliability of the platform.
2. Data Engineering on GCP
Lead development of ingestion pipelines via Direct JDBC connectivity from Oracle and Teradata into the Raw/Bronze layer on GCS.
Develop and optimize PySpark workloads on Dataproc for data cleansing, transformation, and harmonization into the Curated/Silver layer.
Contribute to design of the Gold layer in BigQuery, including table structures, partitioning, clustering, and performance optimization.
3. Migration from SAS to GCP
Translate existing SAS logic into PySpark, ensuring functional parity, improved performance, and operational efficiency.
Provide guidance on PySpark coding patterns, UDFs, optimization strategies, shuffle/skew handling, and best practices for Dataproc jobs
4. BigQuery Engineering & Optimization
Build and optimize SQL models, materialized views, and analytical datasets in BigQuery.
Apply query optimization techniques, cost controls, and data modeling best practices (star/snowflake).
Implement RLS/CLS for secure reporting and work with BI teams to integrate BigQuery into reporting tools.
5. Vertex AI & ML Support
Assist data scientists in building ML pipelines using Vertex AI (training, prediction, feature engineering).
Guide integration of feature pipelines from Silver ? Vertex AI Feature Store.
Ensure reproducibility, lineage, and model monitoring (drift, bias).
6. Data Governance & Security (Dataplex + IAM)
Implement and enforce governance standards using Dataplex, including cataloging, policy tags, and data domains.
Ensure datasets follow proper IAM roles, tagging, and compliance (PII/PCI/PHI masking where needed).
Support lineage metadata, DQ implementation, and documentation.
7. Operations, Monitoring & Cost Optimization
Optimize Dataproc clusters (autoscaling, Preemptibles), GCS storage lifecycle policies, and BigQuery costs.
Establish monitoring dashboards, logs, alerts, and operational KPIs.
Troubleshoot and resolve production issues, ensuring high availability and reliability.
Job Classification
Industry: IT Services & ConsultingFunctional Area / Department: OtherRole Category: OtherRole: OtherEmployement Type: Full timeContact Details:
Company: InfogainLocation(s): Pune
Keyskills: Pyspark Google cloud platform Dataproc Bigquery
Similar positions
Infogain
Infogain is a Silicon Valley headquartered company with software platform engineering and deep domain expertise in the travel, retail, insurance and high technology industries. We accelerate the delivery of digital customer engagement systems using digital technologies such as cloud, mic...
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in