GCP Data Engineer
- e Solutions India
- 5 - 10 years
- Chennai
- 3 months ago
- Email to a friend
- Report this job
Job Description
Position 1- Senior GCP Data Engineer (Ingestion & Bronze Layer Specialist)
Primary Focus: App Interface Layer, Ingestion Layer, Bronze Layer
Key Responsibilities:
- Design and implement real-time data ingestion pipelines using Pub/Sub and Kafka Streams for healthcare data formats (HL7, FHIR)
- Build robust Bronze layer as the single source of truth storing raw, untransformed data in Cloud Storage
- Develop streaming ingestion patterns using Dataflow for real-time data capture with minimal transformation
- Implement batch loading processes using Dataproc for large-volume data from diverse sources (logs, databases, APIs)
- Apply schema inference and basic data type adjustments while preserving raw data lineage
- Design partitioning strategies in Cloud Storage for efficient historical data archival and retrieval
- Establish data landing zone controls including audit logging, versioning, and immutability patterns
- Create automated workflows using Cloud Composer for orchestrating ingestion pipelines
- Implement data catalog and metadata management for raw data assets
Required Skills:
- 5+ years experience with GCP services (Cloud Storage, Pub/Sub, Dataflow, Dataproc, Cloud Composer)
- Strong expertise in Apache Kafka, Kafka Streams, and event-driven architectures
- Proficiency in Python and/or Java for data pipeline development using Apache Beam SDK
- Experience with healthcare data standards (HL7, FHIR) and handling semi-structured data
- Hands-on experience with streaming frameworks (Apache Beam, Dataflow) for near-real-time ingestion
- Knowledge of file formats and compression (JSON, Avro, Parquet) for raw data storage
- Understanding of CDC patterns, incremental loading, and data versioning strategies
- Experience with Cloud Storage lifecycle management and cost optimization
Job Classification
Industry: IT Services & ConsultingFunctional Area / Department: ConsultingRole Category: IT ConsultingRole: Cloud ConsultantEmployement Type: ContractContact Details:
Company: E SolutionsLocation(s): Chennai
Keyskills: Gcp Cloud GCP Pubsub Java Kafka Dataproc Cloud Platform SQL Cloud Storage cloud Data Flow Google Cloud Platforms Python
Similar positions
e Solutions India
Client of E-solutions, The Company is a Manufacturer and supplier of single pump water booster, ice machine filters, combination filter system, coffee machine filters, pumps etc.
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in