Senior Data Engineer
- V Soft Consulting
- 9 - 14 years
- Bengaluru
- 3 months ago
- Email to a friend
- Report this job
Job Description
Role & responsibilities
Design and build a scalable data platform that can ingest, store, manage and stream massive amounts of data while simplifying its analysis and processing to enable rapid development of high-quality data products and services.
Implement and test robust low latency data pipelines for preparing/curating data that will support various data as a service product.
You will work with various cross functional teams to explore data and figure out all the data wrangling needed to clean, curate the data and build the end-to-end data pipeline per requirements.
You will design necessary privacy and security preventive and detective controls into the CICD pipelines of each data pipeline.
You will have a strong bias for operational excellence ensuring error handling, restart-ability ensuring data consistency, logging, monitoring and alerting is built into the pipeline.
You will drive continual improvements for reliability, performance, scalability, quality and own associated KPI.
- 5+ years of experience in development and operation in production of Cloud native data streaming data systems (with an emphasis on scalability, robustness, data delivery low latency and data privacy and quality control)
5+ years experience working in Cloud native real-time streaming data ecosystem such as Spark, Flink, Kinesis, Lambda, Kafka, EMR/EKS platform, Lakehouse platform (i.e Delta.io, Databricks)
5+ years of experience in developing optimizing AWS data system architecture: Infrastructure as Code, integration and deployment automation, security.
- 5+ years of experience in ensuring operation reliability of cloud native data system in production.
5+ years in relational database concepts with a solid knowledge of SQL, SQL Tuning, Big Data NoSQL technologies
5+ years experience with languages like Spark-Scala, Py-Spark and /or Python, Java
Experience with development and deployment of containerized application using Docker, Kubernetes, Helm
Experience in building systems that monitor data losses and data quality control.
Demonstrated knowledge of data structures and algorithms.
Experience in designing foundations of Operational excellence and implementing.
Curiosity and passion to learn with a strong bias to action
Job Classification
Industry: IT Services & ConsultingFunctional Area / Department: Data Science & AnalyticsRole Category: Data Science & Analytics - OtherRole: Data Science & Analytics - OtherEmployement Type: Full timeContact Details:
Company: V2softLocation(s): Bengaluru
Keyskills: Kafka Data Bricks AWS Streaming Pyspark Data Engineering Apache Flink Hadoop Big Data Spark Streaming SQL Aws Kinesis Kinesis Low Latency SCALA Data Lake Spark Data Warehousing ETL
Similar positions
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in