Data Engineer
- Brillio
- 11 - 21 years
- Pune
- 4 months ago
- Email to a friend
- Report this job
Job Description
Role & responsibilities
- Design, build, and maintain scalable data pipelines to collect, process, and store from multiple datasets.
- Optimize data storage solutions for better performance, scalability, and cost-efficiency.
- Develop and manage ETL/ELT processes to transform data as per schema definitions, apply slicing and dicing, and make it available for downstream jobs and other teams.
- Collaborate closely with cross-functional teams to understand system and product functionalities, pace up feature development, and capture evolving data requirements.
- Engage with stakeholders to gather requirements and create curated datasets for downstream consumption and end-user reporting.
- Automate deployment and CI/CD processes using GitHub workflows, identifying areas to reduce manual, repetitive work.
- Ensure compliance with data governance policies, privacy regulations, and security protocols.
- Utilize cloud platforms like AWS and work on Databricks for data processing with S3 Storage.
- Work with distributed systems and big data technologies such as Spark, SQL, and Delta Lake.
- Integrate with SFTP to push data securely from Databricks to remote locations.
- Analyze and interpret spark query execution plans to fine-tune queries for faster and more efficient processing.
- Strong problem-solving and troubleshooting skills in large-scale distributed systems.
Preferred candidate profile
BE / B Tech in Engineering.
Min 4+ years of IT Experience.
Skill set: SQL, Python, AWS, Databricks.
Tableau exposure would be optional, good to have.
Have couple of projects having work exposure on Analytics and Insights based on Big Data.
Exposure on building Data sets on complex Big Data would be an advantage.
Job Classification
Industry: IT Services & ConsultingFunctional Area / Department: Engineering - Software & QARole Category: Software DevelopmentRole: Data EngineerEmployement Type: Full timeContact Details:
Company: BrillioLocation(s): Pune
Keyskills: PySpark Databricks AWS SQL
Similar positions
Brillio
Powered by SophosLabs a global threat intelligence and data science team Sophoscloud-native and AI-enhanced solutions secure endpoints (laptops, servers and mobile devices) and networks against evolving cybercriminal tactics and techniques, including automated and active-adversary breaches, ransomwa...
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in