Data Pipeline Development & Maintenance: Design, build, and maintain end-to-end data pipelines for ingesting, processing, and transforming large datasets from various sources (e.g., relational databases, APIs, flat files, streaming data).
ETL/ELT Process Optimization: Develop and optimize ETL/ELT processes using industry-standard tools and techniques, ensuring data accuracy, efficiency, and scalability.
Database Design & Management: Design, implement, and manage relational databases, including schema design, indexing, performance tuning, and data governance.
Data Warehousing: Design and implement data warehousing solutions to support business intelligence and reporting needs. Experience with star schema, snowflake schema, and other data modelling techniques is essential.
Database Administration: Perform database administration tasks, including performance monitoring, capacity planning, backup and recovery, and security management.
Data Quality & Governance: Implement data quality checks, validation rules, and data governance policies to ensure data accuracy and consistency.
Cloud Platform Expertise: Leverage cloud platforms (e.g., AWS, Azure, GCP) for data storage, processing, and management.
Collaboration & Communication: Collaborate with cross-functional teams (e.g., data scientists, business analysts, software engineers) to understand data requirements and deliver effective data solutions. Clearly communicate technical concepts to both technical and non-technical audiences.
Mentoring & Knowledge Sharing: Mentor junior engineers and share knowledge and best practices within the team.
Automation & Scripting: Automate data engineering tasks using scripting languages (e.g., Python, Bash).
Stay Up to Date: Continuously research and evaluate new data technologies and techniques to improve our data infrastructure.
Production Support: Monitoring the batch/jobs daily, no matter its weekdays or weekend.
Production Release: Actively participation in release process.
Qualifications:
Experience: Minimum of 10 years of experience as a Data Engineer or in a similar role.
Database Expertise:
Expert proficiency in SQL Server: Extensive experience with SQL Server, including database design, performance tuning, query optimization, and database administration.
Expert proficiency in PostgreSQL: Deep understanding of PostgreSQL, including database design, performance tuning, query optimization, and database administration.
Solid experience with other relational databases like MySQL, Oracle, etc. is a plus.
Data Modelling: Strong understanding of data modelling principles and techniques (e.g., dimensional modelling, star schema, snowflake schema).
Cloud Computing: Experience with cloud platforms (AWS, Azure, or GCP) and related data services (e.g., S3, Redshift, Snowflake, Azure Data Lake Storage).
Programming & Scripting: Proficiency in scripting languages such as Python, Bash, or similar.
Data Governance & Quality: Experience implementing data quality checks, data governance policies, and data validation rules.
Problem-Solving & Analytical Skills: Excellent problem-solving and analytical skills with the ability to identify and resolve complex data-related issues.
Communication & Collaboration: Excellent communication, collaboration, and interpersonal skills. Ability to work effectively in a team environment.
Education: Bachelors or masters degree in computer science, Information Technology, or a related field.
Job Classification
Industry: IT Services & ConsultingFunctional Area / Department: Engineering - Software & QARole Category: Software DevelopmentRole: Data EngineerEmployement Type: Full time