Pyspark technical Lead @ Sopra Steria

Experience:

1. 5+ years of relevant experience in data engineering / big data development
2. Proven track record of handling large-scale data solutions

Key Responsibilities:

1. Maintain, develop, and design data ingestion and transformation pipelines using Spark and Scala
2. Process and optimize SQL and PL/SQL queries for large datasets
3. Integrate and process data from multiple sources for analytics and reporting
4. Collaborate with BI and Analytics teams to support reporting and dashboard needs
5. Ensure data quality, performance, and security
6. Provide innovative solutions to complex business/technical problems
7. Identify and resolve technical, integration and development issues
8. Implement best practices, standards and processes to ensure quality of the final product
9. Responsible for development of features independently
10. Continuous build, automated testing, and release management
11. Participate in status meetings and provide regular updates

Required Skills:

Technical:

1. Expertise in Spark and Scala for large-scale data processing
2. Proficiency in Python and SQL, including hands-on experience with PL/SQL
3. Experience in data integration/ETL workflows;
4. Strong knowledge of relational databases such as Oracle and PostgreSQL

Professional Competencies:

1. Excellent verbal and written communication skills
2. Result-oriented with strong analytical and problem-solving skills
3. Ability to work in a fast-paced, agile, and collaborative environment

Desired Skills:

1. Exposure to cloud data services (AWS preferred)
2. Proficiency in Linux/Unix shell scripting for automation and data operation
3. Experience working in Agile environments Scrum and/or SAFe frameworks
4. Exposure to DevOps culture and practices, including CI/CD and automation
5. Understanding of software engineering lifecycle Development, Testing, and Production Support
6. Experience in production issue handling and transformation support management

Good to Have Skills:

1. Exposure to Dataiku preferred
2. Awareness of cloud and big data architecture principles