Data Engineer (PH)
Design, develop, and maintain scalable data processing systems, ETL pipelines, real-time processing systems, data quality controls, and secure AWS-based infrastructure for a Databricks data warehouse.
Responsibilities
- Design, develop, and maintain highly scalable, reliable, and efficient data processing systems
- Collaborate with data analysts, software developers, and business stakeholders to architect data solutions
- Develop and maintain ETL pipelines for extracting, transforming, and loading data into the Databricks data warehouse
- Develop and maintain real-time data processing systems using Spark Streaming and Kafka
- Establish and enforce data quality and validation checks
- Troubleshoot and resolve complex data processing issues across teams
- Monitor and optimize data processing systems for performance, scalability, and reliability
- Utilize AWS services such as EC2, S3, Glue, and Databricks to architect and manage data infrastructure
- Implement security measures and access controls to safeguard data assets
- Stay current with AWS technologies and best practices
Requirements
- Bachelor's or Master's degree in Computer Science or a related field
- Minimum of 5 years of hands-on experience as a Data Engineer
- Good understanding of Databricks platform and Delta Lake
- Familiarity with data job scheduler tools such as Dagster
- Proficiency in one or more programming languages such as Scala, Java, or Python
- Deep expertise in big data technologies including Apache Spark for ETL processing and optimization
- Proficiency with BI tools such as Metabase for data visualization and analysis
- Advanced understanding of data modeling, data quality, and data governance best practices
- Outstanding communication and collaboration skills
- Extensive experience in AWS operational management
- Strong understanding of AWS services such as EC2, S3, Glue, and EMR
- Proficiency in AWS security best practices
- Hands-on experience with automation and DevOps tools such as Terraform
- Ability to read and write in English
Benefits
- Meaningful cross-functional collaboration with opportunities to improve customer experience
- Opportunities for career advancement as the organization expands outside APAC
- A culture that welcomes data-backed ideas and meaningful change