Data Engineer
Summary
Data Engineer building metadata-driven Spark notebooks, ETL pipelines, and Power BI integrations while supporting CI/CD, data quality, and data lineage at SmartestEnergy, a renewable energy supplier.
- Automate metadata driven notebooks with Spark
- Build data ingestion pipelines
- Contribute to Git code reviews and testing cycles
- Develop ETL processes
- Document data flows and transformations
- Implement data quality checks
- Integrate data models with Power BI
- Maintain basic data lineage
- Monitor data quality and validation rules
- Support CI/CD processes
- Write unit tests for pipeline logic
Perks/Benefits:
- Flexible working
- Hybrid working
- Work from anywhere up to 30 days per year