Databricks Data Engineer
Design, develop and optimise scalable data solutions using Azure Databricks, Azure Data Factory and Scala. This role processes large-scale datasets, builds ETL pipelines and works with stakeholders within a London-based long-term data engineering programme.
Free Tailor for ATS: 10/10 runs left
We are looking for an experienced Azure Data Engineer to join a long-term data engineering programme based in London. You will be responsible for designing, developing and optimising scalable data solutions using Azure Databricks, Azure Data Factory and Scala. The role will involve processing large-scale datasets, building reliable ETL pipelines and working closely with technical and business stakeholders to translate requirements into effective data solutions.
This is a hands-on engineering position suited to someone with strong Azure data platform experience and a proven background delivering high-performance data pipelines within complex enterprise environments.
Contract Information
Job title: Databricks Data Engineer
Location: London
Work model: Hybrid (3 days on-site)
Start date: ASAP
Contract length: 12 Months
Contract type: Inside IR35
Key Responsibilities
Design and develop scalable data engineering solutions using Azure Databricks and Azure Data Factory.
Build and maintain complex ETL and data transformation pipelines.
Process and transform large datasets containing millions of records.
Optimise data pipelines and distributed processing workloads for performance, reliability and scalability.
Participate in customer discussions, requirement-gathering sessions and technical solution design.
Translate business and technical requirements into practical data engineering solutions.
Develop clean, maintainable and reusable code in line with engineering best practices.
Perform code reviews, performance tuning and technical troubleshooting.
Monitor delivery progress and communicate project updates to stakeholders.
Identify and escalate technical risks, blockers and critical issues.
Collaborate with engineers, architects and wider project teams within an Agile delivery environment.
Essential Skills and Experience
6–8 years of commercial experience within data engineering or data software engineering.
Strong hands-on experience developing solutions using Azure Databricks.
Proven experience processing and transforming large-scale datasets.
Strong working knowledge of Azure Data Factory, including the development of complex ETL workflows.
Experience optimising data pipelines and distributed data-processing environmens.
Hands-on development experience using Scala.
Strong understanding of relevant Microsoft Azure data and cloud services.
Experience designing scalable, reliable and maintainable cloud-based data solutions.
Strong SQL, data transformation and troubleshooting skills.
Excellent analytical and problem-solving ability.
Strong communication skills and experience working with technical and non-technical stakeholders.
Previous experience working within Agile delivery teams.