You'll design and own end-to-end RAG pipelines for payments domain content using Python, Azure Data Factory, Functions, ADLS, CosmosDB, and AI Search, with CI/CD via GitHub Actions and Azure DevOps. RedCompass Labs builds AI-powered payment transformation and financial crime detection solutions, including the RedFlag Accelerator for identifying human trafficking. You'll collaborate with AI engineers and payments experts to turn regulatory documents and handbooks into a reliable retrieval layer for their proprietary multi-agent AI platform.

You will lead Azure Databricks architecture, designing enterprise Lakehouse solutions with Apache Spark, PySpark, Delta Lake, Azure Data Factory, Data Lake Gen2, Event Hubs, Synapse, Python, SQL, Terraform, and CI/CD. This is part of a major Energy & Utilities transformation programme, specifically supporting critical Energy Trading data.

Capgemini★3.7 (11k)· London

Design, build, and manage data pipelines and models using AWS Snowflake, Azure Data Lake, Databricks, Redshift, and BigQuery. Automate CI/CD with Azure DevOps, GitHub Actions, Jenkins, GitLab CI/CD, and provision infrastructure with AWS CloudFormation, Docker, ECS, EKS, and Kubernetes. Capgemini is an AI-powered global business and technology transformation partner.

Lead the Credit Risk Analytics IT team using S&P Global FRA (fraimWRX), Redis, Kafka, Spark, HBASE, HDFS, Kubernetes, Python, .NET, C#, MS SQL Server, and MS Azure. The team designs, integrates, and supports systems for calculating and reporting Credit Risk across all locations. This is a VP-level role managing onshore, nearshore, and offshore developers.

Design and maintain data pipelines using Azure Data Factory, Databricks, Synapse Analytics, and Data Lake Storage, developing with Python, Scala, or SQL. As a technology consultant, you'll build scalable data platforms for enterprise clients and mentor junior engineers.

Build and evolve ASOS's data and ML platform using Python, Spark, Azure Databricks, ADLS Gen2, Unity Catalog, Terraform, and CI/CD tools. You'll implement IaC, manage data/ML lifecycle services (pipelines, experiment tracking, versioning), and support a centralised AgentOps capability with LLM gateways and safety controls. This team enables consistent, reproducible data engineering and ML workflows across the company.

MUFG★3.3 (1k)· London

Works with S&P Global FRA, fraimWRX (AKS), Redis, Kafka, Spark, HBASE, HDFS, Kubernetes, Python, .NET, C#, MS SQL Server, MS Azure. Leads the Credit Risk Analytics IT team, designing and integrating Middle Office systems for calculating and reporting Credit Risk across all locations. Manages onshore, nearshore, and offshore teams.

Microsoft★4.2 (9k)· London

You will work with Databricks, Microsoft Azure (Data, Infrastructure, AI) services, Azure Data Services, AI Services, and cloud-native technologies. The Global Customer Success Chief Architect’s Office drives adoption of Databricks solutions on the Microsoft platform, leading architecture reviews and designing enterprise-scale data, analytics, and AI solutions.

WNS, a Business Process Management company, seeks a Lead Data Engineer with 10+ years experience to develop features and drive architectural decisions on their cloud delta lakehouse platform using SQL Server, SSIS, T-SQL, Azure Data Factory, ADLS, Azure SQL Database, PySpark, Spark SQL, Databricks, and Delta Lake. The role involves mentoring junior engineers, establishing engineering standards, and leading complex data warehouse initiatives within an agile team in central London.

You'll work with Azure Data Factory, Synapse Analytics, Power BI, SQL, Azure Data Lake, and Microsoft Fabric to design and maintain ETL pipelines and data models for a UK wealth management firm's analytics infrastructure. You'll lead data architecture development, mentor junior engineers, and contribute to platform modernization initiatives while collaborating across technical and business teams in an Agile environment using Azure DevOps.

Lead evolution of a global data landscape onto a primarily Microsoft platform (Azure Data Lake, Databricks, Azure SQL, Power BI Premium, data mesh/fabric). Manage the transition, own the tech stack, and drive data governance and cultural change. Reports to Director, Technical Operations.

Use PySpark, Python, SQL, Azure Databricks, Data Factory, ADLS Gen2, and Delta Lake to design data pipelines and Kimball models on a cloud lakehouse migration project. The platform leverages Medallion Architecture to supply clean data for Power BI and machine learning, following CI/CD and automated testing.

Robert Half★3.9 (231)· London

Work with Microsoft Fabric, Azure Databricks, Data Factory, PySpark, Python, SQL, and Azure Data Platform on a high-profile consulting engagement using lakehouse and medallion architectures. You will lead a team while remaining hands-on delivering ETL/ELT pipelines and CI/CD practices.

Deloitte★3.9 (14k)· London

You will work with Python, JavaScript, Kafka, Azure, Kubernetes, Terraform, Helm, LangChain, PySpark, Databricks, and ADLS. You will design, develop, and deploy production-grade GenAI and machine learning systems for the GenAI team within Deloitte’s Data & AI organisation.

You will work with Azure Data Factory, Databricks, Azure Data Lake, SQL, ETL/ELT, DBT, Python, T-SQL, Spark, and CI/CD. The team builds and optimises Azure data pipelines and supports migration into Azure Data Lake as part of a global data team.

You will work with Azure Infrastructure (Compute, Networking, Storage, Identity, Security), Terraform, Bicep, ARM Templates, Azure Monitor, Log Analytics, Datadog, CI/CD, PowerShell, and Azure CLI. You will lead a 6-month contract to migrate platforms to Azure, covering assessment, deployment, validation, and cutover. This role is for a consultant providing project-based migration execution.

You will work with Azure Databricks, SQL, PySpark/Spark, Azure Data Factory, and Azure Data Lake. The role supports a major data migration and cloud modernisation programme, migrating legacy data environments to a modern Azure-based data platform.

Work with Databricks, Azure Data Factory, Delta Lake, SQL/T-SQL, Python, PySpark, ADLS Gen2, and Azure DevOps. You will design and maintain scalable data pipelines for Wealth Management Europe’s Data Lakehouse platform, turning raw financial data into actionable insights.

Senior Data Engineer role at ASOS. You'll design and build scalable data pipelines using Python, Scala, Spark, and PySpark on Azure (Data Factory, Databricks, Delta Lake) with dbt and Terraform. You'll develop platform components and standards to accelerate data science and analytics teams across product domains like Forecasting and Recommendations.

LSEG★3.7 (402)· London

Design, build, and maintain scalable Azure cloud infrastructure using Terraform, Python, Bash, and SQL. The role enables data platforms (Databricks, Event Hubs, ADLS Gen2) with CI/CD pipelines, containerization (Docker, Kubernetes), and Linux administration for a global financial markets infrastructure company.

Avanade★3.6 (459)· London

Work with Azure technologies (Databricks, Microsoft Fabric, Data Factory, Azure Data Lake Storage Gen2, Purview, Cosmos DB, OpenAI, Azure ML, AI Foundry, Kubernetes) and languages like Python, Scala, PySpark, and SQL. You will design and deliver enterprise-grade analytics solutions on Azure and Databricks for client data platforms. The role also involves leading small development teams.

Avanade★3.6 (459)· London

Work with Microsoft Fabric, Azure (Synapse, Data Factory, Data Lake Gen2, Purview, Cosmos DB, OpenAI, Azure ML, AI Foundry), Python, Scala, PySpark, and SQL to deliver enterprise-grade analytics and AI data platforms. As part of Avanade’s Data & AI practice, you design and deliver modern, scalable solutions for high-profile clients. Senior Consultant roles may lead small development teams.

ClearBank★3.8 (5)· London

ClearBank seeks an Engineering Manager to lead teams working with C#, Azure, SQL, and related cloud technologies on their banking infrastructure API platform. You'll manage hiring, mentoring, and delivery across multiple projects while occasionally coding, reporting to the Director of Engineering. The role involves leading engineering teams of varying skill levels, collaborating with product partners on prioritization, and fostering communication and security standards.