Senior Data Engineer

Senior Data Engineer (AI/ RAG Pipeline)

London, hybrid

About RedCompass Labs:

At RedCompass Labs, we believe there are only two types of payments: good and bad. We enable the good payments and help stop the bad ones.

We are global experts in payments modernization, using the latest applied AI, and deep payments expertise to deliver end-to-end payment transformation projects. Our solutions help clients accelerate their payments modernization programs while reducing costs and regulatory risk.

The RedFlag Accelerator sets the industry standard for detecting human trafficking related crime, assisting both financial institutions and law enforcement. Using a persona-oriented approach, it identifies the actors behind transactions, including victims, money-laundering professionals, perpetrators, and others. By combining AI, the latest red flags, crime models, and OSINT data, our system helps detects crimes across labour, sex trafficking and child sexual exploitation.

With offices in London, Warsaw, Brussels, Miami, Toronto, and Pune, we support clients worldwide. To learn more about RedCompass Labs and our services, visit www.redcompasslabs.com

Mission:

Imagine reducing a 14-day financial analysis task to just 2 hours. At RedCompass Labs, we don't just imagine-we've built it.

We're transforming the trillion-dollar payments industry through intelligent automation. At the heart of our mission is AnalystAccelerator.ai, our proprietary multi-agent AI platform built to accelerate complex analysis in global banking and compliance.

We’re seeking an exceptional Senior Data Engineer to design, build, and own the data pipeline that powers our payments expert agent. Your role: take raw, messy domain content – payments handbooks, regulatory documents, scheme rulebooks, project delivery history – and turn it into a production-grade retrieval layer that an AI agent can reason over reliably. You’ll own this end-to-end, from ingestion through to the RAG interface, in a regulated, high-stakes environment.

As a Senior Data Engineer, you would:

You will own the data pipeline that turns regulated payments domain content into a reliable retrieval layer for our AI agent:

Design, build, and scale the ingestion and processing pipelines that move payments domain content – handbooks, scheme rulebooks, regulatory documents, project history – into structured, retrievable knowledge.

Engineer robust Retrieval-Augmented Generation (RAG) pipelines including chunking, embedding, vector storage, and retrieval, tuned for dense regulatory and technical content.

Stand up cloud-native data infrastructure on Azure – Data Factory, Functions, Blob/ADLS, CosmosDB, and AI Search – with Python as the primary language.

Embed engineering rigour into the pipeline – CI/CD with GitHub Actions and Azure DevOps, containerisation, observability, data quality checks, and re-runnable workflows.

Collaborate closely with the AI Engineer, Data Architect, and payments subject matter experts to translate messy domain content into a retrieval layer the agent can reason over.

Candidate Profile:

We’re looking for a senior data engineer who can own a greenfield pipeline end-to-end – design it, build it, and run it in production.

5+ years of professional data engineering experience, with a strong track record of designing and operating production data pipelines.

Deep hands-on experience building ingestion and processing pipelines for unstructured and semi-structured content (documents, transcripts, structured data), including parsing, chunking, and metadata enrichment.

Strong proficiency in Python.

Hands-on experience with embedding models, Retrieval-Augmented Generation (RAG) architectures, and vector search (e.g. Azure AI Search, pgvector, or equivalent).

Strong working knowledge of the Azure data stack – Data Factory or Synapse, Functions, ADLS/Blob, CosmosDB, AI Search – and comfort with containerised workloads.

Comfortable making architectural decisions on incomplete information, and willing to revise them as the pipeline meets real data.

A ‘t-shaped’ engineering mindset, and a willingness to stretch into adjacent work – DevOps, retrieval evaluation, prompt tuning, backend APIs – when the work needs it.

Bonus Qualifications:

Experience with document processing at scale – OCR, layout-aware parsing, or table extraction from PDFs.

Background in fintech, banking, payments, or compliance industries.

Experience building evaluation tooling for retrieval quality (recall@k, faithfulness, regression tests on a curated eval set).

Our Technology Ecosystem:

Pipeline & processing core: Python, orchestration frameworks such as Azure Data Factory, Airflow, Dagster, etc.

Retrieval-Augmented Generation (RAG) systems and Azure AI Search for vector and hybrid retrieval.

Storage & data platform: Azure Blob/ADLS, CosmosDB, Azure AI Search, with FastAPI for the retrieval API layer.

Ops & quality: GitHub Actions, Azure DevOps, Docker, with observability and data quality checks across the pipeline.

What We Offer

Work with real purpose. At RedCompass Labs, your projects carry weight beyond the delivery plan. Our work sits at the intersection of global financial infrastructure and some of the most important crime-fighting efforts in the world. Through our RedFlag Accelerator, we help financial institutions identify the financial traces of human trafficking and child sexual exploitation - and the Project Leads who keep our client engagements on track are a vital part of making that possible. This is delivery leadership with a genuine reason to care.

A place where your judgement is trusted. RedCompass Labs is built on expertise, not hierarchy. As a Project Lead, you will have real ownership of client relationships and delivery outcomes - not a supporting role on someone else's programme. Your perspective will be sought, your decisions respected, and your contribution visible at the highest levels of the organisation.

A team that genuinely invests in its people. We are committed to your development - not just your output. You will have access to a dedicated Career Development Manager, structured performance reviews, and a culture that encourages continuous learning. Whether you are an emerging Project Lead building your foundation or an experienced practitioner ready for a more complex challenge, we will meet you where you are.

A global community of specialists. With colleagues across London, Warsaw, Brussels, Toronto, Pune, and Miami, you will work alongside some of the sharpest minds in payments. Our engagements span institutions, geographies, and regulatory environments - no two projects are alike, and the breadth of exposure is consistently one of the things our people value most about working here.

Competitive benefits package:

Up to 10% of annual earnings as a personal performance bonus

Life insurance

Group Income Protection

Health Insurance for you and your family

Dental Insurance for you and your family

Pension: the pension is currently 4% employer and 4% employee. You also have the option to increase your contributions to your private pension, if you wish.

28 days annual holiday plus Public & Bank holidays

7 days of sick leave paid 100% per annum

RedCompass Labs is committed to promoting and supporting a diverse and inclusive workplace, ensuring fair and equitable treatment for all. This fuels innovation and enables us to work better together with each other and for our clients