Lead Databricks Engineer - #2179272

Lead Databricks Engineer

Sector: Finance

Contract: 6 months | Outside IR35

Rate: £600

Location: Leeds, with 2 days per week on site

The role

We are looking for a hands-on Lead Databricks Engineer to lead the delivery and improvement of data pipelines within an Azure Databricks environment. The platform uses a medallion architecture across its tables and views, with a strong focus on Python, reliable data processing and engineering quality.

You will combine technical leadership with development, helping the team make sound design decisions, investigate loosely defined requirements and deliver well-tested solutions.

Key responsibilities

• Lead the design, development and optimisation of Databricks pipelines using Python, PySpark and SQL.
• Build and maintain Delta Live Tables pipelines, managed tables and views across bronze, silver and gold layers.
• Define and implement data reconciliation and quality checks, investigating discrepancies between source data and downstream outputs.
• Set clear standards for testing, documentation and peer review, providing technical guidance to other engineers.
• Use Git feature branches, pull requests and code reviews to manage changes and merge approved work back into main.
• Investigate ambiguous tickets, clarify requirements with stakeholders and translate findings into practical delivery tasks.
• Work with Azure Data Factory where required and contribute to the adoption of Lakeflow Connect for ingestion.
• Explore appropriate uses of Databricks Genie and other AI capabilities within the data platform.

Essential experience

• Experience in Payments, Financial Services, Banking and / or Fintech
• Strong hands-on Azure Databricks and Python experience, with the ability to lead technical delivery.
• Experience with Delta Live Tables, Delta Lake, managed tables and medallion architecture.
• Strong SQL skills and experience developing and troubleshooting data transformations.
• A practical approach to data reconciliation, pipeline testing, data quality and peer review.
• Experience with Git branching and collaborative development workflows.
• Ability to work independently through ambiguity and communicate technical decisions clearly.
• Working knowledge of Azure Data Factory.

Desirable experience

• PySpark experience building and optimising distributed data processing.
• Lakeflow Connect experience, particularly for data ingestion.
• Practical use of Databricks Genie or other AI tools in data engineering.