Senior Databricks Engineer – Capgemini (Glasgow, Hybrid)

Duration: contract until 31/12/2026

Location: Glasgow, hybrid 2‑3 days per week onsite

Rate: up to £414 per day (IR35 Umbrellainside)

This role is focused on migrating existing AWS‑based data pipelines to Databricks and building scalable, cost‑efficient data solutions using Apache Spark on Databricks.

Responsibilities

• Lead the migration of existing AWS data pipelines to Databricks.
• Design and implement scalable data engineering solutions using Apache Spark on Databricks.
• Collaborate with cross‑functional teams to translate data requirements into efficient pipelines.
• Optimize performance and cost‑efficiency of Databricks workloads.
• Develop and maintain CI/CD workflows for Databricks using GitLab or similar tools.
• Ensure data quality and reliability through robust unit testing and validation frameworks.
• Implement best practices for data governance, security, and access control within Databricks.
• Provide technical mentorship and guidance to junior engineers.

Must‑Have Skills

• Strong hands‑on experience with Databricks and Apache Spark (preferably PySpark).
• Proven track record of building and optimizing data pipelines in cloud environments.
• Experience with AWS services such as S3, Glue, Lambda, Step Functions, Athena, IAM, and VPC.
• Proficiency in Python for data engineering tasks.
• Familiarity with GitLab for version control and CI/CD.
• Strong understanding of unit testing and data validation techniques.

Preferred Qualifications

• Experience with Databricks Delta Lake, Unity Catalog, and MLflow.
• Knowledge of CloudFormation or other infrastructure‑as‑code tools.
• AWS or Databricks certifications.
• Background in Finance Industry.

• Seniority level: Mid‑Senior level
• Employment type: Contract
• Job function: Information Technology

All profiles will be reviewed against the required skills and experience. Due to the high number of applications we will only be able to respond to successful applicants in the first instance. We thank you for your interest and the time taken to apply!