Job description

Location: Hybrid UK

Cox Automotive is the world’s largest automotive service organisation, providing solutions to our clients at every stage of the vehicle lifecycle. Through our unique insight, based on data spanning the wholesale, retail and consumer sectors, we help dealers, manufacturers, fleet and leasing companies to boost the value of their assets from cradle to grave, improving performance and profitability at every stage.

As a family business we appreciate the value of great people working brilliantly together, so we’ve always focused on achieving it. It means our teams and partners are the brightest minds in this sector, delivering the best innovations using a powerful mix of knowledge, ingenuity and close collaboration. The results speak for themselves: many thousands of satisfied clients, a string of the industry’s most popular solutions, and even an award or two!

Lead Data Engineer

We are looking for a Lead Data Engineer to join and lead our small (but mighty) Data Development team. As a team we are responsible for the technical development and management of our data and analytics platforms, applications, data models, and data assets. Together with our colleagues in Data Science and Business Intelligence, we make up Data Services, whose collective mission is to enable smarter decisions, faster.

This is a fantastic opportunity for either an experienced Lead Engineer to take the reins or a talented Senior Data Engineer looking to step up and take on leading data engineering projects and helping shape the future of our data platform. This is more than just building data pipelines – ultimately our goal is to maximise the potential of our Data Science and Business Intelligence teams, so we need to be one step ahead, innovate, and provide them with tools before they know they need them. [FJ(MSG1]

We are currently beginning the process of re-engineering our data platform to use a fully Databricks native architecture, using tools like Delta Lake, Delta Live Tables, and multistage pipelines to simplify and modernise our estate further. This role will have the freedom to drive this transition forward and put a stamp on how we ingest, process and store data for years to come.

About our platform and tools :

  • 100% PaaS architecture in Azure
  • Azure Databricks platform, with data stored in Azure Data Lake Storage
  • Scala-Spark data pipelines, moving towards PySpark and Delta Lake
  • Several open-source Spark libraries released to the Spark community
  • Constantly reviewing our tools and architecture as technology evolves

What you’ll do :

  • Lead the data engineering team, including line management of Data Engineers
  • Lead project delivery, including solution design, development, testing and maintenance
  • Identify improvement opportunities in our platform/tools and see them through to completion
  • Contribute and manage contributions to our open-source projects: have a look at
  • Be the technology SME and consult with the Data Science, Business Intelligence and wider Software Engineering teams on how to maximise the performance of their applications
  • Explore and prototype the latest technical innovation within Big Data, constantly shaping Data Solutions' platform to keep pace with the industry

About you :

We are looking for someone with:

  • Technical experience working in Big Data
  • Experience developing and delivering data projects using Apache Spark in Scala or Python
  • Strong Apache Spark skills, as well as experience with Hive and other Big Data tooling
  • Strong Scala and/or Python skills
  • Strong focus on writing maintainable software (TDD, abstraction of common behaviour, code reviews)
  • Able to effectively interact and communicate both verbally and written when working with business and technical teams

It would also be good if you had or are interested in (but don’t worry if you’ve got none of these, or only some of them):

  • A desire to work with both Scala and Python
  • Experience mentoring junior employees
  • Experience with functional programming techniques and approaches
  • Experience building and maintaining infrastructure in Azure (although any public cloud knowledge would be useful)
  • Devops knowledge, potentially with infrastructure as code experience (Terraform or Pulumi)
  • Appreciation of other programming languages in the data and Azure world: C# (.net core), R, TypeScript/JavaScript
  • Experience building, deploying and maintaining serverless applications (we’re experimenting with Azure functions)
  • Flexible Working Hours
  • Flexible Holiday
  • Vehicle Discount
  • 25 Days Annual Leave Plus Bank Holidays
  • Casual Dress Environment
  • Best Kit for the Job
  • Pension Scheme
  • Life Insurance
  • Bonus Scheme


We work with a carefully selected set of recruitment agencies and we're not looking to add to our PSL.

We do not accept unsolicited agency CV's sent to the recruitment team or directly to the hiring manager. We will not be responsible for any fees related to unsolicited CV's


Please let the company know that you found this position on this Job Board as a way to support us, so we can keep posting cool jobs.