Principal Data Engineer (Outside IR35, £600/day)

Job Description

Contract Job Description: Principal Data Engineer (Outside IR35, £600/day)

We are seeking a highly experienced Principal Data Engineer for a short-term contract role, offering £600/day, outside IR35. The successful candidate will have expertise in designing and implementing scalable data platforms and will contribute significantly to data engineering projects.

Key Responsibilities

• Architect & Build Scalable Data Platforms: Lead the design and development of a scalable data platform on AWS and Snowflake, including star schema modeling for NLP and Machine Learning (ML) model deployment.
• ML Model Deployment & Automation: Build scalable ML inference serving architecture and automated (re)training pipelines using MLflow to streamline ML workflows.
• Data Ingestion Pipelines: Develop and optimize scalable data ingestion pipelines to integrate data from external vendors such as CapitalIQ, Beamery, and Similarweb.
• Infrastructure Automation: Incrementally enhance the data warehouse, utilizing Terraform for AWS and Snowflake infrastructure deployment, and replace Glue Scheduler with Airflow (MWAA) for efficient job orchestration.
• Tool Implementation & Training: Deploy DBT Cloud and Terraform Cloud, and provide comprehensive training on DBT, MLflow, and Terraform to the wider team.
• Data Modeling & Testing: Actively contribute to Data Modeling processes, establishing robust data tests and creating a data testing platform to ensure high-quality data.
  
  

Key Skills & Experience

• Proven experience architecting and deploying scalable data platforms on AWS and Snowflake.
• Deep understanding of ML model deployment, scalable ML architecture, and automated training pipelines.
• Expertise in data ingestion processes and integrating third-party vendor data.
• Strong knowledge of Terraform, Airflow, DBT Cloud, and MLflow.
• Experience with data modeling and creating automated data tests.
• Strong communication skills with the ability to provide training and knowledge transfer.
  
  

Contract Details

• Day rate: £600/day
• Location: Remote
• Duration: 3 months
• Outside IR35
  
  

If you're a seasoned data engineer with a passion for building robust and scalable data platforms, we want to hear from you!