Technical Lead Data Engineer (databricks / pyspark...

Technical Lead Data Engineer (Databricks, Pyspark, Delta lake) - Hybrid role Bristol Technical Lead Data Engineer (Databricks, Pyspark, Delta lake) £75,000 - £90,000 Bristol depending upon experience, plus private medical, 28 days annual leave, pension, life assurance, hybrid working with 2 days in the Bristol office and the rest working from home. About Technical Lead Data Engineer (Databricks, Pyspark, Delta lake) A leading reinsurance agency based in Bristol. Their focus is providing innovative, data-driven solutions to the reinsurance industry. By utilising advanced machine learning models, cutting-edge technology, and industry expertise, we transform raw data into actionable insights for our clients. Job Description for Technical Lead Data Engineer (Databricks, Pyspark, Delta lake): They are currently seeking an experienced Senior Data Engineer to join our team and play a critical role in the ongoing development and operationalisation of the companies cloud-based data platform. The successful candidate will work across the platform tackling large-scale data processing challenges and diverse types of data. The role encompasses everything from the initial ingestion of raw data through the application of advanced ML models to the visualisation and reporting of model results. Responsibilities for Technical Lead Data Engineer (Databricks, Pyspark, Delta lake): Design, develop, and maintain scalable data pipelines to ingest, process, and store large volumes of structured and unstructured data from various sources. Collaborate closely with data scientists and business stakeholders to define requirements and implement data processing workflows. Optimise and tune data pipelines for performance (cost and execution time) and reliability, ensuring the timely and accurate delivery of data to stakeholders. Implement and maintain robust data quality and governance processes, ensuring data accuracy, completeness, and compliance. Develop and maintain documentation for data engineering processes and systems. Collaborate with the data science team to operationalise ML models and integrate them into data processing workflows. Develop and maintain data visualisation and reporting solutions to support business decision-making.Requirements Technical Lead Data Engineer (Databricks, Pyspark, Delta lake): A Bachelor's degree in Computer Science, Engineering, or a related field. A minimum of 5 years of experience as a data engineer. At least 1 year of experience using Databricks, including PySpark and Delta Lake. Strong programming skills in Python. Strong skills in SQL and relational databases. Experience with cloud computing platforms, such as GCP, AWS, or Azure. Strong understanding of data warehousing and ETL concepts. Experience with data visualisation tools, such as Tableau, Power BI, or similar. Excellent communication and collaboration skills. Strong problem-solving and critical-thinking abilities.Preferred Qualifications for Technical Lead Data Engineer (Databricks, Pyspark, Delta lake): Experience in the reinsurance or insurance industry. Experience with cloud data movement platforms and modern datastack components such as Fivetran/dbt. Experience with containerisation technologies, such as Docker and Kubernetes. How to apply for Technical Lead Data Engineer (databricks, pyspark, delta lake) Please apply by sending your CV to the link below. Please note unfortunately the client cannot offer sponsorship and you must have full UK right to work to apply