Data Engineer

Job Title: Data Engineer Location: Shoreditch (Hybrid) Duration: 12 months contract Salary: £52k-£62k per annum + paid holidays + pension The Challenge: Leverage your expertise in data engineering to drive the next stage of growth at client. The Customer Analytics team is focused on using the power of data to deliver improved insights into our trading and global expansion teams. This role will drive data engineering for a variety of analytical and BI initiatives across a wide variety of strategic projects. As our lead Data Engineering expert, you will have significant responsibility to drive the building out of cloud based multi-faceted datasets, ensuring their provenance and governance are upheld to a high standard. This role is inherently cross functional, and the ideal candidate will work across disciplines. The position requires ability to own things, come up with innovative solutions, try new tools, technologies, and entrepreneurial mindset. What you will do: Build fault tolerant, scalable, quality data pipelines using cloud- based tools. Innovative solutions to help broader organization take decisive actions fast and efficiently. Be a data steward. Develop deep organization wide data understanding. Contribute to data engineering frameworks, tools, and processes Implement discipline around data operations and implement best practices to use resources in an optimum way. Architect data ingestion, data transformation, data consumption, data governance frameworks. Work in a collaborative environment and contribute to the team as well as organization’s success. What you will need: 5+ years of experience as a data engineer. 3+ years demonstrated ability in distributed data technologies e.g., Hadoop, Hive, Presto, Spark etc. 2+ years of experience with Cloud based technologies – Databricks, AWS EMR, S3, Azure Blob Storage, Notebooks etc. Familiarity and usage of different file formats in batch/streaming processing i.e., Delta/Parquet/Avro/ORC etc. Best in class SQL experience. Ability to write complex SQLs across platforms. Solid hands - on experience in Python/PySpark/Scala and ability to manipulate data using Pandas, NumPy, Koalas etc. and using APIs to transfer data. Experience working as an architect to design large scale distributed data platforms. Working experience with CI/CD tools i.e., GitHub, Jenkins etc. Working experience with Open-source orchestration tools i.e., Apache AirFlow / Azkaban etc. Teammate with excellent communication/collaboration skills when it comes to closely working with analysts, data scientists and machine learning engineers daily. Experience of working in commercial organisations and having a good understanding of the questions and information a business needs to make better decisions. Nice to have: Familiarity with Machine Learning Algorithms. Showcase your work if you are an open - source contributor. Passion to contribute to Open-source community is highly valued. Experience with Data Governance tools e.g., Collibra and Collaboration tools e.g., JIRA, Confluence etc.