Senior Data Engineer - ADF & Data Lake

Where you fit in

This role is within Shell's Trading and Supply business, one of the world's largest energy trading operations in the world with key hubs in Singapore, Dubai, Houston, London, and Rotterdam. Based in Shell Energy Europe and Environment Products, the role supports trading across European Power, Gas, regulated Carbon and global weather markets, with a focus on the net zero agenda.

The role involves creating optimal data pipeline architecture, assemble large, complex data sets that meet functional / non-functional business requirements. You will work with stakeholders including the Executive, Product, Data and Design teams to assist with data-related technical issues and support their data infrastructure needs. The role offers opportunities to work with the latest technologies, top-notch professionals, and hands-on exposure to complex projects which is bound to help one’s career growth.

What’s The Role

• Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
• Build the infrastructure required for optimal ETL/ELT of data from a wide variety of data sources using SQL and Azure, AWS ‘big data’ technologies
• Build analytics tools that utilize the data pipeline to provide actionable insights into customer acquisition, operational efficiency and other KPI metrics
• Keep our data separated and secure across national boundaries through multiple data centres and Azure, AWS regions
• Create data tools for analytics and data scientist team members that assist them in building and optimizing our product into an innovative industry leader
• Work with data and analytics experts to strive for greater functionality in our data systems

What We Need From You

• 9+ years of experience in a Data Engineer role and 4+ years of relevant experience in Azure Data Factory (ADF), Azure Data Lake Storage (ADLS), Databricks, PySpark, and Spark SQL
• Hands-on experience with Stream Analytics, SQL DW, COSMOS DB, Analysis Services, Azure Functions, Serverless Architecture, and ARM Template
• Has technical expertise with relational SQL/NoSQL databases, file handlings and API integrations
• Good background with object-oriented/object function scripting languages: Python, SQL, Scala etc.
• Nice to have experience with any of these toolsets: Kafka, Stream sets, Alteryx, HANA, SLT and BODS
• Advanced working SQL knowledge, experience working with relational databases, query authoring (SQL) as well as working familiarity with a variety of databases would be good to complement
• Experience performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement would be an advantage