Big Data Engineer

Description

The Big Data & AI team, also known as Data Science team, employs the use of data and Machine Learning algorithms to solve business problems efficiently.
The Data Engineer is a software engineer specialized in data solutions, including technologies to use and scalability of different solutions. This position is a member of the local commercial Big Data & AI team, directly reporting to the Head of Big Data.
The Data Engineer will gather, ingest, transform, clean and load all relevant data sources into a usable format for Data Scientist to later query them, apply transformations and prepare them to create data science models supported by Machine Learning algorithms.

Job Responsibility

• Designs, constructs, installs, tests and maintains highly scalable data management systems, to business requirements and industry practices through the application of data modelling, data warehousing and data manipulation techniques and standards;
• Builds and deploys individual datasets for further modeling based on commercial and/or analytical requirements by extracting, loading and transforming data from different sources;
• Supports integration of new data sources (e.g. batch and streaming) into shared data environment (e.g. EDW, Big Data Platform, etc.);
• Builds high-performance algorithms, prototypes, predictive models and proof of concepts, translating business requirements into concrete data engineering tasks;
• Researches opportunities for data acquisition and new uses for existing data;
• Develops data set processes for data modelling, mining and production
• Integrates new data management technologies and software engineering tools into existing structures
• Creates custom software components and analytics applications;
• Employs a variety of languages and tools (e.g. scripting languages) to marry systems together;
• Recommends ways to improve data reliability, efficiency and quality;
• Collaborates with data architects, modellers and IT team members on project goals;
• Designs, develops and maintains programmes (e.g. written in Python, Scala, Java) and creates reports and dashboards.

Competencies

• Bachelor’s or Master’s Degree (or equivalent) in Computer Engineering or Science background and system creation skills
• Experience in Software Engineering, follows a systematic process of understanding requirements, with Agile/Dev Ops ways of working and using concepts such as TDD, BDD and automation
• Ability to acquire, aggregate, map, maintain and convert data from one format or structure to another supporting data transformation and integration activities so that data is usable by a new program or system
• Experience in Data Modelling with ability to structure, integrate and organize data to create and extend data models for an information system by applying certain formal techniques typically implemented in a database management system
• Experience in testing and evaluation to define the test strategy, which may include test automation, planning and interpretation to validate that the developed system/service meets business requirements and customer explicit and implicit need
• Hands on experience with design, write, debug and maintain the source code of application programmes.
• Experience with innovative automation technologies to automate a repeatable process to help address pain points, streamline business operations or ensuring quality.