Senior Data Engineer (AASvW)

Company:
Location: Johannesburg, Gauteng

*** Mention DataYoshi when applying ***

Profile Outline

A technology focussed individual, data engineers will need to design, configure, develop and deploy data transformations.

Key Responsibilities

Develop ETL pipelines. The data transformations will be developed in Azure Databricks using Python and on Azure SQL using T-SQL and deployed using ARM templates

Combine and curate data in a central data lake

Serve data for application and analytics through a variety of technologies such as SQL, Server Synapse, CosmosDB and TSI

Build transformation pipelines into dimensions and facts and therefore a strong knowledge of standard BI concepts is mandatory

Build stream pipelines leverage IoT Hub, Event Hub, Databricks streaming and other Azure stream technologies

Work in a fluid environment with changing requirements whilst maintaining absolute attention to detail.

Key Skills:
Python

PySpark

SQL

Solution Architecture

API Design

Containers

CI/CD

Azure Cloud

Data stream patterns and technology

Data engineering design patterns

Qualifications:
Bachelor Degree in Computer Science, Software Engineering or Engineering

Specialist Areas:
Unstructured Data

Applies to /wiki/spaces/DAGDG/pages/1473251810 and any products that handles large scale data such as images.

Strong experience of building large scale file shipping and pipelines, ideally using Azure services such as AzCopy and Azure Data Lake.

Experience of managing unstructured file meta-data, conversions, standardisation and related workflows.

Experience of building analysis jobs that scale on technologies such as Databricks or Azure Batch.

*** Mention DataYoshi when applying ***

Offers you may like...

  • Trianz

    Senior Data Analyst
    Remote
  • Bolt

    Senior Data Analyst, Product
    Warszawa, mazowieckie
  • ATB Financial

    Senior Data Analyst
    Remote
  • The Hong Kong Jockey Club

    Senior Data Analyst (Treasury)
    Happy Valley, Hong Kong Island
  • BIPO SERVICE (SINGAPORE) PTE. LTD.

    Senior Data Analyst
    Tanjong Pagar