Data Engineer - Data Pipelines & Warehousing - 6 M...

Company:
Location: Sydney NSW

*** Mention DataYoshi when applying ***

Working for AMP

Working for AMP means being part of a company that values diverse thinking, encourages collaboration and promotes innovation. It’s an environment that offers challenging and exciting work as well as opportunities for professional growth. We’re flexible enough to allow you to make the most of your life, both professionally and personally.

We are looking for those that have the courage and agility to navigate changing and complex environments so that we can deliver the best solutions for our customers. We value people with integrity, an innate willingness to help others and an eagerness to perform to the best of their abilities.

We’re transforming our business, and we need people like you to join us on this journey.

About the role

As a key technical role, reporting to the Lead Development Engineer – Data Pipelines & Warehousing, the Data Engineer – Data Pipelines & Warehousing is accountable for working closely with data & solutions architects, integration engineers, data analysts and testers to design, build and test core extensible data infrastructure and solutions on AWS technologies.

Key responsibilities

  • Partnering with sponsoring projects, architects, data analysts and other design professionals to define and recommend the most appropriate data solutions for a given need.
  • Working with other data engineers and data analysts to build new and extend existing data structures to support real-time and batch data ingestion pipelines, ETL processes and data warehouse and reporting layer data structures.
  • Research and troubleshoot data quality issues, providing fixes and proposing both short and long-term solutions.
  • Developing, maintaining and supporting complex data ingestion patterns using cloud based and open-source technologies.
  • Developing, maintaining and supporting data processing frameworks that can support big data analytics and machine learning.
  • Conducting data modelling and data analysis to help validate business and solution requirements.
  • Helping design unique data driven solutions involving complex business rules.
  • Assisting in the development and maintenance of SODAs service catalogue, actively seeking new ways and approaches to improve the breadth and efficiency of services.
  • Assisting development managers with the estimation of solutions and the prioritisation of functionality against desired cost benefit outcomes.
  • Building relationships and developing platform advocacy across a variety of stakeholders including internal customers, vendor and partners.

Capabilities & Experience

  • Experience in a data warehousing and data engineering
  • Experience in developing, managing, and maintaining large, multi-terabyte data warehouse with a focus on large, complex data ingestion pipelines.
  • Experience of developing cloud-based data ingestion & ETL/ELT data processing frameworks

Technical Capabilities

Highly proficient in:

  • SQL (mandatory, Expert level)
  • Building data applications on AWS technologies (S3, Spectrum, Athena, Lambda, Data Pipeline, EC2, EMR, RDS, Kinesis, Data Stage, Airflow, SageMaker etc.)
  • Structured Language Programming (e.g., Scala (highly desirable), Java, Python, Spark SQL)
  • Data modelling for both transactional and data warehousing environments including familiarity with Kimball dimensional and 3NF modelling standards.

Experience with:

  • Big data technologies like HDFS, Hive, Impala, Kafka, Apache Spark etc.
  • Agile delivery lifecycle tools (Jira)
  • Source control systems (Git, SVN etc.)

Building and supporting Apache Airflow scheduling workflows.

Wellbeing & Benefits

As a company that values wellbeing, we offer a range of great benefits to support you financially, professionally and personally. These include access to a wide range of flexible working options including the ability to purchase extra leave, retail discounts, onsite wellbeing centre including a gym (Sydney Office), Employee Assistance Program, competitive home loan rates, leading superannuation contribution, discounted financial advice and personal insurance.

Inclusion & Diversity

AMP recognises individual differences and welcomes people from a variety of life and work experiences. The diversity of our people is core to our ability to innovate, grow and to fulfil our collective aspiration of helping people to own their tomorrow. A natural curiosity, a respect for differences and a growth mindset are valued at AMP.

*** Mention DataYoshi when applying ***

Offers you may like...

  • Adentis

    Big Data Engineer (Cloudera/Hadoop)
    Lisboa
  • Noesis Portugal - Consultadoria em Sistemas de Informação, SA

    Junior Data Engineer
    Lisboa
  • Mind Source

    Data Engineer
    Porto
  • Adentis

    Big Data Engineer
    Porto
  • Noesis

    Junior Data Engineer
    Lisboa