May 23, 2023
Apply Now
Brisbane, Queensland, Australia, Australia
May 23, 2023
Apply NowJob description
Who is AEMO?
We are the Australian Energy Market Operator. AEMO’s role is to design and operate a sustainable energy system that provides safe, reliable, and affordable electricity and gas today, and to enable the energy transition for the benefit of all Australians. All Australian governments are now committed to a net-zero economy for Australia by 2050, the first step of which is a net-zero energy system. We are collaborating with industry and other stakeholders to engineer electricity grids that can handle 100 per cent instantaneous renewable generation by 2025. AEMO also plays a significant role in the east coast gas market, which has recently expanded to assessing if there is enough natural gas for customers, including electricity generators. The transition to a renewable energy system is occurring at a world-leading pace. It is an exciting, momentous time to be part of this mission.
About The Role
As a Data Engineer your role will be part of the Enterprise Data Services Team and you will be responsible for co-designing, building, implementing, debugging, and supporting the Data ETL processes. This will include data extraction, ingestion, modelling, and transformation using the latest technology stack including Databricks, Delta Lakehouse, Streaming-based Data Ingestion.
The Data Engineer will also ensure that this data modelling will be satisfying the business requirements, and the Data Lakehouse is reliable and robust in terms of stability, availability, consistency, and efficiency as a product to be used by various business teams.
This role will be supporting additional teams who are using Azure/Databricks as their platforms for purposes such as statistical simulation, machine learning and data quality and consumption problems. You will be responsible for working with the business to understand business logic and requirements, to provide solutions and trouble shooting.
This is a 12 month Fixed Term Contract, with location in either Melbourne, Brisbane or Norwest
What You Will Be Doing
At AEMO, our core values aren’t just ‘words on a wall’ – we are a purpose-based organisation at the centre of our energy future, and we stand for integrity, inclusiveness, empowerment, and excellence.
As we continue to innovate and work on some world-first, ground-breaking projects, we need to attract the brightest people and provide opportunities to contribute, stretch and grow. We actively encourage you to bring your whole self to work, so that together we can do better, be better, and love our work. For this reason, inviting difference and leading inclusively is a core part of our culture. We challenge ourselves to take bold actions to improve the experience for all our people, with a current focus on gender, flexibility, (dis)ability and our First Nation peoples.
If you would like to know more about working at AEMO, please check out our careers page for more information
We are the Australian Energy Market Operator. AEMO’s role is to design and operate a sustainable energy system that provides safe, reliable, and affordable electricity and gas today, and to enable the energy transition for the benefit of all Australians. All Australian governments are now committed to a net-zero economy for Australia by 2050, the first step of which is a net-zero energy system. We are collaborating with industry and other stakeholders to engineer electricity grids that can handle 100 per cent instantaneous renewable generation by 2025. AEMO also plays a significant role in the east coast gas market, which has recently expanded to assessing if there is enough natural gas for customers, including electricity generators. The transition to a renewable energy system is occurring at a world-leading pace. It is an exciting, momentous time to be part of this mission.
About The Role
As a Data Engineer your role will be part of the Enterprise Data Services Team and you will be responsible for co-designing, building, implementing, debugging, and supporting the Data ETL processes. This will include data extraction, ingestion, modelling, and transformation using the latest technology stack including Databricks, Delta Lakehouse, Streaming-based Data Ingestion.
The Data Engineer will also ensure that this data modelling will be satisfying the business requirements, and the Data Lakehouse is reliable and robust in terms of stability, availability, consistency, and efficiency as a product to be used by various business teams.
This role will be supporting additional teams who are using Azure/Databricks as their platforms for purposes such as statistical simulation, machine learning and data quality and consumption problems. You will be responsible for working with the business to understand business logic and requirements, to provide solutions and trouble shooting.
This is a 12 month Fixed Term Contract, with location in either Melbourne, Brisbane or Norwest
What You Will Be Doing
- Implement and orchestrate end-to-end Data ETL Pipelines
- Provide support to business application from the data consumption perspective (SQL queries, DB connection, etc.)
- Troubleshooting performance issues and provide solutions for optimisation
- Monitoring, debugging for both streaming-based and batch-based ETL Pipelines
- Assist team members with code reviewing, change request peer review and knowledge sharing
- Assist the leadership team with decisions regarding the Enterprise Data Platform
- Bachelor’s Degree in computer science, mathematics, information technology, or any branch of engineering
- Previous experience with Databricks for designing, implementing, and debugging data transformation solutions
- Previous experience with Python programming language and using PySpark is essential
- Proven experience with Spark Structured Streaming on Databricks platform
- Experience with performance optimisation by analysing Spark UI and other logs provided by Databricks
- Proven experience with SQL, complex SQL queries writing, NoSQL and semi-structured data such as JSON and XML
- Experience with DevOps and Git version control concepts and can leveraging CICD pipeline to deploy solutions for Databricks and Data Factory, including exposure to Visual Studio (SSDT) and DACPAC
- Data Engineering certificates will be appreciated, preferably certified by Databricks and MS Azure
- R programming experiences, statistical simulation and machine learning is desirable
- Flexible working: work from home, part time, job share, hybrid options, and additional leave options
- Professional development via projects, assignments, industry networks, job rotation, study assistance and formal learning
- A full health and wellbeing program, discounted health insurance and other concessions
- Free salary continuance insurance and annual performance bonuses (subject to eligibility)
- Up to 4 days volunteering leave per year
- A comprehensive and professional Employee Assistance Program (EAP)
At AEMO, our core values aren’t just ‘words on a wall’ – we are a purpose-based organisation at the centre of our energy future, and we stand for integrity, inclusiveness, empowerment, and excellence.
As we continue to innovate and work on some world-first, ground-breaking projects, we need to attract the brightest people and provide opportunities to contribute, stretch and grow. We actively encourage you to bring your whole self to work, so that together we can do better, be better, and love our work. For this reason, inviting difference and leading inclusively is a core part of our culture. We challenge ourselves to take bold actions to improve the experience for all our people, with a current focus on gender, flexibility, (dis)ability and our First Nation peoples.
If you would like to know more about working at AEMO, please check out our careers page for more information
