Job description

Job details

Posted 22 May 2023 Salary$150000 per annum LocationSydney CBD Job type Discipline ReferenceBH-14327

Job Description

Data Engineer

Full Time Permanent Role

Sydney CBD location with Hybrid Working

Our client is looking a skilled and experienced Data Engineer to join our team. As a Data Engineer, you will play a crucial role in designing, implementing, and maintaining our data infrastructure and analytics solutions. You will collaborate with cross-functional teams to understand their data needs and provide scalable and reliable solutions. The ideal candidate has a strong background in SQL, Azure Data Factory, Azure Databricks (coding in PySpark), data architecture, and experience with various data tools such as DBT, Python, Spark, and Terraform.


  • Design and develop data pipelines and ETL processes using Azure Data Factory, ensuring high-quality and scalable data ingestion from various sources.
  • Build and maintain data models and databases that meet the requirements of the organization, ensuring data integrity and security.
  • Collaborate with cross-functional teams to understand their data needs, provide technical guidance, and deliver efficient data solutions.
  • Implement data quality checks, data validation processes, and data transformation logic to ensure accuracy and reliability of data.
  • Optimize and tune data pipelines and ETL processes for performance and scalability, considering data volume and velocity.
  • Develop and maintain data integration processes using Azure Databricks and PySpark for large-scale data processing and analytics.
  • Collaborate with data scientists and analysts to ensure the availability and accessibility of data for advanced analytics and reporting.
  • Manage and maintain data infrastructure, including data storage, databases, and cloud-based services, ensuring high availability and data security.
  • Stay up-to-date with emerging technologies and industry trends in data engineering and provide recommendations for improvements and optimizations.
  • Document technical specifications, data flows, and architectural designs for future reference and knowledge sharing.

  • Bachelor's degree in Computer Science, Engineering, or a related field. Equivalent work experience will be considered.
  • Proven experience as a Data Engineer or similar role, with a strong background in SQL and database technologies.
  • Experience designing and implementing data pipelines using Azure Data Factory, including data ingestion, transformation, and loading.
  • Proficient in coding with PySpark and working with Azure Databricks for data processing and analytics.
  • Strong understanding of data architecture principles and best practices, including data modeling, database design, and performance optimization.
  • Familiarity with data tools such as DBT, Python, Spark, and Terraform for data processing, transformation, and infrastructure management.
  • Experience working with cloud-based data platforms, preferably Microsoft Azure, and its associated services.
  • Knowledge of data warehousing concepts and technologies, including dimensional modeling and ETL processes.
  • Strong problem-solving skills and the ability to analyze complex data requirements and translate them into practical solutions.
  • Excellent communication and collaboration skills, with the ability to work effectively in cross-functional teams.
Preferred Qualifications:

  • Master's degree in Computer Science, Engineering, or a related field.
  • Relevant certifications in data engineering, cloud platforms, or big data technologies.
  • Experience with version control systems, such as Git, for code management and collaboration.
  • Familiarity with containerization technologies, such as Docker and Kubernetes.
  • Knowledge of data governance and data security practices.

Interested candidates please, reach out to our consultant Waqas Kidwai at

Please let the company know that you found this position on this Job Board as a way to support us, so we can keep posting cool jobs.