Data Engineer

Job details

Posted 22 May 2023 Salary$150000 per annum LocationSydney CBD Job type Discipline ReferenceBH-14327

Job Description

Data Engineer

Full Time Permanent Role

Sydney CBD location with Hybrid Working

Our client is looking a skilled and experienced Data Engineer to join our team. As a Data Engineer, you will play a crucial role in designing, implementing, and maintaining our data infrastructure and analytics solutions. You will collaborate with cross-functional teams to understand their data needs and provide scalable and reliable solutions. The ideal candidate has a strong background in SQL, Azure Data Factory, Azure Databricks (coding in PySpark), data architecture, and experience with various data tools such as DBT, Python, Spark, and Terraform.

Responsibilities

• Design and develop data pipelines and ETL processes using Azure Data Factory, ensuring high-quality and scalable data ingestion from various sources.
• Build and maintain data models and databases that meet the requirements of the organization, ensuring data integrity and security.
• Collaborate with cross-functional teams to understand their data needs, provide technical guidance, and deliver efficient data solutions.
• Implement data quality checks, data validation processes, and data transformation logic to ensure accuracy and reliability of data.
• Optimize and tune data pipelines and ETL processes for performance and scalability, considering data volume and velocity.
• Develop and maintain data integration processes using Azure Databricks and PySpark for large-scale data processing and analytics.
• Collaborate with data scientists and analysts to ensure the availability and accessibility of data for advanced analytics and reporting.
• Manage and maintain data infrastructure, including data storage, databases, and cloud-based services, ensuring high availability and data security.
• Stay up-to-date with emerging technologies and industry trends in data engineering and provide recommendations for improvements and optimizations.
• Document technical specifications, data flows, and architectural designs for future reference and knowledge sharing.

Qualifications:

• Bachelor's degree in Computer Science, Engineering, or a related field. Equivalent work experience will be considered.
• Proven experience as a Data Engineer or similar role, with a strong background in SQL and database technologies.
• Experience designing and implementing data pipelines using Azure Data Factory, including data ingestion, transformation, and loading.
• Proficient in coding with PySpark and working with Azure Databricks for data processing and analytics.
• Strong understanding of data architecture principles and best practices, including data modeling, database design, and performance optimization.
• Familiarity with data tools such as DBT, Python, Spark, and Terraform for data processing, transformation, and infrastructure management.
• Experience working with cloud-based data platforms, preferably Microsoft Azure, and its associated services.
• Knowledge of data warehousing concepts and technologies, including dimensional modeling and ETL processes.
• Strong problem-solving skills and the ability to analyze complex data requirements and translate them into practical solutions.
• Excellent communication and collaboration skills, with the ability to work effectively in cross-functional teams.

Preferred Qualifications:

• Master's degree in Computer Science, Engineering, or a related field.
• Relevant certifications in data engineering, cloud platforms, or big data technologies.
• Experience with version control systems, such as Git, for code management and collaboration.
• Familiarity with containerization technologies, such as Docker and Kubernetes.
• Knowledge of data governance and data security practices.

Interested candidates please, reach out to our consultant Waqas Kidwai at waqaskidwai@future-you.com.au