Data Engineer

About Us

Canberra Consulting focuses on delivering highly skilled and qualified professionals to our clients. Canberra Consulting has the experience and the capability to understand your specific requirements and match them to the right environment, ensuring professional satisfaction and ongoing development opportunities. Our focus is ensuring that both you and our client are a good match and that we give you every opportunity to succeed.

The Opportunity

We're on the hunt for a skilled Data Engineer who's well-versed in Spark and Kafka to join forces with one of our esteemed Federal Government Clients. As a Data Engineer, you'll dive into the world of pioneering technologies and team up with colleagues from various disciplines to guarantee seamless data integration.

Key Responsibilities Include

• Manage Spark pipelines for processing extensive datasets
• Ingest data from Kafka into our data lake using various processing techniques
• Execute Spark jobs for data ingestion, curation, and extraction
• Adapt schemas and configurations to align with business needs and maintain data quality
• Collaborate in meetings on data ingestion, fostering cross-functional integration
• Monitor and troubleshoot performance using logging and monitoring tools
• Establish best practices and standards for data processing pipelines
• Stay informed about emerging trends and technologies in big data processing and analytics

To Be Successful In This Role, You Will Need

• 3+ years of experience in data engineering, software development, or a related field
• Expertise in Spark, Kafka, and big data processing concepts/strong programming skills in Scala, Python, or Java
• Experience with data storage solutions, such as Hadoop HDFS or Amazon S3, data serialisation formats and version control systems, such as Git
• Knowledge of database concepts, data modelling, schema design, and normalisation

All candidates applying for this position must be an Australian Citizen and have a minimum of current Baseline security clearance. Negative Vetting Level 1 preferred.

Does this sound like a perfect role for you? Interested in hearing more? Give our Recruitment Consultant, Samantha Gibbs a call on 0427 187 777 or an email at Samantha.gibbs@canberraconsulting.com.au, for a confidential discussion.