ETL Data Engineers - Scala / Spark / Hive

DWS is seeking multiple, experienced hands-on ETL Data Engineers / Senior ETL Data Engineers for a major multi-year program of work in Sydney. Working in an embedded team with our client, a major BSFI giant, you will be working towards the migration of their petabyte scale on-premises Hadoop data warehouse to a new AWS solution.

This is a career defining opportunity, to work with arguably the biggest name in their field on a massive digital transformation project that will be the anchor to all subsequent work across the enterprise in future.

Type: Max term

Length: up to 2 years

Salary: $170-200K (inc super), depending on experience

Location: CBD Sydney, NSW

Style: Hybrid, approx. 3-4 days in the office

Start: Immediate

Essential Skills And Experience

• Significant professional experience and expertise with Scala, Spark, and Hive - building Spark/Scala pipelines on Hadoop on-prem and in the cloud
• Significant expertise in performance tuning, and building frameworks at scale
• Abule to quickly understand existing frameworks, then build automations and manage stakeholders for architectural changes
• Recent experience in an enterprise BFSI (banking, financial services, insurance) environment
• Expertise in data migration strategies
• Worked on major enterprise scale data migration projects
• Experience with CI/CD and build tools including SBT and Maven
  
  

Desirable Skills And Experience

• Proficiency in HDFS and YARN
• Worked with AWS: EMR, S3, Glue, Lake Formation, Redshift
• Experience in data governance, security, compliance
  
  

What You’ll Be Doing

• Design and implement data migration strategies from on-premise Hadoop to AWS
• Develop and optimize Hadoop jobs for data processing and migration
• Ensure data quality and integrity throughout the migration process
• Collaborate with stakeholders to understand and document data requirements
• Perform data mapping and transformation tasks as part of the migration strategy
• Implement and manage data governance and security processes
• Troubleshoot and resolve data migration issues and performance bottlenecks
• Monitor and optimise data workflows for performance and cost efficiency
• Develop and maintain documentation for data processes and architecture
• Coordinate with cross-functional teams to align migration activities with project goals
• Conduct performance tuning and optimisation of data processing tasks
• Design and implement automated data pipelines
• Collaborate with AWS data engineering teams as needed
  
  

Applicants must hold Australian Citizenship, Permanent Residency (PR), or other permanent work rights - DWS are unable to sponsor or support visas.

Working At DWS Offers You

• Ongoing professional and career development
• Paid learning and certifications (chosen entirely by you), with completion bonuses
• An active social club that host regular, subsidised events
• Corporate discounts on a range of services and products, including health insurance, gym memberships, Apple products and much more
  
  

About Us

Established in 1991, DWS is a professional Technology Consultancy and part of HCLTech. With offices across Australia, we have over 700 employees in Melbourne, Sydney, Adelaide, Brisbane and Canberra; providing services to a broad range of blue-chip corporate clients, as well as Federal, State and Local Government agencies. We provide a wide range of services including; IT consulting, managed application services, project management, customer driven innovation, digital transformation, data and business analytics, strategic advisory and productivity, and robotic process automation.

Ready for a new challenge? Click APPLY now!

www.dws.com.au

DWS encourages applications from Aboriginal and Torres Strait Islander peoples.

No agencies please - our in-house talent acquisition team have this covered.