Data Engineer

Company:
Location: Melbourne VIC

*** Mention DataYoshi when applying ***

Position Summary:

We are looking for a Big Data/Cloud Data Engineer who will work on the collecting, storing, processing, and analyzing of huge sets of data. The primary focus will be on choosing optimal solutions to use for these purposes, then maintaining, implementing, and monitoring them. You will also be responsible for integrating them with the architecture used across the company.

  • 4+ years demonstrated coding experience in Spark Streaming, Batch framework & hive/impala. (Language such as Scala/Java)
  • 2+ years expertise in writing applications or APIs using Apache Kafka, AWS Lambda Function, Azure Databricks.
  • Strong experience in working with Spark (Spark Streaming & SQL), Scala, Hive & Impala.
  • Demonstrated hands-on experience in working with Databricks and Notebooks.
  • Demonstrated experience in design and build real-time streaming pipeline using AWS kinesis, lambda, Azure EventHub & Azure synapse
  • Solid Exposure/Understanding of a hybrid data platform (both on premise & Cloud) including Azure and AWS services.
  • Design, Engineer and innovate cloud-based solutions in multi-cloud scenarios across the Microsoft Azure/AWS services.
  • Hands on implementation of end-to-end data pipelines using Azure Data Lake, Azure Data factory, Azure Databricks.
  • Expertise with database services & data lake such as Azure Synapse, Cosmos DB, Azure SQL, Azure Blob Storage, and Azure Data Lake Storage Gen2.
  • Experience in SCM tools – Atlassian Bitbucket, GitLab, Azure Repos
  • Solid experience in setting up advanced deployment techniques like CI/CD using Atlassian Bamboo & Azure DevOps (IAC deployments through Terraform, CloudFormation would be added advantage)
  • End to End project execution, Requirement gathering, transforming legacy design to BigData eco system. Development, Testing, UAT Support and GO Live support.
  • Understanding and creating PowerBI reports would be additional advantage.
  • Inhouse Proprietary Tool - 2+ years of working experience with JOF Job, Merlin and JOF Ingestion framework to develop data ingestions pipelines and deploy the code artifact.
  • 4+ years demonstrated coding experience in Spark Streaming, Batch framework & hive/impala. (Language such as Scala/Java)
  • 2+ years expertise in writing applications or APIs using Apache Kafka, AWS Lambda Function, Azure Databricks.
  • Strong experience in working with Spark (Spark Streaming & SQL), Scala, Hive & Impala.
  • Demonstrated hands-on experience in working with Databricks and Notebooks.
  • Demonstrated experience in design and build real-time streaming pipeline using AWS kinesis, lambda, Azure EventHub & Azure synapse
  • Solid Exposure/Understanding of a hybrid data platform (both on premise & Cloud) including Azure and AWS services.
  • Design, Engineer and innovate cloud-based solutions in multi-cloud scenarios across the Microsoft Azure/AWS services.
  • Hands on implementation of end-to-end data pipelines using Azure Data Lake, Azure Data factory, Azure Databricks.
  • Expertise with database services & data lake such as Azure Synapse, Cosmos DB, Azure SQL, Azure Blob Storage, and Azure Data Lake Storage Gen2.
  • Experience in SCM tools – Atlassian Bitbucket, GitLab, Azure Repos
  • Solid experience in setting up advanced deployment techniques like CI/CD using Atlassian Bamboo & Azure DevOps (IAC deployments through Terraform, CloudFormation would be added advantage)
  • End to End project execution, Requirement gathering, transforming legacy design to BigData eco system. Development, Testing, UAT Support and GO Live support.
  • Understanding and creating PowerBI reports would be additional advantage.
  • Inhouse Proprietary Tool - 2+ years of working experience with JOF Job, Merlin and JOF Ingestion framework to develop data ingestions pipelines and deploy the code artifact.

Mandatory Skills:

  • Big Data Stack: - Hadoop, Spark Streaming, Kafka, Hive, Impala, Hue, Kudu, Scala, Java, Python, Shell, HBase, Flume, PowerBI, Cloudera Manager, Ambari, IBM Control-M.?
  • Azure Stack: - Azure Function, Azure EventHub, Azure HDInsight, Azure Data Bricks, Azure VM, Azure Data Factory, Azure storage, Azure Data Lake, Azure?Cosmos DB, Azure KeyVault, Azure SQLDW, Synapse, Azure Policy, Azure Blueprint, Logic App.
  • AWS Stack: - Kinesis, Lambda, S3, EC2, DynamoDB, API Gateway, VPC, IAM roles & Policies
  • Dev-ops Tools: - Terraform, CloudFormation, Azure DevOps, JIRA, GIT, Bitbucket, Bamboo, Jenkins, Confluence.?
  • Client Specific: - JOF Job, Merlin, JOF Ingestion framework and Paul tool
  • Change Management: - IBM iTAM

Duties and Responsibilities:

  • Associate would be responsible for design, code, test, document and maintain high-quality and scalable Big Data solutions on-prem or on-cloud.
  • Research, evaluate and deploy new tools frameworks and patterns to build sustainable Big Data platform
  • Identify gaps and opportunities for improvement of existing solutions.
  • Define and develop APIs for integration of various data sources.
  • Analyze and define customer requirements.
  • Make accurate development effort estimates to assist management in project planning.
  • Responsible for migrating and transforming data from on-prem to on-cloud.
  • Building CICD pipelines for automation.
  • Salary - >$100,000

#LI-CTSAPAC

Employee Status : Full Time Employee

Shift : Day Job

Travel : No

Job Posting : Jul 20 2021

About Cognizant
Cognizant (Nasdaq-100: CTSH) is one of the world's leading professional services companies, transforming clients' business, operating and technology models for the digital era. Our unique industry-based, consultative approach helps clients envision, build and run more innovative and efficient businesses. Headquartered in the U.S., Cognizant is ranked 194 on the Fortune 500 and is consistently listed among the most admired companies in the world. Learn how Cognizant helps clients lead with digital at www.cognizant.com or follow us @Cognizant.

*** Mention DataYoshi when applying ***

Offers you may like...

  • Babylist

    Data Engineer
    Oakland, CA
  • Darwin Homes

    Data Engineer
    Remote
  • eCivis

    Senior Data Engineer (BI) -Remote Must live in USA...
    Pasadena, CA 91103
  • eCivis

    Senior Data Engineer / ETL Developer- Remote USA
    Pasadena, CA 91103
  • Deloitte

    Virtual Cloud Data Engineer - Solution Specialist ...
    Gilbert, AZ 85297