Data Engineering Lead - Azure - Multiple Locations

Location: India

*** Mention DataYoshi when applying ***

Combine two of the fastest-growing fields on the planet with a culture of performance, collaboration and opportunity and this is what you get. Leading edge technology in an industry that's improving the lives of millions. Here, innovation isn't about another gadget, it's about making health care data available wherever and whenever people need it, safely and reliably. There's no room for error. Join us and start doing your life's best work.(sm)
The project under Data Solutions in Optum is a cloud migration initiative to build a unified data platform to cater unique data needs for our consumers and end users. As part of it we are migrating end to end data delivery stack to azure cloud. Our team is looking experts who has experience in cloud migration of large datasets on warehouse and build complex data pipelines for data load and perform analytics. We are essentially looking for data engineers who has sound experience in working on azure eco system and hands on experience leading and delivering azure cloud solutions.
Primary Responsibilities:
  • Building scalable cloud data solution using MPP and distributed warehouse solution on azure (Synapse or snowflakes) and analytics platforms (Apache spark, Data bricks)
  • Creation and automation of data pipelines i.e., onboarding data using azcopy to Gen2, load data to warehouse (Synapse or snowflakes)
  • Run analytics using (Apache Spark/Scala on Azure Data bricks) and migration large data sets from on perm to cloud
  • Manage orchestration and scheduling of end to end data pipeline using tool like Apache airflow, ADF scheduling, logic apps
  • Writing and optimizing complex SQL queries, store procedure in Synapse or snowflake
  • Focus on driving Infrastructure as a code, scheduling as a code and automating operational activities using Jenkins, azure DevOps and terraform scripts
  • Use Azure Data lake analytics with the help of data bricks to implement data transformation, preparing data
  • Designing solutions that make use of Azure's Run time execution using containers, function app and leverage stateless and serverless designs
  • Effective collaboration with Architecture team in designing solutions and with product owners with validating the implementations
  • Efficient in python or Scala to be develop portable solutions with various resources native to Microsoft and third party
  • Implementing best practices to enable data quality, monitoring, logging and alerting the failure scenarios and exception handling
  • Documenting step by step process to trouble shoot the potential issues and deliver cost optimized cloud solutions
  • Mentoring fellow engineers on implementation aspects and grooming up team member for ideas
  • Comply with the terms and conditions of the employment contract, company policies and procedures, and any and all directives (such as, but not limited to, transfer and/or re-assignment to different work locations, change in teams and/or work shifts, policies in regards to flexibility of work benefits and/or work environment, alternative work arrangements, and other decisions that may arise due to the changing business environment). The Company may adopt, vary or rescind these policies and directives in its absolute discretion and without any limitation (implied or otherwise) on its ability to do so

Required Qualifications:
  • 8+ years of experience in data engineering and working on large data warehouse MPP database systems like Synapse analytics including design and development of ETL on distributed platforms
  • 4+ years of experience in Azure cloud eco system particularly on azure resources for data & ETL migration from on perm to cloud
  • 4+ years of experience in Azure database offering like SQL DB, Postgres DB, constructing data pipelines using Azure data factory, design and development of analytics using Azure data bricks
  • 3+ years of experience on working with cloud native monitoring and logging tool like Log analytics or having experience on any third-party services is a plus
  • 3+ years of experience in scheduling tools on cloud either using Apache Airflow or logic apps or any native/third party scheduling tool on cloud
  • 3+ years of working experience on Azure run time integration services like Azure container, azure function app
  • Knowledge on running SQL file from azure blob storage into database (Synapse or SQL server) by leveraging azure function app
  • Knowledge on multiple solution for integrating ADF data pipeline, SQL scripts, Data bricks job into Airflow DAG or Azure Logic app
  • Automation experience using Git hub and Jenkins and automating deployment process across environment
  • Knowledge on consumption layer solution for end users say Azure WVD, Synapse/data bricks workspace, any other solutions
  • Knowledge on Azure active directory and Azure domain services

Preferred Qualifications:
  • At least 1 programming language, having python is a plus
  • Knowledgeable in cloud shell either Powershell or bash
  • Knowledgeable in Apache Open source software stack for quick integration
  • Azure certifications Ex. AZ 200, AZ 201, AZ – 303, AZ -304, AZ – 400
  • Excellent written and verbal communication skills for communicating with customers, technically and procedurally
  • Highly self-motivated with excellent interpersonal and collaborative skills
  • Strong analytical and strong problem-solving skills
  • Anticipate risks and obstacles and develop plans for mitigation
  • Balance multiple and competing priorities and execute accordingly
  • Excellent documentation experience and skills

8.00-11.00 Years

*** Mention DataYoshi when applying ***

Offers you may like...

  • SmartBLKTrade Limited (SBT)

    Research Data Scientist / Big Data Engineer, AI De...
    Hong Kong
  • scieneers GmbH

    Data Scientist / Data Engineer (m/w/d)
    22761 Hamburg
  • The Upside Travel Company, LLC

    Senior Data Engineer
  • PPL Corporation

    Senior Data Engineer- Remote
    Allentown, PA
  • Artemis Consulting Inc

    Senior AWS Cloud Data Engineer
    Washington, DC 20001