Data Engineer

Company:
Location: Tiong Bahru

*** Mention DataYoshi when applying ***

Roles & Responsibilities

  • Design, develop and maintain Ninja Van’s infrastructure for streaming, processing and storage of data. Build tools for effective maintenance and monitoring of the data infrastructure.
  • Contribute to key data pipeline architecture decisions and lead the implementation of major initiatives.
  • Work closely with stakeholders to develop scalable and performant solutions for their data requirements, including extraction, transformation and loading of data from a range of data sources.
  • Develop the team’s data capabilities - share knowledge, enforce best practices and encourage data-driven decisions.
  • Develop Ninja Van’s data retention policies, backup strategies and ensure that the firm’s data is stored redundantly and securely.

Requirements

  • Solid Computer Science fundamentals, excellent problem-solving skills and a strong understanding of distributed computing principles.
  • At least 3 years of experience in a similar role, with a proven track record of building scalable and performant data infrastructure.
  • Expert SQL knowledge and deep experience working with relational and NoSQL databases.
  • Advanced knowledge of Apache Kafka and demonstrated proficiency in Hadoop v2, HDFS, and MapReduce.
  • Experience with stream-processing systems (e.g. Storm, Spark Streaming), big data querying tools (e.g. Pig, Hive, Spark) and data serialization frameworks (e.g. Protobuf, Thrift, Avro).
  • Bachelor’s or Master’s degree in Computer Science or related field from a top university.

Tech Stack

  • Backend: Play (Java 8+), Golang, Node.js
  • Frontend: AngularJS, ReactJS
  • Mobile: Android, Flutter, React Native
  • Cache: Hazelcast, Redis
  • Data storage: MySQL, TiDB, Elasticsearch, Delta Lake
  • Infrastructure monitoring: Prometheus, Grafana
  • Orchestrator: Kubernetes
  • Containerization: Docker, Containerd
  • Cloud Provider: GCP, AWS
  • Data pipelines: Apache Kafka, Spark Streaming, Maxwell
  • Workflow manager: Apache Airflow
  • Query engines: Apache Spark, Trino

*** Mention DataYoshi when applying ***

Offers you may like...

  • Woolworths Group

    Senior Data Engineer
    Darlinghurst NSW 2010
  • BHP

    Specialist Data Engineer
    Australia
  • Cegeka

    Data Engineer
    Apeldoorn
  • Maderik Institute Of Management AB

    Cloud Data Engineer /Stockholm (remote)
    Stockholm
  • Cognizant Technology Solutions

    Big Data Engineer
    Barcelona, Barcelona provincia