CyberQ Consulting

Data Engineer [Hadoop and Kafka]

Job description

People join us because they want to be a part of an organization where cutting edge technology is played around with. They join us because we hire with attitude and potential. We believe in the innate capability of inspired individuals in managing their own work and ambitions, enabling the leadership to focus on the Companys vision.

What You Will Be Doing

  • Designing and implementing data processing pipelines, as part of diverse, high energy teams
  • Working with our data scientists to take our ML\AI models to production
  • Hands-on programming in Python, Java, Scala
  • Deploying data pipelines in production based on Continuous Delivery practices
  • Recommending the right distributed storage and computing technologies to our clients from a large number of options available in the ecosystem
  • Ensuring that the data is available to the consumers in a reliable, trustworthy and predictable manner
  • Evolving the data platform to make it more robust, scalable

What We Are Looking For

  • 6-8 years of experience working as a data engineer
  • Proficient understanding of distributed computing principles
  • Proficiency HDFS, Spark, Kafka
  • Experience with integration of data from multiple data sources
  • Experience with NoSQL databases Redis, MongoDB, ElasticSearch,
  • Knowledge of various ETL techniques and frameworks
  • Knowledge of how to create and maintain optimal data pipeline architecture,
  • Good understanding of Lambda Architecture, along with its advantages and drawbacks

Great To Have

  • Ability to solve any ongoing issues with operating the cluster
  • Experience with building stream-processing systems, using solutions such as Flink or Spark-Streaming
  • Good knowledge of Big Data querying tools, such as Pig, Hive, and Impala
  • Experience with Cloudera/MapR/Hortonworks

Please let the company know that you found this position on this Job Board as a way to support us, so we can keep posting cool jobs.