Unison Consulting

Data Engineer (Spark ,Scala, Elastic Search)

Job description

  • Design, develop, and implement Spark Scala applications and data processing pipelines to process large volumes of structured and unstructured data
  • Integrate Elasticsearch with Spark to enable efficient indexing, querying, and retrieval of data
  • Optimize and tune Spark jobs for performance and scalability, ensuring efficient data processing and indexing in Elasticsearch
  • Collaborate with data engineers, data scientists, and other stakeholders to understand requirements and translate them into technical specifications and solutions
  • Implement data transformations, aggregations, and computations using Spark RDDs, DataFrames, and Datasets, and integrate them with Elasticsearch
  • Develop and maintain scalable and fault-tolerant Spark applications, adhering to industry best practices and coding standards
  • Troubleshoot and resolve issues related to data processing, performance, and data quality in the Spark-Elasticsearch integration

Requirements

  • 5+ Years as Data Engineer
  • 5 Years experience in Spark, Scala, Elastic Search
  • Develop and maintain scalable and fault-tolerant Spark applications, adhering to industry best practices and coding standards
  • Monitor and analyze job performance metrics, identify bottlenecks, and propose optimizations in both Spark and Elasticsearch components
  • Stay updated with emerging trends and advancements in the big data technologies space to ensure continuous improvement and innovation

Please let the company know that you found this position on this Job Board as a way to support us, so we can keep posting cool jobs.

Similar jobs

Browse All Jobs

data engineer

Neotalent Conclusion
April 17, 2024
Noesis
April 17, 2024

Data Engineer