Design, develop, and implement Spark Scala applications and data processing pipelines to process large volumes of structured and unstructured data
Integrate Elasticsearch with Spark to enable efficient indexing, querying, and retrieval of data
Optimize and tune Spark jobs for performance and scalability, ensuring efficient data processing and indexing in Elasticsearch
Collaborate with data engineers, data scientists, and other stakeholders to understand requirements and translate them into technical specifications and solutions
Implement data transformations, aggregations, and computations using Spark RDDs, DataFrames, and Datasets, and integrate them with Elasticsearch
Develop and maintain scalable and fault-tolerant Spark applications, adhering to industry best practices and coding standards
Troubleshoot and resolve issues related to data processing, performance, and data quality in the Spark-Elasticsearch integration
Requirements
5+ Years as Data Engineer
5 Years experience in Spark, Scala, Elastic Search
Develop and maintain scalable and fault-tolerant Spark applications, adhering to industry best practices and coding standards
Monitor and analyze job performance metrics, identify bottlenecks, and propose optimizations in both Spark and Elasticsearch components
Stay updated with emerging trends and advancements in the big data technologies space to ensure continuous improvement and innovation