Data Engineer (Spark)
Data Engineer
We are seeking a skilled Data Engineer with expertise in Spark to drive optimization initiatives. This role requires a comprehensive understanding of complex Spark pipelines, including code optimization, configuration tuning, and infrastructure enhancement. The ideal candidate is proficient in Spark operations and business logic, capable of rewriting code in Java when necessary, and skilled in reconfiguring infrastructure, such as transitioning compute environments (e.g., from XCd6 to ARM in AWS). Key tasks include optimizing Spark job performance, enhancing underlying infrastructure, and ensuring efficient data validation.
Key Qualifications:
– Extensive experience with Spark and data pipeline optimization
– Proven skills in code optimization, Spark job configuration, and infrastructure management
– Strong understanding of data engineering principles and Spark’s operational logic
– Proficiency in Java and AWS infrastructure transitions