Python Developer with Spark Expertise

Job Details

Position: Python Developer with Spark Expertise

Position Overview:

We are seeking a talented Python Developer with expertise in Apache Spark to join our growing team. The ideal candidate will have a strong background in Python development and hands-on experience with Spark for processing large-scale datasets. As a key member of our team, you will play a critical role in developing data pipelines, optimizing performance, and implementing robust solutions to meet our clients' evolving needs.

Responsibilities:

• Design, develop, and maintain data processing pipelines using Python and Apache Spark.
• Collaborate with cross-functional teams to understand requirements and translate them into technical solutions.
• Optimize Spark jobs for performance, scalability, and reliability.
• Implement data transformations, aggregations, and analytics to derive meaningful insights from large datasets.
• Troubleshoot and resolve issues related to data processing and performance bottlenecks.
• Stay updated on emerging technologies and best practices in data engineering and Spark ecosystem.

Qualifications:

• Bachelor's degree in Computer Science, Engineering, or related field.
• Strong proficiency in Python programming language.
• Hands-on experience with Apache Spark for distributed data processing.
• Familiarity with Spark RDDs, DataFrames, and Spark SQL.
• Experience with Spark streaming and batch processing.
• Solid understanding of data structures, algorithms, and software design principles.
• Excellent problem-solving skills and attention to detail.
• Effective communication and collaboration skills.
• Experience with cloud platforms (e.g., AWS, Azure, Google Cloud Platform) is a plus.
• Prior experience with other big data technologies (e.g., Hadoop, Kafka) is desirable.