Big Data Developer with Python

Job Description

• Role: Big Data with Python Developer
• Location: Raritan, NJ
• Rate: Market

Responsibilities:

• Development, customize and manage integration tools, databases, warehouses, and analytical systems with the use of data-related instruments/instances.
• Create and run complex queries and automation scripts for operational data processing and building out Python ETL processes and writing complex SQL queries.
• Test the reliability and performance of each part of a system and cooperate with the testing team.
• Deploying data models into production environments.
• Monitoring the overall performance and stability of the system.
• Adjust and adapt the automated pipeline as data/models/requirements change.

Requirements and Skills:

• Excellent understanding of ETL cycle.
• Analyze and organize raw data, build data systems and pipelines.
• Combine raw information from different sources, explore ways to enhance data quality and reliability, interpret trends and patterns from the raw data.
• Experience in using Python/ PySpark and/or Scala for data engineering.
• Understanding of data types/handling of different data models.
• Good knowledge in various phases of SDLC Requirement Analysis, Design, Development, and Testing on various Development and Enhancement Projects.
• Desirable to have experience with Spark, Flink, Kafka, Flask, Scala, PySpark for Data engineering.
• Experience with Microsoft Azure or AWS data management tools such as Azure Datafactory, Datalake, and Databricks or AWS Snowflake is a plus.
• Experience with data visualization tools is a plus (PowerBI, Tableau).
• Understanding of descriptive and exploratory statistics, predictive modeling, evaluation metrics, decision trees, machine learning algorithms is a plus.
• Good scripting and programming skills.