Middle Data Engineer

Project Overview

Our client is a leading retirement product planning company in the US. They have performed a lift and shift migration of SQL Server 2008 into Azure SQL Database (MI) and have designed a new architecture based on cloud services. Our project will help them validate and implement the new architecture based on cloud services.

Required Skills:

• 3+ years of hands-on experience in data engineering with Python and SQL.
• 2+ years of experience with Azure, DWH services (Databricks), cloud storage (Azure Storage).
• Strong experience with ETL/ELT/orchestration tools (e.g., Airflow, ADF, Glue, Nifi).
• Contribute to the design and implementation of data warehouse and data lakes.
• Good knowledge of code versioning (git) and building CI/CD for data projects.
• Exposure requirement gathering and documentation.

Will Be a Plus:

• Experience with NoSQL.
• Experience with stream processing (e.g., Apache Kafka, Apache Flink, Spark Structured Streaming).

Responsibilities:

• Collaborate with business stakeholders and technical teams to understand and analyze data requirements.
• Support the design and implementation of data models and database structures that meet business needs.
• Profile, refactor, and optimize performance in database systems.
• Implement ETL processes to extract, transform, and load data from various source systems into the data warehouse.
• Ensure data integrity, consistency, and accuracy through data quality assurance measures.
• Review and support team members, providing guidance and mentorship.