Lead Data Engineer

Job Description

Must have skill set: Spark, S3, Glue, AWS Redshift, Python, and StreamSet experience.

6-8 years of IT experience focusing on enterprise data architecture and management.

Experience and Expertise

• Conceptual/Logical/Physical Data Modeling
• Relational and Dimensional Data Modeling
• Experience with Databricks & on Prem, Structured Streaming, Delta Lake concepts, and Delta Live Tables required
• Experience with Spark Scala Data Lake concepts such as time travel and schema evolution and optimization

Additional Experience

• Structured Streaming and Delta Live Tables with Databricks a bonus
• Experience leading and architecting enterprise-wide initiatives
• Advanced level understanding of streaming data pipelines and batch systems
• Formalize concepts of handling late data, defining windows, and data freshness

Skills Overview

• Understanding of ETL and ELT
• Ability to handle different incremental data loads
• Experience with Data Migration Service and Great Expectations
• Advanced level SQL experience
• Indexing and partitioning strategy experience

Architecture and Environment

• Experience with large-scale, high-performance enterprise big data application deployment
• Architecture experience in AWS environment a bonus
• Experience working with Lambda and CloudWatch
• Experience with Gitlabs and CloudWatch for supporting CI/CD pipelines

Additional Qualifications

• Familiarity with Schema Registry and message formats
• Ability to thrive in a team-based environment
• Experience briefing technology solutions to various stakeholders