Job Description
We're looking for a Senior AWS Data Engineer skilled in Python open to working hybrid in Houston! This role is contract-to-hire.
Purpose
Establish data feeds to the digital platform and manage ETL processes. Design raw file data structures to be used in data science applications. Associated commercial data includes data required for forecasting, analysis, optimization modeling and data visualization. Work with Data Analysts to ensure enterprise data is cataloged in accordance with data governance standards. Develop system test and integration plans, execute test procedures, support user community, and obtain approvals for change management. Provide hands-on technical configuration, application development, integration and testing. Implement cloud data integration strategies between cloud providers. Provide technical expertise to the business and internal IT with respect to digital solutions, including data visualization and facilitate business expansion with solutions to scale, managing the critical input and output data.
Essential Job Functions
- Developing, testing, and deploying data pipelines, data lakes, data warehouses, and data marts using AWS services such as S3, Glue, Athena, Redshift, EMR, Kinesis, Lambda, and more.
- Implementing data ingestion, processing, transformation, and delivery using various methods such as batch, streaming, and real-time.
- Applying best practices for data quality, security, reliability, and performance using AWS services such as Lake Formation, IAM, KMS, CloudTrail, CloudFormation, and more.
- Collaborating with data architects, data analysts, data scientists, citizen data scientists, and developers to understand data requirements and provide data solutions that meet business needs and follow the AWS Well-Architected Framework.
Responsibilities
- Architect digital data solutions for GEM US for all Commercial Operations (Trading & Assets).
- Implement ETL for required enterprise data.
- Work with other IT operations support resources in a collaborative manner to achieve cross-functional goals.
- Identify deficiencies and recommend solutions to management and the user community.
- Provide automation where applicable to minimize business impact and ongoing IT support.
- Continual performance assessment of processes and applications.
- Oversee and mentor key support resources, training them to provide primary and backup system operations and maintenance functions.
Requirements
- Knowledge of energy market data.
- In-depth knowledge of digital platforms such as AWS, Dataiku, and Databricks.
- In-depth proven software engineering experience using Python and machine learning knowledge.
- Designing and building scalable data architectures on AWS, with strong experience using AWS data storage and processing services such as S3, EMR, Data Lake Formation, AWS Glue, and Amazon Athena.
- Experience in working with DevOps tools and methodologies such as Git, Jenkins, Docker, or Kubernetes.
- Proficiency in programming languages such as Python, Java, Scala, SQL, Spark, and Unix shell scripting, including AWS CLI.
Education/Experience
- Bachelor’s degree in a technical field or 15+ software development experience in lieu of degree.
- 10-15 years of information technology, systems, and application development experience.
- Minimum 5 years’ experience supporting the needs of an Energy Trading or Asset Management business preferred.
- Proficient with Microsoft Office products (Excel, Word, Visio, PowerPoint).