Role Overview
:
We are seeking a highly skilled
Senior Data Engineer
with 5+ years of experience to join our team. This role is focused on designing and building robust data assets through high-performance data pipelines. You will be a key player in modernizing our data infrastructure, transitioning legacy codebases into clean, scalable architectures, and ensuring the highest standards of code quality through Test-Driven Development (TDD).
Key Responsibilities:
Data Pipeline Development:
Design, develop, and maintain complex ETL/ELT pipelines to build high-value data assets.
Legacy Modernization:
Lead the
code refactorization
of legacy codebases, improving readability, maintainability, and performance.
System Optimization:
Perform deep code optimization using
Spark SQL
and
PySpark
to handle large-scale datasets efficiently.
Quality Assurance:
Implement a
Test-Driven Development (TDD)
approach, writing comprehensive unit tests to ensure functionality and catch bugs early in the lifecycle.
Complex Problem Solving:
Isolate and resolve difficult bugs, including those related to performance bottlenecks, concurrency issues, and complex logic flaws.
Cloud Architecture:
Design and deploy solutions utilizing the full AWS stack, explaining the trade-offs and benefits of specific services for various use cases.
Technical Requirements:
Core Programming & Data Engineering
5+ years of experience
in hands-on programming with
Python
and
PySpark
.
Expertise in
Boto3
and various Python frameworks and libraries, adhering strictly to Python best practices (PEP 8).
Strong experience in
Spark SQL
and PySpark optimization techniques (e.g., partitioning, caching, broadcast joins).
Cloud & Infrastructure (AWS)
Deep architectural knowledge of
AWS services
, including:
S3, EC2, Lambda, Redshift, CloudFormation
DevOps & Tools
Advanced understanding of
Git
(branching strategies, PR reviews).
Experience with
JFrog Artifactory
for dependency management and artifact storage.
Proficiency in CI/CD pipelines and automated testing frameworks.
Professional Attributes:
Analytical Mindset:
Ability to debug complex, non-obvious issues in distributed systems.
Clean Coder:
Passion for writing "clean code" and mentoring junior engineers on maintainability.
Architectural Thinking:
Ability to explain the "why" behind choosing specific AWS components over others.
