Role: Senior Developer – Modern Data Platform Architect
Role Summary
We are seeking a highly skilled Senior Developer / Data Platform Engineer to design and build scalable data pipelines and modern Lakehouse solutions on
Databricks-based platforms
. The candidate should have strong expertise in
PySpark, ETL/ELT frameworks, Data Quality, Apache Iceberg
, and
job orchestration
, with exposure to
asset management domain
considered a strong advantage.
Key Responsibilities
1. Data Platform Development & Engineering
Design and develop scalable
data ingestion and transformation pipelines
using PySpark and SQL.
Build and manage
batch and streaming ETL/ELT pipelines
on Databricks.
Implement data processing frameworks supporting
high-volume enterprise datasets
.
2. Lakehouse Architecture & Iceberg Implementation
Design and maintain
modern data Lakehouse architecture (Bronze/Silver/Gold layers)
.
Work on
Apache Iceberg / open table formats
including schema evolution, partitioning, and compaction strategies.
Ensure interoperability across engines and optimize data storage formats for performance.
3. Data Ingestion & Pipeline Management
Design and monitor
end-to-end ingestion pipelines
(batch + near real-time).
Build reusable ingestion frameworks for structured and semi-structured data.
Manage dependencies, data lineage, and pipeline orchestration workflows.
4. Data Transformation & dbt (Nice to Have)
Implement transformation logic using
dbt or similar ELT frameworks
.
Standardize transformation pipelines for maintainability and versioning.
5. Data Quality & Observability
Implement robust
data quality checks, validation rules, and reconciliation processes
.
Develop monitoring frameworks for pipeline failures, SLA tracking, and alerting.
Ensure data governance and audit compliance across datasets.
6. Job Orchestration & Workflow Management
Implement and manage orchestration tools (Airflow / Databricks Jobs / other schedulers).
Handle pipeline scheduling, retries, dependency chaining, and failure recovery.
7. CI/CD & DevOps Enablement
Build and maintain
CI/CD pipelines for data workflows and infrastructure
.
Automate deployment of Databricks jobs, notebooks, and configurations.
Manage code versioning, release management, and environment promotion.
8. Performance Optimization
Optimize Spark jobs for
cost, performance, and scalability
.
Tune queries, partitioning strategies, and resource utilization.
9. Collaboration & Stakeholder Engagement
Work with business teams, data architects, and upstream/downstream systems.
Translate business requirements (especially in
asset management domain
) into technical solutions.
Support data modeling (Data Vault / Star Schema exposure is a plus).
10. Platform Governance & Best Practices
Follow enterprise standards for
data security, access control, and governance
.
Implement Unity Catalog / RBAC / metadata management best practices.
Ensure compliance with platform architecture and operational standards.
Required Skills
Must Have
Strong experience in
Databricks
and PySpark
ETL/ELT pipeline design and development
Data ingestion and pipeline orchestration
SQL and distributed data processing
Data quality and validation frameworks
Experience with
Apache Iceberg / open table formats
Cloud experience (AWS preferred)
Good to Have
Exposure to
Asset Management / Financial Data domain
dbt (Data Build Tool)
Data modeling (Data Vault / Star Schema)
Knowledge of Snowflake / Unity Catalog
Experience with CI/CD (Terraform, Git-based deployments)
About State Street
Across the globe, institutional investors rely on us to help them manage risk, respond to challenges, and drive performance and profitability. We keep our clients at the heart of everything we do, and smart, engaged employees are essential to our continued success.
We are committed to fostering an environment where every employee feels valued and empowered to reach their full potential. As an essential partner in our shared success, you’ll benefit from inclusive development opportunities, flexible work-life support, paid volunteer days, and vibrant employee networks that keep you connected to what matters most. Join us in shaping the future.
As an Equal Opportunity Employer, we consider all qualified applicants for all positions without regard to race, creed, color, religion, national origin, ancestry, ethnicity, age, disability, genetic information, sex, sexual orientation, gender identity or expression, citizenship, marital status, domestic partnership or civil union status, familial status, military and veteran status, and other characteristics protected by applicable law.
Discover more information on jobs at
StateStreet.com/careers
Read our
CEO Statement
