Role Summary
State Street Investment Management’s Data, Analytics & AI Services (DAAIS) team is seeking a Senior Data Engineering Lead (Assistant Vice President) for our Bangalore, India office. This role is accountable for the design, build, and operate of modern data platform and software capabilities that enable the investment management business. The successful candidate will bring deep hands-on expertise in large-scale data platform architecture, distributed computing, data pipeline engineering, and modern cloud architecture. This role will execute on the broader platform roadmap in partnership with senior leadership and global stakeholders and may lead a small team/pod to deliver reliable, scalable platform outcomes.
Key Responsibilities
Build and optimize batch and streaming data pipelines using PySpark/Spark SQL and Databricks workflows.
Design and maintain Bronze/Silver/Gold (multi-hop) data architectures to support analytics and ML.
Develop robust ingestion patterns from sources such as RDBMS, APIs, event streams, object storage (ADLS/S3/GCS).
Implement incremental loads, CDC patterns, partitioning, and data compaction strategies.
Required Qualifications
14+ years in software/data engineering; BS/MS in CS/Engineering (or equivalent experience).
Strong experience with Databricks (jobs/workflows, clusters, notebooks, repos, DBFS/Volumes).
Advanced proficiency in Apache Spark concepts: transformations/actions, shuffles, partitioning, caching, joins. Proficiency in Python (PySpark) and strong SQL skills required.
Expert-level Spark SQL and query optimization.
Demonstrated delivery of production-grade data pipelines in a cloud environment.
Ability to write clean, testable, production-grade code (modules, packaging, CICD).
Experience with Apache Iceberg and table/catalog governance concepts (e.g., Unity Catalog or equivalent).
Experience with orchestration tools: Databricks Workflows, Airflow, ADF, or equivalent.
Familiarity with ingestion patterns: batch loads, CDC, streaming pipelines.
Experience with at least one major cloud ecosystem (Azure/AWS).
Working knowledge of cloud object storage and security patterns like AWS S3 + IAM policies
Familiarity with SLAs, SLOs, incident response, and root cause analysis for data failures.
Strong problem solving, debugging, and performance tuning mindset.
Executes independently on defined scope; contributes to best practices and code reviews.
Preferred Qualifications
Financial services experience (asset management preferred).
Unity Catalog administration and advanced governance (lineage, catalogs, grants, policies).
DBT on Databricks (models, tests, documentation) and semantic layer concepts.
Streaming technologies: Kafka, Event Hubs, Kinesis, Structured Streaming best practices.
Experience with ML feature engineering pipelines and collaboration with data science teams.
About State Street
Across the globe, institutional investors rely on us to help them manage risk, respond to challenges, and drive performance and profitability. We keep our clients at the heart of everything we do, and smart, engaged employees are essential to our continued success.
We are committed to fostering an environment where every employee feels valued and empowered to reach their full potential. As an essential partner in our shared success, you’ll benefit from inclusive development opportunities, flexible work-life support, paid volunteer days, and vibrant employee networks that keep you connected to what matters most. Join us in shaping the future.
As an Equal Opportunity Employer, we consider all qualified applicants for all positions without regard to race, creed, color, religion, national origin, ancestry, ethnicity, age, disability, genetic information, sex, sexual orientation, gender identity or expression, citizenship, marital status, domestic partnership or civil union status, familial status, military and veteran status, and other characteristics protected by applicable law.
Discover more information on jobs at
StateStreet.com/careers
Read our
CEO Statement
