Senior Data Engineer – Financial Services

Job Title: Senior Data Engineer – Financial Services (Data Lakehouse & Real-Time Systems) About the Role We are seeking seasoned Data Engineering and architecture leaders to architect and deliver a next-generation data lake/house platform for a leading financial services client. This engagement will involve thorough data architecture and engineering components, analysis of complex data and its ecosystem and integrating real-time trade, crypto, securities and fixed income data, analytics, and risk systems for various use-cases. The ideal candidate combines deep domain experience in retail brokerage with hands-on expertise in AWS, MS- Azure Databricks, Spark, and modern data lake/house architectures, and can balance delivery under pressure with architectural foresight and team mentorship. This role requires a deep skill-set in real-time architectural systems and pipelines, adept with financial services specifically, investment domain and a hands-on engineer with a knack of innovative insight on integration of various data architecture/s within the cloud ecosystem. Key Responsibilities • Architect and lead the design, implementation, and optimization of large-scale data lake/house systems on Azure Databricks using Spark, Delta Lake, and Python. (advanced) • Build and optimize streaming pipelines for real-time ingestion, processing, and analytics of trade and market data across asset classes (FX, equities, futures, options, bonds, and crypto). (intermediate) • Collaborate closely with the CTO and data architects to evolve the end-to-end application architecture for scalability, resilience, and low-latency performance. (advanced) • Mentor and guide junior and mid-level engineers, establishing best practices in data modelling, orchestration, and operational monitoring. • Ensure data integrity, lineage, and governance within regulatory and compliance frameworks. (advanced) • Partner cross-functionally with business and technology teams to align data infrastructure with business KPIs and capital markets product goals. • Drive automation and CI/CD for data pipelines using Terraform, GitHub Actions, and cloud-native DevOps practices. (advanced) • Deliver under tight timelines, managing sprint-level execution and ensuring production readiness within a short time-line for GO-LIVE. Required Skills and Qualifications: • 15+ years of professional experience, with at least 10 years in financial services and 8+ years in data engineering leadership roles. • Proven track record building real-time, high-throughput data platforms at enterprise scale. • Strong proficiency in Azure Databricks, Apache Spark, and Python (PySpark, Panda). • Solid experience with streaming technologies (Kafka, Kinesis, Event Hubs, or Flink). • Deep understanding of data lake/house design principles (Cloud Engineering and architecture, AWS, Delta Lake, Iceberg, or Hudi) and data modelling for analytics and ML pipelines. • Strong background in retail brokerage and trading systems, including exposure to FX, equities, futures, options, bonds, and crypto. • Experience building or integrating risk management systems, trade surveillance, or crypto exchanges. • Practical knowledge of data governance, lineage, and metadata management tools (e.g., Azure Purview). • Proven ability to design scalable, cost-efficient pipelines on Azure using Data Factory, Synapse, and related services. • Excellent communication, collaboration, and documentation skills, with the ability to influence senior stakeholders. Preferred Qualifications • Experience working with microservices-based or event-driven architectures supporting capital markets workloads. • Familiarity with machine learning pipelines for predictive risk and trading analytics. • Exposure to Shopify-like multi-tenant or marketplace data architectures. • Cloud certifications: Microsoft Certified: Azure Data Engineer Associate or equivalent. • Previous experience in start-up or fast-paced delivery environments with adaptability and velocity. Core Attributes • Strong sense of ownership, reliability, and confidentiality in managing critical financial data. • Ability to balance delivery urgency with architectural integrity. • A collaborative mentor who uplifts engineering standards across the team. • Strategic mindset with hands-on execution capability. Engagement Details • Number of Positions: 3 • Location: Mumbai • Engagement Type: Full-Time / Contract-to-Hire • Timeline: Immediate start; production launch targeted for January • Reports To: Engagement Lead in conjunction with Client CTO.