Job Description:
Title:
Lead data engineer
DCF Level:
L40
About the Role
We are seeking a highly skilled and delivery-focused Lead GCP Data Engineer to support the design, development, and implementation of next-generation enterprise data and AI platforms on Google Cloud Platform (GCP).
This role will work closely with Enterprise Architects, platform leaders, and cross-functional engineering teams to build scalable, reusable, and AI-ready data foundations that enable advanced analytics, intelligent automation, and enterprise AI adoption.
The ideal candidate combines strong hands-on expertise in cloud-native data engineering, modern data platform development, semantic data enablement, and scalable pipeline engineering with the ability to lead engineering teams and drive high-quality delivery across multiple initiatives.
This role is expected to play a critical leadership position within the engineering organization by driving implementation excellence, mentoring teams, and operationalizing modern data architecture patterns.
Key Responsibilities
1. Enterprise Data Platform Engineering
Design, develop, and optimize scalable cloud-native data platforms and pipelines on GCP.
Implement robust batch, streaming, and event-driven data processing solutions supporting enterprise analytics and AI use cases.
Collaborate with Enterprise Architects to translate target-state architecture into scalable engineering implementations.
Contribute to modernization of legacy data ecosystems into reusable, governed, and AI-ready cloud platforms.
Support implementation of scalable ingestion, transformation, serving, and orchestration frameworks.
2. Data Product Engineering
Develop reusable and domain-oriented data products aligned with data mesh and data-as-a-product principles.
Implement scalable and modular data pipelines supporting multiple downstream consumers including analytics, AI/ML, and operational applications.
Contribute to implementation of:
Data contracts
Schema management
Metadata enrichment
Data quality frameworks
Reusable transformation patterns
Enable discoverability, trust, and operational reliability of enterprise data assets.
3. Semantic Layer & Consumption Enablement
Support implementation of semantic and business-consumption layers that simplify enterprise data access.
Collaborate with analytics and BI teams to enable standardized business metrics, reusable dimensions, and governed KPI definitions.
Contribute to semantic modeling and metadata integration initiatives supporting self-service analytics and AI consumption.
Assist in improving enterprise data usability, consistency, and discoverability across platforms.
4. GCP-Native Engineering & Development
Develop and optimize solutions leveraging GCP-native services including:
BigQuery
Dataflow
Dataproc
DBT
Pub/Sub
Cloud Storage
Cloud Composer (Airflow)
Cloud SQL
Build scalable ETL/ELT frameworks and real-time streaming pipelines.
Optimize data processing performance, reliability, scalability, and cost efficiency.
Implement CI/CD pipelines and engineering automation for data platform delivery.
5. AI/ML & GenAI Data Enablement
Build AI-ready data pipelines and scalable feature engineering workflows supporting enterprise AI initiatives.
Support integration with:
Vertex AI
BigQuery ML
Vector databases
LangChain
Generative AI Studio
Contribute to implementation of RAG architectures, semantic search, and AI-assisted data interaction patterns.
Partner with AI/ML teams to operationalize scalable ML and GenAI workflows.
6. Engineering Leadership & Delivery Excellence
Lead day-to-day engineering activities across multiple data engineering workstreams.
Guide and mentor junior and mid-level data engineers on modern engineering best practices.
Ensure adherence to coding standards, architecture guidelines, and operational best practices.
Drive engineering quality through automated testing, observability, monitoring, and performance optimization.
Collaborate with architects, product owners, analysts, and client stakeholders to ensure successful delivery outcomes.
7. Governance, Reliability & Observability
Implement data governance, lineage, monitoring, and observability frameworks.
Support enforcement of enterprise standards around security, reliability, scalability, and operational readiness.
Contribute to platform monitoring, incident management, and continuous improvement initiatives.
Ensure production readiness of pipelines and data services through robust testing and validation processes.
Technical Expertise Required
Area
Skills / Technologies
Cloud Data Engineering
GCP, BigQuery, Dataflow, Dataproc, Pub/Sub, Cloud Storage, Cloud SQL
Data Transformation
DBT, PySpark, SQL, ETL/ELT frameworks
Streaming & Pipelines
Apache Beam, real-time processing, event-driven architectures
Semantic Layer & Modeling
Semantic modeling concepts, Looker modeling, business metrics standardization
AI/ML Enablement
Vertex AI, BigQuery ML, LangChain, Vector Databases, GenAI integration
Orchestration & Automation
Cloud Composer (Airflow), CI/CD, Workflows
Metadata & Governance
Data Catalog, lineage, metadata management, observability frameworks
Programming
Python, SQL, PySpark
Qualifications
Bachelor’s or Master’s degree in Computer Science, Engineering, Information Systems, or related field.
7+ years of experience in data engineering and cloud-native data platform development.
Minimum 4+ years of hands-on experience delivering enterprise-scale solutions on GCP.
Strong expertise in building scalable batch and streaming data pipelines.
Experience working on modern enterprise data platforms supporting analytics, AI/ML, and GenAI use cases.
Good understanding of semantic layer concepts, reusable data models, and governed data consumption patterns.
Experience working within large-scale data modernization and cloud transformation initiatives.
Strong problem-solving, debugging, and performance optimization skills.
Proven ability to lead engineering teams and collaborate across architecture, product, and business functions.
Excellent communication and stakeholder management skills.
GCP certifications such as Professional Data Engineer preferred.
Location:
DGS India - Mumbai - Thane Ashar IT Park
Brand:
Merkle
Time Type:
Full time
Contract Type:
Permanent
