Data Platform Engineer
As a Data Platform Engineer, you will be a key contributor in building and maintaining the new Operational Data Store. You will develop the infrastructure that captures, stores, and publishes master data across the enterprise, transitioning away from legacy point-to-point integrations to a highly decoupled event-driven architecture
Expanded
Responsibilities
Pipeline Engineering:
Build and maintain batch and real-time data pipelines using
Dataflow
and
Kafka
.
Orchestration:
Develop and monitor complex
Airflow (Cloud Composer)
DAGs to ensure data arrives on time.
Infrastructure:
Use
Terraform
to deploy and manage GCP resources, ensuring environmental consistency.
Software Development:
Create internal APIs and microservices using
FastAPI
and
Golang
to enhance platform utility.
Data Modelling:
Use
dbt
to transform raw data into clean, tested models within
BigQuery
.
Reliability:
Manage
Docker
containers and CI/CD pipelines (GitLab/Jenkins) to support seamless deployments.
Technical
Profile
Additions
Proficiency:
Python (Advanced), Golang (Intermediate), and SQL.
GCP Expertise:
Cloud Run (Services, Jobs, Functions)
Databases: BigQuery, Cloud SQL for PostgreSQL, Cloud Spanner
Cloud Storage
Managed Airflow (Cloud Composer)
Pub/Sub, Eventarc
Artifact Registry
Secret Manager
NICE TO HAVE:
Project Management & Documentation:
Jira, Confluence, ServiceNow
Frontend Technologies:
React.js
Educational Background & Qualifications
Required Education:
Master’s or Bachelor’s Degree in Computer Science, Data Engineering, Software Engineering, or a related technical field (e.g., Electromechanical Engineering with a focus on Automation)
Specialized Coursework: Strong academic foundation in Distributed Systems, Database Management Systems (DBMS), Algorithm Design, and Real-time Computing.
Preferred Certifications:
Google Cloud Professional Data Engineer.
