Senior Data Engineer
qlubBengaluru и ещё 1 городУдалённоот 5 лет
PythonSQLPostgreSQLCI/CDClickHouseAirflowGCPDBTBigQuery
Описание
Responsibilities
- Own how data gets in: CDC from our transactional databases, SaaS sources, application events, and third-party APIs.
- Handle the unglamorous parts well: backfills, retries, schema changes, late-arriving data.
- Own the dbt project end to end: staging through marts, tests, docs, incremental models, and the naming and contract conventions everyone follows.
- Keep BigQuery fast and cheap: partitioning, clustering, query tuning, and cost monitoring.
- Serve low-latency analytics from an OLAP store such as ClickHouse: table design, sorting keys, materialized views, aggregation strategy.
- Look after the GCP estate around the warehouse: Cloud Storage, Cloud Run services, IAM, and the occasional read from sources hosted elsewhere.
- Run production Airflow: scheduling, dependencies, retries, alerting, and the operational habits that go with it.
- Put freshness and quality checks in place, and fix root causes when they fire rather than restarting the job.
- Set up CI/CD and tests for data code so changes ship safely.
- Find where LLMs and agents genuinely help (data discovery, quality monitoring, troubleshooting), and ship those.
- Review code, mentor junior and mid-level engineers, and write down the decisions and runbooks the team will need later.
Requirements
- 5+ years in data or backend engineering, most of it on production systems.
- Strong Python, with years of writing and debugging production pipelines and services.
- Advanced SQL: window functions, complex transformations, and knowing why a query is slow.
- 2+ years of production dbt experience, including incremental models, macros, tests and docs.
- Experience running production Airflow DAGs, including backfills, monitoring and troubleshooting when things break.
- Strong BigQuery skills: modelling, partitioning, clustering, and query and cost tuning.
- Good knowledge of GCP, including Cloud Storage, IAM and general cloud data infrastructure.
- Solid PostgreSQL experience: indexing, query performance and incremental extraction.
- Experience with stream or distributed processing, such as Apache Flink or similar.
- Comfortable working on a Linux VM: Bash, cron, Docker, reading logs, SSH and basic networking.
- Strong engineering practice with Git, CI/CD, tests and code review, writing code other people can maintain.
Nice to have
- Experience with ClickHouse or production OLAP: tuning, materialized views and cluster operations.
- Experience with managed ETL/ELT platforms such as Airbyte or Fivetran.
- Experience with Datastream for CDC and database replication.
- Experience with Kafka and Debezium for event streaming and change data capture.
- Experience with Kubernetes and Terraform.
- Experience with BI tools such as Tableau or Power BI, and a feel for what makes dashboards slow.
- Experience with LLM APIs, RAG and agent frameworks.
- Experience with data observability tools such as Soda, Great Expectations or Monte Carlo.
- Experience with AWS (EC2, S3, RDS).
- A CS or engineering degree, though a strong track record matters more.
We offer
- Real ownership over products that impact millions of diners and thousands of restaurants globally
- The opportunity to solve challenging problems in payments, hospitality, and customer experience
- A fast-moving environment where great ideas quickly become reality
- Close collaboration with experienced operators, builders, and industry experts
- A chance to leave a visible mark on one of the fastest-growing restaurant technology companies globally