Smartly 5 дней назад

Senior Data Engineer

PythonSQLData EngineeringAirflowDBTBigQueryGoogle Cloud

Описание

Responsibilities

  • Build and operate batch and incremental ingestion pipelines in Google Cloud to Smartly Data Platform.
  • Develop Airflow workflows using Python with best orchestration practices, including dag dependencies, retries, backfills, alerting, and recovery.
  • Build and maintain dbt transformations, from source-aligned staging models to curated warehouse datasets. Add tests and documentation, and support version control and deployment checks.
  • Design practical storage patterns across Cloud Storage and BigQuery, considering formats, data grain, partitioning, clustering, retention, lifecycle, and access controls.
  • Improve data warehouse performance and cost through ongoing query and storage optimization.
  • Document data flows and operational practices so pipelines are understandable, supportable, and safe to extend.
  • Work with analytics, product engineering, and platform colleagues to clarify requirements and deliver dependable datasets.
  • Monitor freshness, completeness, volume, and schema changes so data problems are visible and addressed early.
  • Investigate pipeline failures and data discrepancies from source through downstream models, correct the root cause, and validate recovery.

Requirements

  • Professional experience on building data warehouses starting from data ingestion and landing to publishing data products.
  • Strong SQL and Python skills, with a track record of writing clear, maintainable code.
  • A solid understanding and hands on experience on data storage and warehouse design, including how structure, volume, access patterns, and retention affect reliability, performance, cost and data modelling.
  • Experience developing and supporting Airflow workflows or similar orchestration tool
  • Practical dbt experience, including models, tests, documentation, and version-controlled deployment workflows.
  • Experience with solving production issues in data warehouse.
  • Clear written and spoken English, and the ability to work effectively with engineers and analysts.

Nice to have

  • Google cloud development stack: Google Cloud Storage, BigQuery, Cloud SQL, PostgreSQL, or other Google Cloud data services.
  • Kafka or other event-streaming systems, and familiarity with Kubernetes.
  • Data lineage, cataloguing, observability, Tableau, or Redash.
  • Advertising, marketing, or product analytics data.

We offer

At Smartly, we offer a place where you can do meaningful technical work while growing with a global team.