Описание
Responsibilities
- Build and operate batch and incremental ingestion pipelines in Google Cloud to Smartly Data Platform.
- Develop Airflow workflows using Python with best orchestration practices, including dag dependencies, retries, backfills, alerting, and recovery.
- Build and maintain dbt transformations, from source-aligned staging models to curated warehouse datasets. Add tests and documentation, and support version control and deployment checks.
- Design practical storage patterns across Cloud Storage and BigQuery, considering formats, data grain, partitioning, clustering, retention, lifecycle, and access controls.
- Improve data warehouse performance and cost through ongoing query and storage optimization.
- Document data flows and operational practices so pipelines are understandable, supportable, and safe to extend.
- Work with analytics, product engineering, and platform colleagues to clarify requirements and deliver dependable datasets.
- Monitor freshness, completeness, volume, and schema changes so data problems are visible and addressed early.
- Investigate pipeline failures and data discrepancies from source through downstream models, correct the root cause, and validate recovery.
Requirements
- Professional experience on building data warehouses starting from data ingestion and landing to publishing data products.
- Strong SQL and Python skills, with a track record of writing clear, maintainable code.
- A solid understanding and hands on experience on data storage and warehouse design, including how structure, volume, access patterns, and retention affect reliability, performance, cost and data modelling.
- Experience developing and supporting Airflow workflows or similar orchestration tool
- Practical dbt experience, including models, tests, documentation, and version-controlled deployment workflows.
- Experience with solving production issues in data warehouse.
- Clear written and spoken English, and the ability to work effectively with engineers and analysts.
Nice to have
- Google cloud development stack: Google Cloud Storage, BigQuery, Cloud SQL, PostgreSQL, or other Google Cloud data services.
- Kafka or other event-streaming systems, and familiarity with Kubernetes.
- Data lineage, cataloguing, observability, Tableau, or Redash.
- Advertising, marketing, or product analytics data.
We offer
At Smartly, we offer a place where you can do meaningful technical work while growing with a global team.