Cloudflare 2 часа назад

Senior Data Engineer, Cloudforce One

  • London, United Kingdom
  • Гибрид
  • Опыт от 5 лет
PythonSQLKubernetesCI/CDClickHouseAirflowSparkIceberg

Описание

Responsibilities

  • Design, build, and operate scalable batch data pipelines on Kubernetes-hosted Airflow and Spark clusters that extract, enrich, and publish threat and abuse signals from Cloudflare’s network and platform telemetry.
  • Build and evolve our data stores in Apache Iceberg (R2) and the curated data marts that serve Threat Events, Trust & Safety initiatives, and Threat Intelligence investigations.
  • Model datasets deliberately — choosing grain, partitioning, and schema evolution strategies that keep data correct, queryable, and cost-efficient.
  • Write and optimize SQL against large-scale analytical data in ClickHouse, balancing completeness, performance, and cost.
  • Make pipelines safe to retry, resume, and backfill, so downstream consumers receive exactly the data we intend to publish.
  • Build data quality checks, reconciliation, and monitoring that catch problems before they reach analysts, detections, or customers.
  • Treat privacy and data minimization as design requirements from the start.
  • Enable Threat Intelligence and Trust & Safety partners with the data they need for tooling and investigations, translating ad hoc requests into reusable datasets.
  • Partner with data scientists to turn exploratory analysis into repeatable, production-quality data products and ML-ready datasets.
  • Raise the team’s engineering bar through code review, testing, CI/CD, and documentation.

Requirements

  • 5+ years of experience designing, building, and operating scalable production data pipelines.
  • Proven experience building analytical data stores and data marts, including data modeling, partitioning, and schema evolution.
  • Strong Python for data engineering, including testing and maintainable, well-structured code.
  • Strong SQL, including window functions, query optimization, and performance tuning on large datasets.
  • Hands-on experience with Apache Airflow in production, including Kubernetes-hosted deployments.
  • Experience with distributed data processing (e.g. Spark or similar) at large scale.
  • Experience with Apache Iceberg (or a comparable open table format) on object storage such as R2 or S3.
  • Experience with ClickHouse or a comparable columnar analytical database.
  • Experience with CI/CD, version control, and containerized deployments.
  • Clear communication and the ability to turn ambiguous requirements from researchers and analysts into well-defined data products.

Nice to have

  • Experience with security, threat intelligence, fraud, or trust & safety / abuse data.
  • Solid understanding of how the Internet works (DNS, HTTP, TLS, BGP).
  • Experience supporting data science or ML workflows (feature stores, training datasets).
  • Experience with Cloudflare’s developer platform (Workers, R2, R2 Data Catalog).
  • Trino/Presto or other distributed query engines.
  • Experience using AI-assisted development tools as part of your workflow.

We offer

  • This role may require flexibility to be on-call outside of standard working hours to address technical issues as needed.

Hiring process

  • Please note that applicants who progress to the offer stage of the interview process may be asked to attend an in-person interview within one of the Cloudflare Offices or Cloudflare Hubs. More details about this will be available at that stage of the interview process.