Cloudflare 2 часа назад
Senior Data Engineer, Cloudforce One
- London, United Kingdom
- Гибрид
- Опыт от 5 лет
PythonSQLKubernetesCI/CDClickHouseAirflowSparkIceberg
Описание
Responsibilities
- Design, build, and operate scalable batch data pipelines on Kubernetes-hosted Airflow and Spark clusters that extract, enrich, and publish threat and abuse signals from Cloudflare’s network and platform telemetry.
- Build and evolve our data stores in Apache Iceberg (R2) and the curated data marts that serve Threat Events, Trust & Safety initiatives, and Threat Intelligence investigations.
- Model datasets deliberately — choosing grain, partitioning, and schema evolution strategies that keep data correct, queryable, and cost-efficient.
- Write and optimize SQL against large-scale analytical data in ClickHouse, balancing completeness, performance, and cost.
- Make pipelines safe to retry, resume, and backfill, so downstream consumers receive exactly the data we intend to publish.
- Build data quality checks, reconciliation, and monitoring that catch problems before they reach analysts, detections, or customers.
- Treat privacy and data minimization as design requirements from the start.
- Enable Threat Intelligence and Trust & Safety partners with the data they need for tooling and investigations, translating ad hoc requests into reusable datasets.
- Partner with data scientists to turn exploratory analysis into repeatable, production-quality data products and ML-ready datasets.
- Raise the team’s engineering bar through code review, testing, CI/CD, and documentation.
Requirements
- 5+ years of experience designing, building, and operating scalable production data pipelines.
- Proven experience building analytical data stores and data marts, including data modeling, partitioning, and schema evolution.
- Strong Python for data engineering, including testing and maintainable, well-structured code.
- Strong SQL, including window functions, query optimization, and performance tuning on large datasets.
- Hands-on experience with Apache Airflow in production, including Kubernetes-hosted deployments.
- Experience with distributed data processing (e.g. Spark or similar) at large scale.
- Experience with Apache Iceberg (or a comparable open table format) on object storage such as R2 or S3.
- Experience with ClickHouse or a comparable columnar analytical database.
- Experience with CI/CD, version control, and containerized deployments.
- Clear communication and the ability to turn ambiguous requirements from researchers and analysts into well-defined data products.
Nice to have
- Experience with security, threat intelligence, fraud, or trust & safety / abuse data.
- Solid understanding of how the Internet works (DNS, HTTP, TLS, BGP).
- Experience supporting data science or ML workflows (feature stores, training datasets).
- Experience with Cloudflare’s developer platform (Workers, R2, R2 Data Catalog).
- Trino/Presto or other distributed query engines.
- Experience using AI-assisted development tools as part of your workflow.
We offer
- This role may require flexibility to be on-call outside of standard working hours to address technical issues as needed.
Hiring process
- Please note that applicants who progress to the offer stage of the interview process may be asked to attend an in-person interview within one of the Cloudflare Offices or Cloudflare Hubs. More details about this will be available at that stage of the interview process.