Описание

This role involves building Cyera's next-generation data platform, which will serve as the lakehouse foundation for data processing across the entire product. It combines infrastructure ownership with designing data pipelines, focusing on end-to-end platform management, including its infrastructure, frameworks, tooling, and data pipeline design.

Responsibilities

Design, deploy, and operate the Spark-on-Kubernetes compute platform, managing autoscaling, resource tuning, and multi-tenancy
Own the lakehouse storage layer built on Apache Iceberg and AWS Glue catalog, covering table design, partitioning, compaction, schema evolution, and retention
Build and operate orchestration on Airflow, including DAG standards, deployment flows, environment promotion, and reliability
Own production operations of the platform: monitoring, alerting, incident response, and continuous hardening
Build code frameworks, libraries, and templates for other engineers to write pipelines, enabling rapid development of new production-grade Spark jobs
Define and enforce standards for pipeline structure, testing, observability, and deployment across teams
Own CI/CD for data workloads: image builds, artifact promotion, and GitOps-based delivery
Act as a technical partner to product and research teams building on the platform, supporting other engineers
Design and build scalable batch and streaming pipelines for complex, high-volume datasets from diverse sources
Lead large-scale backfills and migration initiatives, ensuring data consistency and integrity across evolving storage and compute platforms
Design event-driven data flows over large-scale queue systems (Kafka) for reliable, efficient data movement
Continuously balance cost against performance: right-size compute, tune queries and jobs, optimize storage layout and file sizes, and choose the correct engine for each workload
Build cost visibility and attribution into the platform to inform data-driven trade-offs

Requirements

5+ years of experience in software engineering, with significant time spent building and operating large-scale data platforms
Strong hands-on experience with distributed processing engines (Spark strongly preferred), including performance tuning and debugging in production
Practical experience deploying and operating workloads in Kubernetes-based environments, with an enjoyment for infrastructure work
Experience building shared frameworks, libraries, or internal tooling for other engineers, demonstrating a product mindset with clean APIs, documentation, versioning, and backward compatibility
Strong proficiency in SQL and data modeling: complex analytical queries, query tuning, and partitioning strategies
Solid software engineering fundamentals in Python (and/or Scala/Java), including testing, code review culture, and CI/CD
Experience with cloud-native data infrastructure on AWS (or equivalent) at high scale
A strong sense of ownership from design through deployment, operation, and cost
Ability to thrive in a fast-changing environment, making pragmatic decisions with incomplete information

Nice to have

Hands-on experience with Apache Iceberg or other open table formats (Delta Lake, Hudi), including compaction, schema evolution, and catalog management
Experience with EMR on EKS, Spark Operator, or similar Spark-on-Kubernetes setups
Experience with Airflow at scale (custom operators, deferrable operators, multi-environment deployment)
Experience with Kafka-heavy or event-driven architectures; CDC pipelines (Debezium or similar)
Knowledge of GitOps tooling and infrastructure-as-code
Experience with FinOps / cloud cost optimization for data workloads
Background in cybersecurity-related data infrastructure or compliance-constrained environments (e.g., FedRAMP / GovCloud)
Contributions to open-source data engineering tools or frameworks

We offer

Opportunity to own what you build and how you work, taking initiative and moving quickly
Work at the technical heart of the company's data platform, solving problems at real scale
Shape how an entire engineering organization works with data
A culture that challenges the status quo, encourages fast learning, and continuously raises the bar
A supportive environment for diverse backgrounds, perspectives, and experiences

Hiring process

Feel free to apply even if your experience doesn't tick every box