Crusoe 1 час назад

Staff Engineer, Managed Data Transfer (Distributed Systems)

215 000 – 260 000 $в год
  • San Francisco, CA, US
  • Опыт от 8 лет
KubernetesGoDistributed systemsData TransferPetabyte Scale

Описание

Responsibilities

  • Own the end-to-end design of MDT: the job API and scheduler, the transfer workers on Kubernetes, storage integration and network paths.
  • Evaluate the production prototype, decide what gets refactored, extended or replaced, and move its current users onto the managed service without interruption.
  • Set the data-plane strategy (sharding, concurrency, straggler handling, checksum verification) and the benchmarks that prove it, aiming for line-rate transfers over dedicated interconnects.
  • Own the interfaces between MDT and Storage, Network, Kubernetes, IAM, Console, Billing and Monitoring.
  • Design for new clouds and Crusoe-internal transfers with a pluggable transfer engine and regional worker pools, and evaluate open-source and vendor options before the team builds.
  • Set the bar for reliability, observability and on-call for every system the team ships.
  • Work with Product on scope, pricing trade-offs and milestones, and surface edge cases early enough to change the plan.
  • Raise the team's level through design reviews, RFC feedback and direct coaching, so the team runs well without depending on you.

Requirements

  • Hands-on experience designing and operating data-intensive distributed systems at scale, including the failure modes that only appear in production.
  • 8+ years shipping production software, including leading a service from prototype to general availability.
  • Understanding of object storage semantics, file systems and TCP throughput, and the ability to profile and fix a transfer that is slower than it should be.
  • Proven ability to evolve a live system toward a better design while keeping risk to current customers low.
  • Experience defining API boundaries and schema contracts with partner teams, and negotiating them to agreement.
  • You have carried customer-facing on-call and built systems that are maintainable, observable and predictable.
  • Proficiency in Go or another modern compiled language (Go strongly preferred).
  • You can explain an architectural decision to an engineer, a PM and an EM, and you know which parts matter to each.

Nice to have

  • Experience building or operating bulk data transfer systems, such as AWS DataSync, Google Storage Transfer Service, or rclone at petabyte scale.
  • Familiarity with cloud networking and interconnects, such as AWS Direct Connect or GCP Interconnect.
  • Experience with cross-account cloud IAM and threat modeling for services that hold customer credentials.
  • Approximate nearest-neighbor (ANN) indexing in depth: HNSW, IVF, product and scalar quantization, DiskANN-style indexes that live on SSD.
  • Experience with AI, HPC or Physical AI data pipelines.

We offer

  • Competitive compensation and equity packages
  • Restricted Stock Units
  • Paid time off, paid holidays & leave of absence programs
  • Comprehensive health, dental & vision insurance
  • Employer contributions to HSA account
  • Paid parental leave
  • Paid life insurance, short-term and long-term disability
  • Professional development & tuition reimbursement
  • Mental health & wellness support
  • Commuter benefits (parking & transit)
  • Cell phone stipend
  • 401(k) Retirement plan with company match up to 4% of salary
  • Volunteer time off
  • Global travel insurance & emergency assistance
  • Daily meals allowance
  • Additional perks & programs specific to location

Hiring process

  • Compensation will be paid in the range of up to $215,000 -$260,000 + Bonus.
  • Restricted Stock Units are included in all offers.