Crusoe 1 час назад
Staff Engineer, Managed Data Transfer (Distributed Systems)
215 000 – 260 000 $в год
- San Francisco, CA, US
- Опыт от 8 лет
KubernetesGoDistributed systemsData TransferPetabyte Scale
Описание
Responsibilities
- Own the end-to-end design of MDT: the job API and scheduler, the transfer workers on Kubernetes, storage integration and network paths.
- Evaluate the production prototype, decide what gets refactored, extended or replaced, and move its current users onto the managed service without interruption.
- Set the data-plane strategy (sharding, concurrency, straggler handling, checksum verification) and the benchmarks that prove it, aiming for line-rate transfers over dedicated interconnects.
- Own the interfaces between MDT and Storage, Network, Kubernetes, IAM, Console, Billing and Monitoring.
- Design for new clouds and Crusoe-internal transfers with a pluggable transfer engine and regional worker pools, and evaluate open-source and vendor options before the team builds.
- Set the bar for reliability, observability and on-call for every system the team ships.
- Work with Product on scope, pricing trade-offs and milestones, and surface edge cases early enough to change the plan.
- Raise the team's level through design reviews, RFC feedback and direct coaching, so the team runs well without depending on you.
Requirements
- Hands-on experience designing and operating data-intensive distributed systems at scale, including the failure modes that only appear in production.
- 8+ years shipping production software, including leading a service from prototype to general availability.
- Understanding of object storage semantics, file systems and TCP throughput, and the ability to profile and fix a transfer that is slower than it should be.
- Proven ability to evolve a live system toward a better design while keeping risk to current customers low.
- Experience defining API boundaries and schema contracts with partner teams, and negotiating them to agreement.
- You have carried customer-facing on-call and built systems that are maintainable, observable and predictable.
- Proficiency in Go or another modern compiled language (Go strongly preferred).
- You can explain an architectural decision to an engineer, a PM and an EM, and you know which parts matter to each.
Nice to have
- Experience building or operating bulk data transfer systems, such as AWS DataSync, Google Storage Transfer Service, or rclone at petabyte scale.
- Familiarity with cloud networking and interconnects, such as AWS Direct Connect or GCP Interconnect.
- Experience with cross-account cloud IAM and threat modeling for services that hold customer credentials.
- Approximate nearest-neighbor (ANN) indexing in depth: HNSW, IVF, product and scalar quantization, DiskANN-style indexes that live on SSD.
- Experience with AI, HPC or Physical AI data pipelines.
We offer
- Competitive compensation and equity packages
- Restricted Stock Units
- Paid time off, paid holidays & leave of absence programs
- Comprehensive health, dental & vision insurance
- Employer contributions to HSA account
- Paid parental leave
- Paid life insurance, short-term and long-term disability
- Professional development & tuition reimbursement
- Mental health & wellness support
- Commuter benefits (parking & transit)
- Cell phone stipend
- 401(k) Retirement plan with company match up to 4% of salary
- Volunteer time off
- Global travel insurance & emergency assistance
- Daily meals allowance
- Additional perks & programs specific to location
Hiring process
- Compensation will be paid in the range of up to $215,000 -$260,000 + Bonus.
- Restricted Stock Units are included in all offers.