Senior Software Engineer - Husky Storage
192 000 – 240 000 $в год
DatadogBoston и ещё 2 городаГибрид
JavaKubernetesKafkaGoDistributed systemsRustS3IcebergParquetData fusionArrowStorage Engines
Описание
Senior software engineer role on Datadog's Husky distributed petabyte-scale event storage team.
Responsibilities
- Drive Husky to the next level of adoption by migrating our storage format to open source Parquet and re-imagine compaction on a format-agnostic, DataFusion-based substrate
- Drive concrete cost-reduction work: block-level inverted index compression, combining the database with external objects at the row level, and storage tiering/caching
- Scale our solution to enable customers to manage event versioning at scale, using temporal-database techniques
- Play an influential role in operating and deciding how to best operate our service going forward
- Work across the write path (with Ingestion) and toward a cleaner Iceberg-based handoff to ADP/Lakehouse for batch/analytical workloads
Requirements
- Enthusiastic and experienced with large-scale distributed systems and/or storage engines
- Comfortable with improving and operating large distributed systems that customers rely on for critical business functions
- Strong fundamentals in columnar/file storage formats (Parquet, ORC, Arrow) or comparable systems
Nice to have
- Experience with Go, Java, or Rust
- Experience with compaction or compression/encoding schemes (RLE, dictionary, delta, gorilla)
- Experience with object storage (S3/GCS/Azure Blob) at scale, including caching and cost optimization
- Experience with Kafka and stream processing
- Experience with Kubernetes
We offer
- New hire stock equity (RSUs) and employee stock purchase plan (ESPP)
- Continuous professional development, product training, and career pathing
- Intradepartmental mentor and buddy program for in-house networking
- An inclusive company culture, ability to join our Community Guilds (Datadog employee resource groups)
- Free, global mental health benefits for employees and dependents age 6+
- Competitive salary and equity package, and may include variable compensation
- Wide range of best in class, comprehensive and inclusive employee benefits including healthcare, dental, parental planning, and mental health benefits, a 401(k) plan and match, paid time off, fitness reimbursements, and a discounted employee stock purchase plan
- Salary: $192,000 - $240,000 USD per year