Senior ML Engineer | Kimchi (LLM Inference Optimization)

Описание

This role focuses on optimizing LLM inference, specifically improving throughput, latency, and KV cache utilization to achieve faster, cheaper inference for customers and improve margins. It involves leading the technical direction of inference optimization at Kimchi, working with a system that automatically and continuously matches workloads to the most cost-efficient and best-performing LLM and serving configuration on a customer's infrastructure.

Responsibilities

• Push throughput: continuous batching, speculative decoding, chunked prefill, kernel-level tuning across vLLM, SGLang, and TensorRT-LLM, finding and raising the ceiling on each GPU SKU
• Cut latency: attack TTFT and TPOT separately, profile, identify the actual bottleneck (compute, memory bandwidth, scheduling, networking), and fix it
• Get more out of the KV cache: paged attention, prefix caching, eviction policies, cache reuse across requests, quantized KV
• Quantize without regressing quality: INT8, INT4, FP8 across weights, activations, and KV, measuring quality on real workloads
• Shrink cold starts and memory footprint: faster init, smarter weight loading, tighter memory accounting
• Scale across nodes: distributed inference topologies, network-aware placement, checkpointing strategies that do not bottleneck on storage or interconnect
• Set the technical direction: decide what to benchmark, adopt, and build, and bring the team along with strong writeups and reproducible experiments

Requirements

• 5+ years building real ML systems, with a portfolio showing depth in inference or training infrastructure (not just model training notebooks)
• Strong Python skills for production services, not scripts
• Hands-on experience with at least one of vLLM, SGLang, or TensorRT-LLM, and a working mental model of why an inference engine performs as it does on a given GPU
• Fluency with quantization tradeoffs, having measured quality regressions, not just compression ratios
• Comfort with distributed systems: collective communication, sharding strategies, and practical failure modes of multi-GPU and multi-node setups
• A bias toward measurement: instrument before optimizing, and differentiate between real wins and benchmark artifacts
• Self-direction with a wide mandate

We offer

• Competitive salary (depending on the level of experience)
• Flexible, remote-first global environment
• Equity options
• Quick feedback with a fast-paced workflow; most feature projects are completed in 1 to 4 weeks
• Spend 10% of work time on personal projects or self-improvement
• Learning budget for professional and personal development, including access to international conferences and courses
• Annual hackathon
• Team-building budget and company events
• Equipment budget
• Extra days off

Hiring process

• Screening call with Recruiter
• Hiring Manager interview
• Technical interview (system design)
• Live coding
• Culture Check interview with an executive
• A background check may be conducted at the final stage of recruitment through Checkr
• Cast AI does not provide any form of visa sponsorship/work permit