Senior / Lead Machine Learning Engineer, Serving
Work on optimizing realtime inference for state-of-the-art voice models, taking models from research into production by containerizing, optimizing, and operating them reliably. Build distributed systems for high-throughput inference and treat performance, latency, and reliability as first-class product features.
Responsibilities
- Optimize realtime inference using serving frameworks such as vLLM and TRT-LLM.
- Apply quantization, distillation, caching, continuous batching, paged attention, and speculative decoding.
- Profile and optimize NVIDIA GPU performance using C++, CUDA, Rust, or optimized Python.
- Build and scale distributed systems with Kubernetes, Ray, and custom load balancing.
- Support reliable multi-GPU and multi-node inference for thousands of concurrent connections.
- Take models from research through containerization, serving optimization, and reliable production operation.
- Design benchmarks and prototypes to validate architectural decisions.
- Collaborate daily with US-based leadership and engineering teams.
Requirements
- Deep understanding of modern serving frameworks and inference optimization techniques.
- Hands-on experience with quantization, distillation, caching, continuous batching, paged attention, and speculative decoding.
- Proficiency in C++, CUDA, Rust, or highly optimized Python.
- Experience with Kubernetes, Ray, custom load balancing, and multi-GPU or multi-node inference.
- Non-trivial systems programming projects, open-source contributions, or deep-dive technical write-ups.
- Full-cycle experience taking models from research to production.
- PhD in computer science, physics, or mathematics, or equivalent practical experience building backend or ML systems.
- Professional fluency in written and spoken English.
Benefits
- Full U.S. visa and relocation support may be available for candidates interested in relocating to the San Francisco Bay Area, subject to business needs and applicable legal and work authorization requirements.