GenAI Staff Machine Learning Engineer, AI Runtime

You will:

• Design and productionize state of the art tooling and open source technologies to enable the development of frontier foundation models for Databricks customers

• Solve complex problems at scale around data preprocessing, model training, hyperparameter tuning and model evaluation

• Implement advanced optimization techniques to reduce the resource footprint of models while preserving their performance and balancing usability for our developers and customers

• Collaborate with product managers and cross-functional teams to drive technology-first initiatives that enable novel business strategies and product roadmaps

• Facilitate our user community through documentation, talks, tutorials, and collaborations

• Contribute to the broader AI community by publishing research, presenting at conferences, and actively participating in open-source projects, enhancing Databricks' reputation as an industry leader.

• Below are some example projects:

• Composer: Large-scale distributed deep learning training library

• Streaming: Library for efficient data loading from cloud object storage

• LLM Foundry: Framework for developing and evaluating Large Language Models

We look for:

• Hands on experience with the internals of deep learning frameworks (e.g. PyTorch, TensorFlow) and GenAI models (e.g. GPT, StableDiffusion)
• Experience with large scale, distributed training on GPUs (e.g., Nvidia, AMD) and alternative deep learning accelerators
• Strong sense of design and usability
• Effective communication skills and the ability to articulate complex technical ideas to cross-disciplinary internal and external stakeholders
• Prior history of contributing to or developing open source projects is a bonus but not a requirement

We value candidates who are curious about all parts of the company's success and are willing to learn new technologies along the way.