LatticeStaff Engineer, AI Quality and Evaluation
Engineering at Lattice: AI Engineering team defines how intelligence works across our platform. Responsible for architecting and scaling infrastructure powering AI quality, reliability, and reuse across Lattice.
This is Engineering at Lattice
At Lattice, we build software that helps people and organizations thrive. Our AI Engineering team defines how intelligence works across our platform - how AI systems are measured, improved, and trusted in production.
This Staff-level role shapes the foundations that determine AI quality, reliability, and impact at scale.
What You Will Do
You will architect and scale the infrastructure that powers AI quality, reliability, and reuse across Lattice.
AI Evaluation & Quality
- Design and scale an end-to-end AI evaluation framework spanning offline evals, production tracing, and human feedback loops.
- Define meaningful performance metrics (task completion, hallucination, response quality, engagement, business impact) and build the datasets and automated scoring systems that prevent regressions.
- Identify and quantify the drivers of agent quality improvement and set methodological standards for evaluation across the organization.
Agent Architecture & Production Systems
- Architect reusable agent infrastructure (multi-turn workflows, LLM DAGs, recommendation systems, standardized topologies) using LangGraph or comparable frameworks.
- Build and scale RAG pipelines, vector retrieval systems, and production-grade AI infrastructure with strong reliability, observability, and performance.
- Make principled build-vs-buy decisions across LLM providers, agent frameworks, and evaluation tooling, balancing capability, cost, latency, and risk.
- Engineer AI systems as reusable internal platforms that multiply product engineering velocity at Lattice.
Technical Leadership
- Own projects end-to-end: scope, design, execution, and delivery.
- Set technical direction for agent quality and evaluation strategy across Lattice engineering teams.
- Lead rigorous discussions on AI system design and evaluation methodology.
- Raise the AI engineering bar through mentorship, code review, and clear technical communication across engineering and leadership.
What You Will Bring to the Table
Experience
- 8+ years of professional experience writing and maintaining production-level code, with 5+ years in designing, delivering, and operating AI/ML systems in production.
- Deep production experience with LLM systems (prompting, RAG, agent orchestration, evaluation frameworks, fine-tuning).
- Experience building and operating agentic systems (multi-step workflows, multi-agent topologies) and managing their failure modes.
- Strong command of AI evaluation methodology and statistical experimentation.
- Strong system design judgment across scalability, latency, accuracy, reliability, and cost.
Technical Skills
- Production-grade Python (clean, maintainable, testable systems).
- Experience with LangGraph (or comparable agent orchestration frameworks) and LLM observability/evaluation tooling (e.g., LangSmith).
- Vector databases and retrieval system design (Pinecone or similar).
- Experience operating AI systems in AWS or comparable cloud environments, including CI/CD, monitoring, and deployment workflows.
- Familiarity with TypeScript is a plus.
- Actively engaged in AI research and industry trends.
Nice to Have
- Experience with RLHF, LoRA, or other model adaptation techniques.
- Background in traditional ML and judgment in selecting ML vs. LLM approaches.
- Experience with MLOps tooling (MLflow, DataDog).
- Published research, talks, or open-source contributions in AI/ML.
- Experience in HR tech or other trust-sensitive domains.
The estimated annual cash salary for this role is CAD $146,250 - CAD $195,000. This position is also eligible for incentive stock options, subject to the terms of Lattice’s applicable plans.
Benefits: The Company offers the following benefits for this position, subject to applicable eligibility requirements: Medical insurance; Dental insurance; Life, AD&D, and Disability Insurance; Natural Disaster Support Program; Wellness Apps; Paid Parental Leave, Paid Time off inclusive of holidays and sick time; Working Remotely Stipend; One time WFH Office Set-Up Stipend; Retirement Plan; Financial Planning; and Learning & Development Budget.
Note on Pay Transparency:Lattice provides an estimate of the compensation for roles that may be hired as required by regulations. Compensation may vary based on (a) location, as Lattice factors in specific location when benchmarking compensation for most roles; (b) individual candidate skills and qualifications; and (c) individual candidate experience. Additionally, Lattice leverages current market data to determine compensation, so posted compensation figures are subject to change as new market data becomes available. The salary, other compensation, and benefits information is accurate as of the date of this posting. Lattice reserves the right to modify this information at any time, subject to applicable law.
Lattice welcomes and encourages applications from people with disabilities. Accommodations are available on request for candidates taking part in all aspects of the recruitment and selection process.
About Lattice
Lattice is on a mission to build cultures where employees and their companies thrive. In an age where employees have more choices than ever before, businesses that put employees first are winning 🏅– and Lattice is building the tools to empower those people-centric companies.
Lattice is a people success platform that offers performance reviews, employee engagement surveys, real-time feedback, weekly check-ins, goal setting, and career planning in a way that allows companies to focus on employee development, growth, and engagement – yielding stronger employee retention, performance, and impact to the bottom line 📈. Since launching in 2016, we have grown to over 5,000+ customers globally, including brands like Loom, Robinhood, and Gusto.
Lattice is committed to equal treatment and opportunity in all aspects of recruitment, selection, and employment without regard to gender, race, religion, national origin, ethnicity, disability, gender identity/expression, sexual orientation, veteran or military status, or any other category protected under the law. Lattice is an equal opportunity employer; committed to a community of inclusion, and an environment free from discrimination, harassment, and retaliation.
By clicking the "Submit Application" button below, you consent to Lattice processing your personal information for the purpose of assessing your candidacy for this position in accordance with Lattice's Job Applicant Privacy Policy.