Responsibilities

• Design and ship LLM-based agents and multi-step workflows: tool use, planning, retrieval (RAG), memory, and multi-agent orchestration.
• Make deliberate architecture choices and integrate agents with internal data and APIs via function calling and MCP, with guardrails, fallbacks, and human-in-the-loop checkpoints.
• Take systems from prototype to production, owning latency, cost, and reliability budgets.
• Define what "good" means for each use case with business stakeholders, define eval criteria and build eval suites
• Develop and calibrate LLM-as-judge graders against human labels; measure judge agreement and bias.
• Evaluate both final outputs and trajectories and run evals in CI so prompt, model, and tool changes are gated on measured quality, not vibes.
• Instrument agents with structured tracing (spans for LLM calls, tool calls, retrieval).
• Monitor quality, hallucination and grounding rates, latency, token cost, and failure modes on live traffic.
• Run systematic error analysis on traces, detect drift and regressions when upstream models or data change, fine-tune or distill models (LoRA, SFT, preference tuning).
• Train classical or deep models where they beat an LLM on cost, latency, or accuracy 
• Design experiments with proper statistics, set team standards for evals and observability
• Explain system behavior, eval results, and trade-offs clearly to non-technical leaders.

Skills & Experience

• Degree in Computer Science or a related field; Master's or PhD in ML is a plus.
• 5+ years of hands-on ML engineering, with models you trained, evaluated, and deployed to production
• Strong grasp of the fundamentals: generalization, overfitting and leakage, evaluation metrics, calibration, and experimental design.
• Experience with PyTorch and the Hugging Face ecosystem; working understanding of transformer internals, embeddings, and tokenization.
• Experience with MLOps
• 2+ years building LLM applications, including agentic or multi-step tool-using systems.
• Hands-on with at least one agent framework (LangGraph, Pydantic AI, OpenAI Agents SDK, or similar) and a clear view of when not to use one.
• Working knowledge of modern RAG design.
• Experience with structured outputs, function calling, and prompt and context engineering for frontier models (Claude, GPT, Gemini) and open-weight models.
• Experience in building eval framework or suite from scratch, not just run benchmarks.
• Experience with LLM tracing and eval tooling such as Langfuse, LangSmith, Arize Phoenix, Braintrust, or Weights & Biases Weave.
• Advanced Python and production-quality code: typing, testing, async, and API design.
• Strong SQL for analyzing traces, logs, and large datasets.
• Cloud deployment experience, containers, and CI/CD.

Nice to have

• Background in AI safety, red-teaming, or guardrails (prompt injection, PII leakage, jailbreak testing).
• Experience in financial services or other regulated domains where auditability and traceability matter.
• Publications, open-source contributions, or public write-ups on evals, agents, or ML systems.
• Comfort working in ambiguous, fast-moving environments and taking ideas from PoC to production with light guidance.