Over 1.000 comments
AI Evals For Engineers & PMs – No.1 Course at Maven
Here’s What You Get:
WHAT TO EXPECT
This course will provide you with hands-on experience. Get ready to sweat through exercises, code and data! We will meet two times a week for four weeks, with generous office hours (read below for course schedule).
We will also hold office hours and host Discord community where you can communicate with us and each other. In return, you will be rewarded with skills that will set you apart from the competition by a wide margin. (see testimonials below). All sessions will be recorded and available to students asynchronously.
—
COURSE CONTENT
Lesson 1: Fundamentals & Lifecycle LLM Application Evaluation
– Why evaluation matters for LLM applications – business impact and risk mitigation
– Challenges unique to evaluating LLM outputs – common failure modes and context-dependence
– The lifecycle approach from development to production
– Basic instrumentation and observability for tracking system behavior
– Introduction to error analysis and methods for categorizing failures
Lesson 2: Systematic Error Analysis
– Bootstrap data through effective synthetic data generation
– Annotation strategies and quantitative analysis of qualitative data
– Translating error findings into actionable improvements
– Avoiding common pitfalls in the analysis process
– Practical exercise: Building and iterating on an error tracking system
Lesson 3: Implementing Effective Evaluations
– Defining metrics using code-based and LLM-judge approaches
– Techniques for evaluating individual outputs and overall system performance
– Organizing datasets with proper structure for inputs and reference data
– Practical exercise: Building an automated evaluation pipeline
Lesson 4: Collaborative Evaluation Practices
– Designing efficient team-based evaluation workflows
– Statistical methods for measuring inter-annotator agreement
– Techniques for building consensus on evaluation criteria
– Practical exercise: Collaborative alignment in breakout groups
Lesson 5: Architecture-Specific Evaluation Strategies
– Evaluating RAG systems for retrieval relevance and factual accuracy
– Testing multi-step pipelines to identify error propagation
– Assessing appropriate tool use and multi-turn conversation quality
– Multi-modal evaluation for text, image, and audio interactions
– Practical exercise: Creating targeted test suites for different architectures
Lesson 6: Production Monitoring & Continuous Evaluation
– Implementing traces, spans, and session tracking for observability
– Setting up automated evaluation gates in CI/CD pipelines
– Methods for consistent comparison across experiments
– Implementing safety and quality control guardrails
– Practical exercise: Designing an effective monitoring dashboard
Lesson 7: Efficient Continuous Human Review Systems
– Strategic sampling approaches for maximizing review impact
– Optimizing interface design for reviewer productivity
– Practical exercise: Implementing a continuous feedback collection system
Lesson 8: Cost Optimization
– Quantifying value versus expenditure in LLM applications
– Intelligent model routing based on query complexity
– Practical exercise: Optimizing a real-world application for cost efficiency
What you’ll get out of this course
Acquire the best tools for finding, diagnosing, and prioritizing AI errors.
We’ve tried all of them so you don’t have to.
Learn how to bootstrap with synthetic data for testing before you have users
And how to best leverage data when you do have users.
Create a data flywheel for your applications that guarantees your AI will improve over time.
Data flywheels ensure you have examples to draw from for prompts, tests, and fine-tuning.
Automate parts of your AI evaluation with approaches that allow you to actually trust and rely on them.
How can you really trust LLM-as-a-judge? How should you design them? We will show you how. We will also show you how to refine prompts, generate metadata, and other tasks with the assistance of AI.
Ensure your AI is aligned to your preferences, tastes and judgement.
We will show you approaches to discover all the ways AI is not performing
Avoid common mistakes we’ve seen across 35+ AI implementations.
There are an infinite number of things you can try, tests you can write, and data you can look at. We will show you a data-driven process that helps you prioritize the most important problems so you can avoid wasting time and money.
Hands-On Exercises, Examples and Code
We will provide end-to-end exercises, examples and code to make sure you come away with the skills you need. We will NOT just throw a bunch of slides at you!
Personalized Instruction
Generous office hours ensures students can ask questions about their specific issues and interests.
What’s included
Lifetime Access to All Recordings & Materials
Revisit the materials and lectures anytime. Recordings and slides are made available to all students.
Lifetime Access To Discord Community
Private discord for questions, job leads, and ongoing support from the community (over 1000+ students and growing).
9+ Office Hour Q&As
Open office hours for questions and personalized feedback.
150+ Page Course Reader (Draft of Our O’Reilly Book)
We provide a comprehensive course reader with detailed notes to serve as a future reference on evals
4 Homework Assignments With Solutions & Walkthroughs
Optional coding assignments & walkthrough videos so you can practice every concept.
Professionally Edited Lectures (Unique To This Course)
High production quality lectures, edited & organized by chapters (with notes) to save you time. This is the only course on Maven like this.
Certificate of Completion
Share your new skills with your employer or on LinkedIn.
Reviews
There are no reviews yet.