Skip to content
Module
Module 5 of 6Lesson 2 of 2~17 min

Test the contract on 20 cases chosen to break it

Testing an AI output on easy examples proves nothing. This lesson teaches you to build a small set of cases designed to break your contract, decide what you will measure and set your acceptance thresholds before you run the test, so the result cannot be bent to fit what you hoped to see.

Take this lesson: your first course is freeSee the full syllabus

Lesson objective

By the end of this lesson, you will be able to build a set of 20 robustness cases spread over eight families, choose the metrics to measure and set acceptance thresholds before measuring.

Topics covered

  • robustness testing
  • edge cases
  • acceptance thresholds
  • LLM evaluation

Where it fits

Keep the contract alive

How do I evolve the schema without breaking the product, and how do I know it holds?

Lessons in this module

  1. Evolve the contract without breaking the product
  2. Test the contract on 20 cases chosen to break it (this lesson)

What you will learn in this module

This lesson is part of the course Get reliable, product-ready AI outputs (JSON, schemas, function calling)

  • Version and evolve an output contract without breaking its consumers, accounting for provider differences and the split of roles between PM and developers.
  • Build a set of 20 robustness cases and set acceptance thresholds before going to production.

Related courses