Skip to content
Module
Module 3 of 6Lesson 3 of 3~14 min

Check an LLM's coding on a sample

A model that sorts thousands of feedback items in minutes inspires confidence, rightly or wrongly. This lesson gives a lightweight method to find out whether its labels hold up, theme by theme, and what to do when they do not. You will be able to show your team why the numbers deserve their trust.

Updated November 14, 2026.

In short

To validate an LLM's coding, you randomly pick a sample of feedback, have two people label it independently, then compare their choices with the model's, one theme at a time. For a PM, this comparison shows where the figures are solid and which themes need redefining before any decision is taken.

Lesson objective

By the end of this lesson, you will be able to measure the quality of an LLM's coding on a sample reviewed by humans and fix the taxonomy where the model gets it wrong.

Questions this lesson answers

  • How do you check the reliability of an LLM's coding?

    By comparing its labels with those of human coders on a random sample, one theme at a time, and repeating the check after every change to the definitions.

  • Does an LLM code qualitative data as well as a human?

    Published research often finds agreement close to human levels when the model has clear definitions, but across very different datasets: you need to confirm it on your own data.

Topics covered

  • validate LLM coding
  • intercoder reliability
  • validation sample
  • AI coding accuracy

Where it fits

Use what your customers already tell you

How do you get a reliable reading out of hundreds of tickets, reviews and comments, and know what it does not tell you?

Lessons in this module

  1. Take stock of what your customers already tell you
  2. Code feedback at scale with an LLM
  3. Check an LLM's coding on a sample (this lesson)

What you will learn in the course

This lesson is part of the course Set up continuous discovery with AI