> For the complete documentation index, see [llms.txt](https://docs.artific.nl/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.artific.nl/en/for-administrators/quality/scenarios.md).

# Scenarios and checks

Writing what to test. Found on the **Scenarios** tab of a quality control.

## Scenarios

A scenario is one situation. It has a **Title** and a **Description**, and contains one or more steps.

Use one scenario per topic rather than one enormous scenario with twenty steps. Failures are much easier to interpret when the scenario name tells you what area broke.

Good scenario titles describe the situation, for example *Leave policy: carrying days over*, *Refuses to give legal advice*, *Order lookup with a valid order number*.

## Steps

A step is a question plus the checks its answer must pass. **Add step** creates one.

**When the user asks** holds the question, phrased the way a real user would phrase it. Use their words, including the awkward ones. This is a test of what happens in practice, not of your ideal phrasing.

Steps in a scenario run in order and build on each other, so you can test a conversation rather than a single exchange: ask a question, then a follow-up that only makes sense in context.

## Checks

Each step carries one or more checks. Each has a **Required to pass** box: required checks failing produce **Issues**, optional ones produce a **Warning**.

| Check                          | What it verifies                                             | Give it                        |
| ------------------------------ | ------------------------------------------------------------ | ------------------------------ |
| **Exact match**                | The answer matches exactly.                                  | The expected answer.           |
| **Close match**                | The answer says the same thing, allowing for wording.        | The expected answer.           |
| **Contains text**              | A specific phrase appears.                                   | The text that must appear.     |
| **Does not contain text**      | A phrase does not appear.                                    | The text that must not appear. |
| **Correct format**             | The answer follows a required shape.                         | A description of the format.   |
| **No hallucination**           | The answer is grounded, not invented.                        | Nothing.                       |
| **Valid URLs**                 | Any links given actually work.                               | Nothing.                       |
| **Polite tone**                | The tone is appropriate.                                     | Nothing.                       |
| **Must use tool**              | The assistant used a tool rather than answering from memory. | Nothing.                       |
| **Responds within time limit** | The answer arrived in time.                                  | Maximum seconds.               |
| **LLM Judge**                  | Anything else, assessed by the judge model.                  | A judge prompt.                |

**Attachments** lets you upload files the check can refer to.

## Choosing the right check

**Close match** rather than **Exact match**, almost always. Models rephrase. Exact match fails on a comma and produces alarms nobody trusts.

**Contains text** for facts that must be present: the actual number of days, the correct deadline, the phone number.

**Does not contain text** for things that must never appear: a competitor's name, an outdated policy name, a phrase like "I am an AI language model" if you have told it not to say that.

**Must use tool** for anything that should come from a live lookup rather than the model's memory. This catches a whole class of silent failure.

**LLM Judge** for everything the mechanical checks cannot express. Write the judge prompt as a criterion, not a question:

> *Evaluate whether the response correctly states that unused leave expires at the end of March, mentions that an extension can be requested from HR, and does not give advice about individual contracts.*

Be specific about what must be present. A vague judge prompt gives vague, unstable scoring.

## Practical advice

* **One thing per check.** A step with a single clear check tells you what broke. A step with six does not.
* **Make critical checks required and stylistic ones optional.** Otherwise every run is red and people stop reading them.
* **Test refusals as well as answers.** *"Should not answer questions about individual salaries"* is exactly as important as the questions it should answer.
* **Add a scenario whenever something goes wrong in real use.** That is how the suite grows into something valuable.