Fix the test conditions before you begin
Agree on participants, business questions, reference answers and a data snapshot. Record the product and model versions, deployment environment, access identity and data source scope. Save each natural-language question exactly as asked, along with its business meaning, so the target stays consistent throughout testing.
Agree in advance which errors prevent launch and which differences are acceptable. You might allow a specified rounding tolerance for amounts. Unauthorized disclosure, an incorrect period or treating missing data as fact generally require separate handling. Both project teams set the actual thresholds.
Include cases the system cannot answer
| Test category | What to observe |
|---|---|
| Routine question | Correctness, units, period and source |
| Follow-up questions | Conditions retained and changes explicitly requested |
| Missing data | An accurate description of the gap and a next step |
| Out-of-scope access | No disclosure of unauthorized content |
| Outdated material | Recognition of version and freshness limits |
| Report export | A complete file that opens and whose references can be checked |
Keep an acceptance record for each question
- Record the question number, wording, test identity, input scope and agreed expectation.
- Retain the raw output and supporting sources alongside the final screenshot.
- Record start and finish times. Distinguish the query, cache and generation stages.
- Mark each test as passed, partially passed or failed. Describe any difference and the conditions for retesting.
- After changing configuration, definitions or versions, rerun the same questions and keep both sets of results.
Use the findings to make a launch decision
A short live demonstration is suitable for checking interaction and verifiability. It does not cover all the data cleaning, integration or security assessment required for launch. Assign owners, scope and acceptance criteria to pilot findings, then test recovery, access revocation and exception handling.
Write a brief conclusion identifying supported business uses, cases that still require human review and capabilities outside this test. A record of what passed gives future teams more to work with than a general positive impression.
