Define what the timer measures
The time from submitting a question to receiving a usable answer can include entry-point handling, task planning, querying, retrieval, generation and transfer. A fast internal stage may have little effect on the end-to-end wait. Time to the first visible word and time to a complete result measure different experiences.
Define start and finish conditions before measuring. Record time to the first visible output, complete answer and final file separately, along with the operations that ran.
Separate cold starts, first questions and warm requests
A first run may establish connections or provision resources. Repeating a question may hit a cache. Measure and report those groups separately. For a cached answer, also check data freshness and access scope.
Record the deployment environment, model and configuration, data volume, query complexity, question set and concurrency pattern. When one condition changes, identify it before comparing the new figure with an earlier measurement.
Look beyond the average
A few slow requests can dominate the experience for the users who encounter them. Report the median, a higher latency percentile, the failure rate and unfinished tasks. State the population and sample size used to calculate each percentile.
Distinguish users present at the same time, requests submitted simultaneously and sustained throughput of completed work. A user count alone tells you little about what the users do or how resource use and latency change.
Report answer quality alongside timing
| Measurement | What to record |
|---|---|
| Speed | Timing boundaries and separate groups for first and cached requests |
| Stability | Completion rate, failure types and cancellation behavior |
| Quality | Correctness against the baseline, sources and metric definitions |
| Load | Request spacing, concurrency pattern and duration |
| Scope | Environment, data, versions and known limitations |
Make the test repeatable
Keep the synthetic or approved question set, expected results, run configuration and original timestamps. Review errors and outliers before increasing load. Preserve citations, permission checks and answer quality throughout the benchmark so improvements in timing remain comparable.
This article describes an evaluation method. It reports no measured performance for an AI-Soda Brain deployment. Base a selection decision on validation that matches your business questions, data scope and operating conditions.
