A quick answer to one short question shows that a local language model basically works. For an infrastructure decision, testing the tasks you will actually run and the load they generate is much more informative. This includes answer quality, waiting times, longer inputs, concurrent use and operation after a failure.
A pilot is particularly useful when the decision it should enable is clear in advance. For example: is the existing environment sufficient, is different hardware needed, or would an external or hybrid service be a better fit? That question guides the test plan.
Start with a small, representative set of tasks
Suitable test tasks come from the intended workflow: a short technical question, a summary of a longer approved document or a draft script to review. For each task, define what makes a result useful.
Knowledge questions should include expected sources and cases with insufficient evidence. Code needs technical review and appropriate tests. Confidential original data is used only in an environment approved for that purpose; suitable sanitised or purpose-built test data may be sufficient for preparation.
Assess quality and speed separately
A model that produces incorrect results quickly does not meet the objective. First assess practical usefulness, required corrections and handling of missing information. Then compare the response time and effort needed to achieve that quality.
Record the model version, quantisation, inference software, settings and test data. Otherwise, a faster response may reflect a changed task or a different quality level. Changes to the document collection can also affect comparisons of a RAG application.
Break waiting time down into its components
- Time to first token
- This describes when output starts. Queuing and input processing can affect it. Also measure the perceived time from submitting a request in the application to seeing the response.
- Output generation
- Generation speed affects how long a longer answer takes. High total server throughput does not by itself indicate how quickly an individual user receives an answer.
- Total duration
- This covers the entire application operation. Search, document processing, tool calls and network paths may add time.
- Variation and errors
- Consider typical values alongside slower edge cases. The number of measurements, timeouts and abandoned tasks are part of the result.
The official vLLM metrics documentation (external link) distinguishes between waiting, input and output measurements, among others. Check which metrics the particular runtime provides for the version in use. Measurements from different components must refer to the same period and task.
Test short chats, long contexts and concurrency
A long context requires additional processing and memory. Concurrent requests share the available resources. The Ollama context documentation (external link) and the vLLM optimisation documentation (external link) explain these relationships; specific settings depend on the version and hardware.
A useful test therefore combines short and longer inputs with concurrency that reflects the intended use. Include expected peaks as well as a steady flow of requests. If batching increases throughput, still check whether individual response times meet users’ needs.
An example test plan
First run the same technical tasks individually. Then test short requests alongside approved document tasks. Next, simulate an expected peak period and observe behaviour under overload. Agree target values with users beforehand.
This plan is an adaptable example. It contains no measured DAXS performance figures and does not replace an assessment of the particular environment.
Distinguish startup, normal operation and recovery
After a restart, the model may need to be loaded first. Measure this cold start separately from operation with the model already loaded. Fast NVMe drives can help with loading and data operations; that does not imply a general improvement in token generation for a model held entirely in GPU memory.
The pilot also covers a lost connection, an interrupted job and a service restart. The application should show its state clearly and provide a sensible retry or handover path. Overload must not silently result in incomplete answers.
Turn results into an architecture decision
Compare useful results, correction effort, latency, capacity and ongoing operations. Total costs also include administration, maintenance, energy, backup and required availability. For external services, include their actual usage costs and the agreed data paths.
The outcome may be a smaller suitable model, additional resources, a bounded queue or a different operating model. Assess local, external and hybrid approaches against the same task. A pilot should enable a sound next decision, rather than retrospectively justify buying hardware already assumed necessary.
DAXS plans the test, sets up a suitable environment and evaluates the results with your IT team. Our specialist page on local LLMs and infrastructureexplains the technical basis. For connected knowledge collections, the quality of search, sources and permissions belongs in the same test framework.
