Local LLMs.
Sized to fit.
From the right model to a supported AI service: DAXS plans local LLM infrastructure, tests hardware under realistic load and integrates the solution into your existing IT.
An AI server starts with the workload.
A local language model can process internal texts, help draft code or form part of a knowledge search. A useful solution brings together model quality, permitted data flows and expected usage. DAXS captures these requirements, plans the infrastructure and handles installation, integration and agreed operations — alongside your IT department or as a defined service package.
Task and quality
Which questions, languages, documents or coding tasks should the solution handle? A brief chat and an analysis of lengthy documents call for different models and contexts.
Usage and waiting time
How many requests arrive at once? How quickly should the first response begin, and how long can a complete answer take? Occasional individual requests and a continuously used team service need different capacity reserves.
Data and operations
Which sources and functions are approved? Where may the model, search index, logs and backups reside? Who handles updates and faults, and what rollback path is needed?
Connect the application, inference and operations.
The user interface is one component. Identity and access controls sit in front of it; the inference service, model storage and optional knowledge search or business interfaces work behind it. Monitoring, updates and recovery belong in the same plan.
This example architecture shows how the components work together. A reverse proxy can secure TLS and incoming access. The application still needs to enforce user permissions. Model APIs are restricted to the required networks and functions; a reachable inference port needs further integration to become a business service.
More than the size of the model file.
We assess weights, runtime requirements, context, concurrency and the complete data path. A GPU may have enough memory yet still miss the required response time. A smaller, carefully selected model may perform the task more economically.
| Component | What we assess | Why it matters |
|---|---|---|
| GPU and VRAM | Architecture, drivers, inference backend, supported numerical formats, memory capacity and bandwidth. | Weights and the KV cache need space. Compute performance and memory transfers affect different processing phases; an advertised TOPS figure alone says little about the application. |
| One or more GPUs | Model partitioning, connections between GPUs, PCIe connectivity, power and cooling. | VRAM does not automatically become a freely usable shared pool. The chosen stack must support partitioning and synchronisation; additional GPUs can introduce new bottlenecks. |
| CPU and system RAM | Runtime, tokenisation, data preparation, parallel processes and, where applicable, CPU inference or offloading. | Offloading may make a model runnable while changing its performance profile. Transfers between RAM and GPU can slow responses. A GPU server also needs suitable CPU and RAM reserves. |
| NVMe and storage | Model versions, loading times, search index, imported data, logs, free capacity and backups. | Fast NVMe SSDs help with model loading and certain data operations. Token generation for a model held entirely in GPU memory is mainly governed by compute, memory and runtime factors. |
| Network and facilities | Connectivity, permitted data flows, space, power demand, heat output and maintenance access. | Even an excellent compute component delivers limited value if the service faces thermal, organisational or data-access constraints under real load. |
A sizing aid for initial planning.
A rough lower bound for the weights alone is number of parameters × bits per weight ÷ 8 . A hypothetical model with eight billion parameters and four bits per weight gives four gigabytes, or around 3.73 GiB.
Quantisation metadata, potentially higher-precision model components, the KV cache, workspaces and runtime add to this. The figure therefore does not establish how much VRAM your application needs. For mixture-of-experts models, the active compute portion and the total weights that must be held also need to be distinguished.
A short answer is a different test from a team service.
The context window covers the information a model can consider for a request. Depending on the application, this includes instructions, conversation history, retrieved document passages and the planned output. A nominally large context window guarantees neither suitable hardware nor reliable use of the information.
KV cache
The key-value cache stores intermediate attention states. Its requirements depend on factors including model architecture, data type, the context actually occupied and concurrently active sequences. More context and greater concurrency often increase memory demand substantially.
Prefill and decode
During prefill, the model processes the input context; during decode, it generates further tokens. These phases place different demands on hardware. Long documents can increase time to first token even when the subsequent output appears fluent.
Batching and queues
Processing requests together can increase overall throughput. Individual users may nevertheless wait longer. Reserved capacity, queues, limits and appropriate concurrency planning are therefore part of the design.
We measure the actual use case.
A pilot includes typical short and long requests as well as concurrent access. We assess time to first token, complete response time, output rate, queues, error rate and memory usage. Tokens per second become meaningfully comparable only alongside model version, quantisation, context, load and measurement conditions.
The stack follows the model and its use.
Ollama, llama.cpp and vLLM are possible components. Selection and configuration depend on model support, hardware, required concurrency, operations and integration. We assess the specific version and test how it fits the production architecture.
Ollama
A possible starting point for model management and local deployment. Model storage, approved models, context, concurrency and service security must fit the environment.
llama.cpp
A flexible component for suitable models across different CPU/GPU backends, with quantised formats and server operation. Hardware support, partitioning and the API functions actually required are tested in practice.
vLLM
An option for suitable server-side inference and load profiles. Scheduling, KV cache, batching and, where appropriate, multiple GPUs are configured together. Higher overall throughput does not necessarily mean shorter waiting times for every individual request.
Container, virtual machine or dedicated host?
Containers can package software versions reproducibly. A virtual machine can provide further organisational and technical separation. The suitable approach depends on device passthrough, drivers, maintenance and existing operational procedures. A compatible API simplifies integration, but a functional test with your application is still needed.
When does Proxmox with GPU passthrough fit?
GPU passthrough assigns a suitable physical GPU to a virtual machine. The platform and firmware, IOMMU groups, host and guest drivers and device reset behaviour must all be suitable. We assess the specific combination before making a commitment.
An assigned GPU cannot automatically be shared flexibly between multiple VMs. vGPU and direct passthrough are different concepts with their own requirements and potential licensing conditions. Maintenance, backups and a possible host move also need an appropriate procedure; live migration must be assessed for the specific setup.
Infrastructure and virtualisation · Explore Proxmox migration
The right model must pass the task.
We compare models using approved examples from the intended application: German-language specialist questions, document extracts, multilingual requests or a specific scripting task, for example. A coding model and an embedding model serve different purposes. Knowledge search requires retrieval, embeddings and the answer model to work together.
Quantisation often reduces memory demand. Its effect on answer quality, tool calls and speed is tested for the selected format. A larger model is not automatically better or more economical. Nor does a long maximum context establish that the model will reliably use every piece of information it contains.
- Licence: check the chosen model weights, commercial use, redistribution and required notices specifically; downloading the model is only the starting point.
- Version: record the model, quantisation, tokenizer and runtime clearly.
- Acceptance: test subject-matter suitability, disallowed answers, language quality and failure cases against a repeatable question set.
- Changes: test new model versions first, retain a workable rollback path and rebuild search indexes appropriately when embeddings change.
Local, external or a suitable combination.
Own hardware can make sense with suitable utilisation, clear data requirements and an established operational framework. An external service may fit better where usage fluctuates, particular model capabilities are needed or internal operational capacity is limited. A hybrid approach combines specifically defined tasks and data flows.
The comparison covers purchase or usage costs, power and cooling, licences, integration, updates, support, spare capacity and the consequences of downtime. Local processing requires its own security and GDPR assessment: models, embeddings, telemetry, updates, external tools and logs must also fit the agreed data-handling plan.
We first plan a manageable pilot with measurable requirements. Its results inform the decision to expand, adapt or choose a simpler alternative. Further background is available under Cloud & SaaS.
A supported service needs clear responsibilities.
DAXS can plan and install hardware and software, connect them to existing identities and applications and take on agreed ongoing operations. This includes availability and resource monitoring, version maintenance, fault handling and documentation. Required availability and response procedures are agreed specifically.
We define which configurations, application components, index data and any adapted models are backed up. Replaceable model weights and sensitive operational data have different backup requirements. A recovery test forms part of a suitable operating plan.
Requirements gathering, an architecture proposal, model and load testing, documented data flows and a clear recommendation for hardware, deployment and the next integration steps.
We clarify this in our conversation.
How much VRAM does a local LLM need?
That depends on the model and quantisation, runtime, context and concurrent requests. The model file alone is insufficient for sizing. We test the required task, including long requests and load peaks, before recommending a specific configuration.
Is a second GPU automatically faster?
No. The chosen stack must support partitioning. Connections between GPUs, synchronisation and model architecture affect the outcome. More memory can enable execution without accelerating every individual answer.
Can an existing Proxmox server be used?
It may be suitable if the hardware, IOMMU groups, drivers and GPU passthrough fit. Capacity reserves for other services and procedures for maintenance, backups and recovery must also be appropriate. We assess the existing platform specifically.
Are internal documents trained into the model?
This is often unnecessary for knowledge search. RAG retrieves suitable, authorised document passages and supplies them to the model as context. Adjusting model weights is a separate measure with its own justification.
When is an external service more economical?
With low or highly variable usage, particular model requirements or limited operational capacity, an external service may fit better. We compare costs, data flows, quality and operations over the planned period of use.
Monitor models, load and costs together.
Alongside GPU utilisation and response time, we assess cost per useful task. Before deployment, a model update is compared against the approved test set. Error rates, queues, memory limits and failures need named owners and a rollback path. An external fallback service is an option only when the data classification and approval permit it.
Let's discuss your use case.
What should become easier, which data is involved and which systems are already in place? We use this to develop a practical starting point and a manageable next step.