Consider a hypothetical situation. A small team runs an internal document assistant on a single GPU server, and the demonstration goes well. Within weeks, other departments ask for access. Someone asks why it gave a confident answer from a procedure withdrawn months ago.
These questions extend beyond model selection. Who decides which teams can use it, and which documents each may see? Who updates the model and checks the new one is no worse? Who investigates a poor answer? If the server fails on a Monday morning, who restores it?
The pilot showed the technology works, not what it costs to run as a service.
This article concerns self-hosted inference: existing models on enterprise-controlled servers for shared use, not individual laptop use or model training. Not every deployment needs a large GPU estate or round-the-clock support, but whatever service level is promised has to be funded.
Where the estimate becomes incomplete
Hardware, power, cooling and software arrive as invoices, so they reach the business case. Continuing work arrives as staff time, and is easier to omit.
Some work needs new spending: an extra support contract, a recovery server, specialist help. Other work is absorbed by existing staff. That is not free, but charging a whole engineer's salary to one assistant overstates the cost as badly as charging nothing understates it. The honest figure is the share of time the service will realistically consume, estimated before approval and checked after a few months.
It also helps to separate work that exists however the model is hosted from work that self-hosting adds. Evaluation, source permissions and application ownership are needed with a managed AI service too. Self-hosting adds the platform beneath them: patching the stack, providing capacity, recovering the service and replacing the hardware.
Keeping the platform current
A local inference service is a stack that changes on different schedules: operating system, GPU driver, compute libraries, serving runtime, dependencies and model files. The layers are coupled. CUDA toolkit and driver versions must satisfy NVIDIA's documented compatibility requirements; some toolkit updates run on an existing driver, while others need a driver change.
Return to the document assistant. A security fix arrives for its serving runtime, and the new version needs a newer driver. The platform team tests the combination; the application team reruns its evaluation questions to confirm answers have not shifted; both agree a restart window. That effort may be absorbed into existing team budgets rather than appearing as a separate AI service cost.
The work to fund is a controlled change process: pinned versions, a test environment and a rollback path. The ownership question: who approves a change to this stack, and who can reverse it out of hours?
Capacity and reliability
A server that answers one person quickly can make twenty wait at the start of the day. Size fixed hardware for the peak and much of it sits idle; size it for the average and the peak becomes a queue. Where there is only one serving instance, maintenance that requires restarting it interrupts the service unless requests can go elsewhere.
Recovery should match criticality: a drafting aid may tolerate a day's outage; an assistant in a daily process may need a second node. Whatever level is chosen, recovery must restore the service configuration, the approved model and runtime versions, and the retrieval data or a tested way to rebuild it.
A healthy server can still deliver a poor service
A service can be up and responsive while users receive wrong answers. Retrieval may return a superseded document, indexing may split a table badly, or an update may change behaviour nobody retested. Infrastructure health metrics alone will not reveal these failures.
Consider the withdrawn procedure. Someone has to establish why it surfaced: was it still in the source library, still in the index, or ranked above its replacement? Depending on the cause, the fix may require correcting the source, updating or rebuilding the affected index, and rerunning evaluation questions. One document has triggered investigation, platform and testing time.
The work to fund is representative questions with known good answers, run before any change; a way for users to report poor answers; and someone able to locate the fault in the source, retrieval or model. Answer quality needs explicit accountability. That may sit with an existing service or use-case owner; it does not always need a separate person or team.
Data and security do not stay in the server room
Local hosting still leaves application, data and infrastructure risks to manage. If the document index is built with an account that can read everything, the assistant may show users content they are not permitted to see. Retrieval has to respect source permissions, and the index has to reflect documents that are changed, restricted or deleted. OWASP lists vector and embedding weaknesses as a distinct risk in its 2025 Top 10 for LLM applications.
Prompts and logs need retention decisions. Serving software needs hardening: the vLLM security documentation notes that its API key protects only some endpoints. Model files carry provenance risk; Hugging Face's pickle guidance explains that loading some formats can execute code.
People and lifecycle ownership
Infrastructure supports the server and the application team the interface; "the answer was wrong" falls between them unless support boundaries and escalation are written down. Specialist knowledge often sits with one or two people, so leave cover is a cost to plan.
An open-weight model is not automatically free of conditions: the Llama 3.2 Community License, for example, incorporates an acceptable use policy and requires very large licensees to seek a separate licence. Each model change is also a licence check, and replacement and retirement, including disposal of indexes and logs, belong in the plan.
The operating obligations at a glance
| Operational obligation | Work that must be funded | Owner to name |
|---|---|---|
| Platform currency | Tested updates to OS, drivers, runtime and models; pinned versions; rollback | Platform owner |
| Capacity and continuity | Testing at realistic concurrency; maintenance windows; recovery matched to criticality | Service owner |
| Answer quality | Evaluation questions; retesting before changes; investigating reported answers | Use-case owner |
| Data access | Permission-aware retrieval; index updates and deletions; log retention | Data owner, with security |
| Supply chain and licences | Model and software provenance checks; licence review for each model version | Security and legal reviewers |
| Lifecycle | Skills cover and escalation; replacement planning; retirement and data disposal | Service owner, with budget holder |
These are roles to assign, not a standard structure; one person may hold several.
Comparing costs fairly
A fair comparison holds constant the workload, quality standard, demand pattern, security requirements and service level over the same period. Count shared work, such as evaluation, permissions and application ownership, on both sides. On the local side, add hardware allocated over a stated useful life, support, licences, power and cooling, patching, capacity and recovery. On the managed side, add usage charges at realistic volume.
Be clear which question is being answered. An incremental decision, whether to add this service to an existing platform, counts the costs the service adds. A fully allocated comparison also includes a share of shared-platform costs. Both an incremental cost and an allocated share can appear when they are different expenses; the rule is never to count the same expense twice.
GPU price against API token price is incomplete: one is a capital cost whose cost per answer depends on utilisation, the other a usage charge, and neither includes the surrounding work. Tokens per second shows how fast text is produced, not whether answers are correct or users are queuing; time to a useful answer under realistic concurrency is the more useful measure.
When local is the sound choice
Local inference tends to be sustainable when a model of manageable size handles the task well, demand is steady, the organisation already runs comparable platforms, the data-control requirement is documented rather than assumed, the service must work offline, or processing benefits from sitting close to its data or devices.
"Local" does not by itself make a service secure, compliant, private or disconnected; the complete design and its controls do. Nor is cloud always cheaper or simpler; hybrid adds integration work and a second operating model.
Five questions before a pilot becomes a service
- What business outcome and service level are we committing to?
- Who owns the whole service, and who funds its continuing work?
- What measured workload and quality evidence supports the design?
- How will changes, incidents and recovery be handled?
- What would trigger scaling, replacement, migration or retirement?
Before widening access or making a service commitment, resolve these questions. If self-hosting creates responsibilities the organisation cannot support, assess a managed option against the same requirements. When they are resolved, approve the service's operating commitment, with named owners, funded work, a stated service level and a review point, in the same decision as its infrastructure investment.