Skip to content
Comparison

Self-hosted LLM for enterprise: build it yourself or have it managed

Updated
By
The MagnolIA team
On this page
11 min read

Short answer

Self-hosting an LLM is feasible for a company with a technical team: the tooling is open and so are the model weights. The hard part comes once it is running: users, permissions, concurrency, auditing and updates. A managed service keeps the data inside just the same and hands that day-to-day operation to the provider. MagnolIA delivers it that way, for a fixed fee.

Back to contents

What do you have to solve to self-host an LLM?

A self-hosted LLM is one form of on-premise AI: the model runs on hardware the company controls. Getting a model to answer on one machine takes an afternoon. Turning it into a service the whole company uses is a different job, on six fronts.

  1. The hardware

    Choosing GPUs, memory and storage for the models you want and the number of people. Undersize it and it is slow; oversize it and you pay for idle capacity.

  2. The model

    Picking among open-weight models, testing them on your real work and deciding which one suits each task: quick answers, code, writing or analysis.

  3. Concurrency

    A model that answers one person well can choke with twenty at once. Requests have to be distributed across machines and queues managed.

  4. Users and permissions

    Each person with their own identity, permissions and quota, and making sure nobody sees anyone else's conversations. Tools built for a single user do not provide this.

  5. Auditing and security

    A record of who used what, encrypted access, keys that can be revoked, and backups. A model server exposed to the internet without protection is a liability.

  6. Updates

    Better models appear frequently. Someone has to evaluate them, replace the old one and keep the serving software up to date.

Back to contents

What software do you use to self-host an LLM?

Software you download and host yourself, most of it open source. No single piece does everything: the usual set-up combines an engine that runs the model, an interface for people and, in front of both, the company's own identity and encryption. Here are four general-purpose tools, described in the terms of their own documentation. Ollama and llama.cpp are released under the MIT licence and vLLM under Apache 2.0. What has to be added before one of them can serve several people is covered in running Ollama in production.

Common tools for serving models on your own hardware
ToolWhat it isWorth knowing
OllamaA program for downloading and running open models on a machine with very little configuration.By default it listens only on the machine itself (localhost, port 11434) and each model handles one request at a time, though this is configurable. Its local API requires no authentication: if you open it to the network, identity has to be put in front of it. It also offers models in its own cloud, which are optional and can be switched off.
llama.cppAn inference engine written in C/C++, designed to run with minimal set-up on a wide range of hardware.It supports models quantised from 1.5 to 8 bits and splits the work between CPU and GPU when the model does not fit in the card's memory.
vLLMA library for LLM inference and serving, originally developed in the Sky Computing Lab at UC Berkeley.It batches requests that arrive together (continuous batching) and ships an OpenAI-compatible API server. Its documentation warns that the API key alone is not enough to secure it in production and recommends putting it behind a reverse proxy.
Open WebUIA self-hosted platform with a chat interface that connects to Ollama and to any OpenAI-compatible API.It includes roles, groups and per-resource permissions, plus federated sign-in (SSO, OIDC, LDAP). Its code is public, but under its own licence which, according to its documentation, is not an OSI-approved open-source licence and restricts removing the branding: read it before any commercial use.
The descriptions come from each project's documentation and repository, accessed in November 2026. These are not the only tools, nor a recommendation: they change fast and each solves one part.

What none of them solves alone is the whole: who creates accounts, which model each task uses, what gets logged and who answers when it breaks. That is integration and operations, and it is where the time goes.

Back to contents

What hardware do you need to run a model locally?

The part that decides everything is graphics card (GPU) memory. To answer fluently, the whole model has to fit in it, plus the memory each open conversation consumes.

16 GB

is roughly what the weights of an 8-billion-parameter model take up at 16 bits, the format they are usually loaded in. Quantised to 8 bits they take about 8 GB, and at 4 bits, 4 to 5 GB. Fewer bits save memory at the cost of some accuracy. That is the weights alone: the memory for each conversation comes on top.

Source: Hugging Face, accessed November 2026
Model size
More parameters tend to give better answers and need more memory: about 2 bytes per parameter at 16 bits, according to the same documentation.
Quantisation
Storing the weights in fewer bits. It lets a large model fit on more modest hardware, with a loss of accuracy you have to measure on your own tasks.
People at the same time
Every simultaneous request takes memory and compute time. A machine that is comfortable with one person can queue with twenty.
Document length
The more text fits in one request (the context window), the more memory is reserved for each conversation.
Everything else
Storage to keep several models downloaded, power, ventilation and a wired connection to the office network.

This guide gives no price and no card model, because they depend on the model you choose, the number of people and the month you buy. The reliable way to size it is to test with your real usage before buying.

Back to contents

What does it cost to build, and when does it pay for itself?

Buying the hardware is the visible part. The total cost has more lines, and the ones that never arrive as an invoice are the ones people forget.

  • The hardware: GPUs, server, storage and networking. Paid up front, and it ages.
  • Electricity and space: the machine's consumption, its ventilation or cooling, and somewhere with controlled access.
  • The hours of whoever runs it: installing, monitoring, updating, creating users and handling incidents. It is a cost even if that person is already on the payroll.
  • The software around it: identity, backups, monitoring and, depending on the tool, licences.
  • Replacement: more capable models tend to need more memory, and a machine that is fine today can fall short before it has paid for itself.

The other way to pay is a fee that already includes the hardware and its operation. What MagnolIA's covers, and when it is not worth it, is on the on-premise AI cost calculator.

Back to contents

Which mistakes are worth avoiding when self-hosting?

  • Sizing for the demo. It is tested with one person and bought for one person; with the whole office, it queues.
  • Opening the port to the internet. A model server with no identity in front of it answers whoever finds it.
  • No owner. Someone enthusiastic builds it and, when they move to another project, nobody updates it.
  • Choosing the model by this month's leaderboard. What matters is how it answers on your documents, not its rank in a table.
Back to contents

Build it yourself or have it managed, point by point

A self-built AI server compared with a managed service
CriterionYou build itManaged (MagnolIA)
Where the data isIn your network.In your network. MagnolIA only sees the machine's monitoring.
Who chooses and buys the hardwareYou do. It is an upfront investment.MagnolIA sizes it for your use and includes it in the fee: hardware as a service.
Who selects and updates the modelsYour technical team.MagnolIA selects, tests and updates them.
Identity, permissions and quotasTo be built or integrated.Included in MagnolIA Server: users, roles, quotas and narrowly scoped API keys.
Audit and backupsTo be designed.A chained audit log kept for two years and AES-256 encrypted backups.
SupportYour own team, plus each tool's community.Support and continuous monitoring.
CostThe hardware purchase plus the hours of whoever maintains it.A fixed monthly fee per capacity tier.
Technical controlTotal: you choose every component and can change it.Less: users have no technical access to the system. MagnolIA operates it.
Back to contents

When does it make sense to build it yourself?

There are cases where it is the best option.

  • You have a technical team with time, and the appetite to maintain the platform, not just set it up.
  • You want full control over every component: the exact model, its configuration, how it is deployed.
  • It is for a few technical people, who can tolerate something breaking and know how to fix it.
  • You are exploring. To learn what a local model can do, building one is the best school.
  • AI is part of your product and you need to integrate it at a depth a closed service does not allow.
Back to contents

When is a managed service the better choice?

  • There is no technical team, or there is one and it has other work to do.
  • Non-technical people will use it: back office, lawyers, clinical staff. They need it to work every time, and someone to answer when it does not.
  • You must be able to show who accesses what, with logging and auditing from day one.
  • You do not want to buy hardware that will need replacing in a few years.
  • The data is sensitive and a configuration mistake is expensive.

With MagnolIA, the service comes with the apps people actually work in as well as the server: MagnolIA Nexus and the VS Code extension. How it is built is explained in MagnolIA's AI server, and what the fee includes in what on-premise AI costs.

Are you in a position to self-host? A checklist

Seven questions. If the answer is yes to all of them, building it yourselves is reasonable. If two or more get a no, look at the managed option first.

  1. Is there an owner?

    A named person with hours allocated to run it, and someone who can cover for them.

  2. Has it been tested on real use?

    At least two models, on your own documents, with several people at once.

  3. Does each user have their own identity?

    And can access be revoked the same day someone leaves?

  4. Is it closed to the outside?

    The server is reachable only from the internal network or through encrypted, authenticated access.

  5. Are there logs and backups?

    Usage is recorded and a restore from backup has actually been tried.

  6. Who changes the model?

    It is decided who evaluates new models, and with which tests.

  7. Have the full sums been done?

    With staff hours and hardware replacement, not just the purchase.

If the underlying reason is keeping information away from third parties, the possible routes are compared in private AI for business. And if the documents contain personal data, either option benefits from one step beforehand: anonymising documents before giving them to the AI.

Back to contents

Can you combine the two?

Yes, in the way that matters to a technical team. MagnolIA Server exposes an API compatible with OpenAI's and Anthropic's, so your developers connect their own tools and integrations by changing the server address and the key. They get a maintained service and keep building on top of it.

If the underlying question is a different one (whether to keep AI in-house or use a cloud service), the comparison is in on-premise AI vs cloud AI. And if it cannot depend on a connection, see air-gapped AI.

Back to contents

Frequently asked questions about the AI server

What software do you use to self-host an LLM?

A self-hosted LLM is built with general-purpose tools, most of them open source: an engine that runs the model, such as Ollama, llama.cpp or vLLM, and an interface, such as Open WebUI. Tying them to the company's identity system, sizing them and keeping them up to date is integration and operations work: what a managed service like MagnolIA takes on.

What hardware do you need to run an AI model locally?

To run an AI model locally at a usable speed you need, above all, a graphics card (GPU) with enough memory to hold the model. According to Hugging Face's documentation, the weights of an 8-billion-parameter model take about 16 GB at 16 bits and 4 to 5 GB quantised to 4 bits. Every simultaneous request adds memory.

How many users can a MagnolIA machine support?

The number of users depends on the tier. MagnolIA's hardware is called Núcleo and comes in four indicative tiers: Núcleo S, around 18 users; Núcleo M, around 40; Núcleo L, around 100; and Núcleo XL, around 250. MagnolIA sizes it according to how many people will use the AI and what they will use it for.

What happens if my team grows?

If your team grows, capacity is added without downtime: MagnolIA adds hardware to the Núcleo without interrupting the service and without anything changing for users. The server distributes requests across the available machines and keeps each conversation on the same one to make use of its cache.

Who administers the system?

MagnolIA administers the system. Users only interact with the chat, the app or the endpoint; they have no technical access to the system, the models or the infrastructure. Operation, hardware maintenance and model updates are our responsibility, and administration requires two-factor authentication.

Is it compatible with the tools my team already uses?

Yes. MagnolIA Server exposes endpoints compatible with the OpenAI API and the Anthropic API. The tools your team already uses, such as Claude Code, Cursor, Cline or your own integrations, work by changing only the server address and the key.

Sources

External figures on this page link to their original source.

  1. Hugging Face. Transformers documentation: Quantization, overview. Accessed November 2026. huggingface.co/docs/transformers/main/en/quantization/overview

  2. arXiv (Pan, Chodnekar, Roy and Wang). A Cost-Benefit Analysis of On-Premise Large Language Model Deployment: Breaking Even with Commercial LLM Services (arXiv:2509.18101, version 3; preprint). 11 November 2025. arxiv.org/abs/2509.18101

  3. Ollama. Official documentation: FAQ. Accessed November 2026. docs.ollama.com/faq

  4. Ollama. Official API documentation: Authentication. Accessed November 2026. docs.ollama.com/api/authentication

  5. llama.cpp. Official project repository: description and features. Accessed November 2026. github.com/ggml-org/llama.cpp

  6. vLLM. Official documentation. Accessed November 2026. docs.vllm.ai/en/latest/

  7. vLLM. Official documentation: Security. Accessed November 2026. docs.vllm.ai/en/latest/usage/security/

  8. Open WebUI. Official documentation: features. Accessed November 2026. docs.openwebui.com/features/

  9. Open WebUI. Official documentation: Open WebUI License. Accessed November 2026. docs.openwebui.com/license/

Build or buy? Let's work it out together

Tell us what technical team you have, how many people would use AI and with what data. If building it yourselves is the better deal, we will say so too.

Request a pilot