Many companies want to use AI chatbots to summarize documents, answer internal questions or write drafts, but they do not want to send contracts, financial data or customer files to an outside service. One answer is to run a language model on the company’s own server. It can be done, but it is very different from installing an ordinary application, and it should be chosen with eyes open.
What does running a model locally mean?
Many language models are now released with “open weights”: the model file can be downloaded and run on your own hardware. Families such as Llama, Mistral and Qwen are well-known examples. To run them, tools such as Ollama or llama.cpp make it easy to get started, while tools such as vLLM are designed to serve many users. The model runs on your server, and users work with it through a web interface or an internal API.
Benefits
- Data stays in the company: Questions and documents never leave the internal network. This is the main reason to choose this approach.
- Less dependence on outside services: Changes in a provider’s terms, loss of access, or restrictions that may apply to users in Iran will not stop your work.
- Predictable cost: The main cost is buying and maintaining hardware, not paying per request.
- Room to customize: You can connect the model to internal documents or tune it for a specific task.
Drawbacks and limitations
- Different quality: Models that fit on an ordinary server usually fall short of the largest cloud models on complex tasks. Test them on your own real work.
- Upfront hardware cost: A suitable graphics card, plus power and cooling, is not a small expense.
- Ongoing maintenance: Updating the model, drivers and libraries, and fixing problems, is your team’s job, not a vendor’s.
- Security is on you: A service you launch on the internal network becomes an entry point if it is poorly configured.
- License terms: Open-model licenses are not all the same, and some attach conditions to commercial use. Read the license for the exact version before you use it.
Technical prerequisites
- GPU memory (VRAM): This is the main factor in choosing a model. The memory needed is roughly the number of model parameters times the bytes per parameter, plus extra room for the input text. Compressed (quantized) versions need less memory but may lose some accuracy.
- Virtualization: If the server is virtualized, the graphics card must be given to the virtual machine through GPU Passthrough or vGPU. Capabilities and licensing differ between Proxmox, VMware and Hyper-V, and between card types.
- Fast storage: Model files are large and load much faster from NVMe drives.
- Network and access: Put the service behind the firewall, on a separate VLAN, with user authentication.
Never leave the model service’s API open to the internet, or even to the whole internal network, without authentication. Expose these tools’ default port only to the authorized server or interface.
Security and data governance
Running locally does not make anything secure by itself. Decide who can access the service, whether conversations are stored, and for how long. Download model files only from the official source and verify their checksum when possible. Back up settings, scripts and any data you have connected to the model on a regular schedule.
Checklist
- You have written down the specific tasks the model must handle and tested it on real samples of them.
- You have checked the model’s license for commercial use.
- You have estimated GPU memory, power and cooling for the chosen model.
- The service sits behind the firewall and requires authentication.
- There is a clear policy for retaining conversation text.
- Someone is clearly responsible for updating and monitoring the service.
If you decide to run AI locally, ITPlus724 engineers can prepare and secure the server, virtualization, network, firewall and backup your company needs for the service.