How to self-host an LLM on your own server

Why self-host an LLM at all

The main reason is control. Your prompts and documents never leave a machine you rent, nothing is logged by a third party, and nobody can change or retire the model under you. That matters for customer data, internal documents and anything covered by a contract or privacy law.

The second reason is predictable cost at high, steady volume. A server costs the same whether you send ten requests or ten thousand. The flip side: an idle GPU still bills by the hour, so self-hosting only pays off when you keep it busy or switch it off between jobs.

Ollama, llama.cpp or vLLM: which runtime

All three are free and open source. They differ in who they are built for.

• Ollama - one install script, then `ollama pull` and `ollama run`. Great for a single user, a small team, or an app that calls a local API. Uses quantized models, so it fits more into less memory.
• llama.cpp - the engine underneath many tools, including Ollama. Use it directly when you want fine control, CPU-only inference, or the smallest possible footprint.
• vLLM - a serving engine for GPUs. It batches many requests at once and exposes an OpenAI-compatible API, so it shines when several users or services hit the model at the same time.
• Put Open WebUI or a similar front end on top if you want a ChatGPT-style chat window.

CPU or GPU: what your server needs

Memory decides whether a model runs at all; the processor decides how fast. On a CPU-only VPS the model sits in regular RAM, which is cheap, but answers come slowly. That is fine for background jobs such as summaries, tagging or automations that nobody watches in real time.

For chat that feels instant, for larger models, or for more than one person at a time, you want the model in GPU memory (VRAM). Don't guess the number: open the VRAM calculator, pick the model and the quantization, and it tells you how much memory the weights and the context need and which cards fit.

Remember that the context window costs memory too. Long documents and many parallel chats can push a model that barely fits over the edge.

Renting a GPU by the hour

You do not need to buy a graphics card. GPU clouds rent single cards or whole multi-GPU machines by the hour, and many offer ready templates with Ollama, vLLM or a notebook already installed. The GPU table below shows current hourly prices for the cards we track.

Two habits keep the bill sane. First, stop or delete the instance when you are done - a month of 24/7 running is about 730 hours, so a forgotten pod adds up fast. Second, keep model files on a persistent volume so you do not download tens of gigabytes every time you start.

When an API is the cheaper choice

Be honest about your volume. If you send a few hundred requests a day, paying per token to a hosted model is usually cheaper than keeping a GPU warm, and you get a stronger model with no maintenance. Self-hosting wins when usage is heavy and steady, when data cannot leave your control, or when you need a specific open model.

Our Self-host vs API tool does the math for you: enter your expected tokens per month and it compares an hourly GPU against current API prices.

A simple first setup

Start small and grow only when you hit a limit.

• Rent a Linux server or GPU pod sized with the VRAM calculator.
• Install Ollama (or start a vLLM container) and pull one model.
• Keep the model port closed to the internet; reach it through a VPN, SSH tunnel or a web UI with a login.
• Add a reverse proxy with HTTPS if an app or teammates need access.
• Watch memory and response times for a week before you upgrade.