How to set up a private LLM (beginner guide)

v1.3.0 · Last verified: 25 September 2026 · Pattern guide — verify official docs

Independence: Independent educational resource. Not affiliated with Meta, Alibaba, Mistral, Google, Microsoft, OpenAI, Anthropic, xAI, NVIDIA, Apple, Ollama, LM Studio, or any model, tool or hardware maker. privatellm.app and other “PrivateLLM” products are unrelated. Accuracy: content is educational and may be incomplete or outdated — verify official docs and live prices before you buy or deploy. See disclaimer and affiliate disclosure.

Educational content only. Content on privatellm.au is general educational information about private and local large language models, software tools, hardware classes, and related topics. It is written for a general audience in Australia and is not personalised advice for your business, product, security posture, employment, or legal situation.

May be incomplete or outdated. We work to keep guides useful, but we do not guarantee completeness, currency, or accuracy of any model name, command, specification, price range, licence summary, or third-party feature. Always verify against official documentation and current retailer listings before you buy, deploy, or process sensitive data. If something looks outdated, treat official sources as correct.

This page: Installers, CLI flags, and library model names change. Download software only from official project sites. We do not host models or guarantee any pull tag.

Disclaimer · Affiliate disclosure · Terms · Privacy

Goal: one successful local chat on hardware you already own. Upgrade GPUs only after that win — not before.

Prerequisites

Decision check (60 seconds)

1. Privacy / sensitivity need

High (regulated data, legal/health/finance, client privilege, air-gapped) → private strongly preferred. Medium → hybrid often sensible. Low → cloud often wins on capability.

2. Budget & existing hardware

Start with what you own. Low extra budget → small models. Mid budget → stronger GPU or more unified memory for mid-size models. High budget → workstation-class options. Exact B-counts and prices move — treat as bands, not quotes.

3. Primary use-case

Casual chat/summaries; coding & agents (see vibecoding.au); RAG over private docs; fine-tuning / multi-user serving; fully offline / air-gapped.

4. Honest exit ramps

Need frontier reasoning daily with zero hardware budget, or a non-technical team needing polished multimodal → stay on ChatGPT/Claude Team (or hybrid) and revisit later.

Step-by-step pattern

  1. Choose a small model tier

    Existing laptop/Mac: start with a small instruct-style model from your runner’s library. If replies are unusably slow, go smaller — do not jump to “biggest model” guides. Hardware notes: hardware page.

  2. Install a runner — Ollama first (or LM Studio for a GUI)

    Download only from the official Ollama or LM Studio websites. Keep this guide open in a second window while you install. After install, confirm the CLI (output varies by version):

    Example check (if you installed Ollama) 
    ollama --version

    Local weights vs Ollama Cloud

    Ollama can run a model on your machine, or send the prompt to Ollama’s cloud under a :cloud tag or a signed-in hosted model. The window looks the same. The data flow is not. If privacy is the point, download a local tag and leave cloud models alone. Official docs: docs.ollama.com/cloud. On a Mac you can also look at MLX later — see Tools. Windows beginners stay on Ollama or LM Studio.

  3. Download a first model from the current library

    Open the official model library. Start smaller than you think: one instruct tag you can finish downloading tonight, one prompt, then stop. As of late September 2026, families you will often see include small Qwen and Gemma 4 (E2B / E4B / 12B-class), Phi-4-mini class, gpt-oss, and Llama. The ~27B-class (often Qwen3.8) is a second sitting, on a 24 GB-class GPU or a high unified-memory Mac — verify the live card. Commands look like this pattern — replace MODEL_NAME with a real name from today’s library (we do not freeze tags). Family map: Models.

    Pattern only — use a live library name 
    ollama pull MODEL_NAME

    Large downloads: start overnight; confirm free disk space first.

  4. Run your first prompt

    Pattern only 
    ollama run MODEL_NAME "Summarise in two sentences why someone might run an LLM locally."

    Evaluate honestly: acceptable speed? Coherent answers? If it crawls, drop model size before buying hardware.

  5. Optional: nicer UI later

    Browser UIs (for example Open WebUI-class projects) and local APIs come after a working first chat. See Tools. For building apps with AI coding helpers, see vibecoding.au.

Offline / air-gapped notes

Once weights are on disk, many runners work without internet. Cloud tags will not. A :cloud or signed-in hosted model needs the network and sends the prompt off the machine. True air-gapped environments need a secure transfer path for models and a patch process for the host OS. Do not improvise USB workflows for regulated environments without your security team’s process. This site does not provide an air-gap certification checklist.

Troubleshooting (common, not exhaustive)

FAQ

Ollama or LM Studio first?

Ollama if you are happy with a terminal or simple API. LM Studio if you want a desktop GUI to browse and chat. Both can coexist later.

Which first model should I pull?

A small, well-documented instruct model from your runner’s *current* library that fits your RAM/VRAM comfortably. Confirm a successful first chat before chasing large models. Do not rely on blog posts for exact pull tags — they go stale.

Why is my model slow?

Common causes: model larger than VRAM (heavy CPU offload), thermal throttling, spinning disk, or too many background apps. Drop quant size or model parameter count first.

Can I use this offline?

Yes after models are downloaded. Air-gapped setups need a transfer path for weights and careful patch discipline — plan that before you disconnect.

Next steps

Hardware orientation Model families Business notes