What is a private LLM?
A private LLM (large language model) is an AI model you run on hardware you control — your laptop, desktop, Mac, mini-PC, or a private server — so prompts, documents, and outputs do not need to be sent to a public cloud AI product.
How it differs from ChatGPT, Copilot, Claude, Gemini
Cloud products run inference in the vendor’s environment. That delivers frontier capability, polished apps, and continuous model upgrades — with the trade-off that your content leaves your device (subject to each vendor’s terms, retention, and training policies).
Private/local setups download open (or licensed) weights, load them with a runner such as Ollama or LM Studio, and generate answers on your machine. You trade peak quality and convenience for data control, offline capability, and cost predictability after hardware.
How it works (simple path)
-
Choose a model tier that fits your hardware
Start smaller than a 27B. Tiny first chats: Gemma 4 E2B / E4B class, Phi-4-mini class, small Qwen tags. Laptop / 16 GB-class: a small-to-mid instruct tag. A ~27B Q4 (often Qwen3.8-class) is a later step on a 24 GB GPU or a high unified-memory Mac — verify the live card; do not treat 17 GB vs 24 GB as gospel.
-
Install a runner
Ollama is the fastest first win for most people; LM Studio if you want a GUI browser of models.
-
Download weights once
Models can be multi-gigabyte. Plan NBN time. After download, chat can be offline.
-
Run a first prompt
Confirm latency and quality before building RAG, agents, or multi-user servers.
Who private LLMs are for
- Businesses handling client or internal documents that should not train public models
- Legal, health-adjacent, finance, and professional services teams evaluating sovereignty options
- Developers working on private codebases (pair with vibecoding.au)
- Regional users who need offline capability when NBN is poor
- Hobbyists who want to learn how models actually run
Common misconceptions
- “Local means free forever.” Hardware, power, and time are real costs.
- “I installed Ollama, so nothing leaves the box.” False if you pick a
:cloudtag or a signed-in hosted model. Same window; data leaves. - “70B on a laptop is fine.” Usually heavy offload and painful speed. Start small.
- “Private equals secure.” Local malware, sloppy RAG permissions, and staff USB sticks still exist.
- “It matches ChatGPT, Claude, Gemini, or Grok.” Usually not on peak reasoning. That is the trade-off.