How to Build a Local AI Chat Assistant on Your Laptop in 30 Minutes
Install a local model runner, pick a small model that fits your laptop's memory, and start chatting offline in about half an hour — no cloud account required.
A good local chat assistant is not a downgrade. It is a smaller, quieter tool that keeps working when the Wi-Fi drops, holds your notes on your own disk, and costs nothing per message. You can have one running before your coffee goes cold. The build takes two downloads and a handful of typed commands, and you need no graphics card, no account, and no programming experience.
Here is the full path, from checking your hardware to typing your first prompt with the network cable unplugged.
What You're Building
Two parts do the work. A model runner is the program that loads a language model into memory and handles the back-and-forth. A model is the file of weights that generates the words. You install the runner once, then pull whichever model you want to try — swapping models is about as involved as switching browser tabs.
There is no account, no subscription, and no usage meter. Prompts, replies, and anything you paste in stay on your machine unless you deliberately wire up a tool that sends data elsewhere. That is the whole appeal: less magic, more ownership. The day-to-day experience is the same as any chat window you have used, except it is running three feet from your chair.
Check Your Laptop Before You Download Anything
Most laptops from the last five years can handle a small model. The constraint is memory, not raw speed.
- Memory: 8 GB of RAM is the realistic floor and runs a compact 3-billion-parameter model. With 16 GB you can run a 7- or 8-billion-parameter model comfortably. Below 8 GB, look for a 1–2 billion parameter model.
- Disk space: Models are large files. Keep 5–10 GB free, more if you want several installed.
- Processor: Apple Silicon, recent Intel and AMD chips, and most modern laptop CPUs are fine.
- Graphics: A dedicated GPU makes replies noticeably faster, but integrated graphics work. Expect a short pause on the first response while the model loads into memory.
If your machine struggles, choose a smaller model rather than a bigger one. Quality drops, but the assistant stays usable.
Step One: Install the Runner
Ollama is the friendliest starting point and runs on macOS, Windows, and Linux. Download the installer from the official project page, run it, and the runner starts as a background service that keeps models warm between sessions.
Open a terminal — Terminal on macOS, PowerShell on Windows — and confirm it is alive with ollama --version. If you see a version number, installation is done. If the shell says the command is not found, close the window and open a new one; the installer needs a fresh session to update your path. On Linux, the usual route is the install script from the project's documentation.
That is the hardest part of this whole process. Everything after it is typing.
Step Two: Choose a Small Model
Model names look cryptic at first. The number in the name is the parameter count, a rough measure of capacity, and it is the main lever on both quality and file size.
- Llama 3.2 3B — a solid default for general chat and writing help.
- Gemma 2 2B — the lightest of the group, quick on older hardware.
- Phi-3 Mini (3.8B) — concise, decent at short explanations and code snippets.
- Mistral 7B or Llama 3.1 8B — noticeably better answers if you have 16 GB of RAM.
As a rough guide, a 3-billion-parameter model downloads at around 2 GB and a 7- or 8-billion-parameter model at 4–5 GB, since most are distributed in a compressed form. To fetch one, type ollama pull llama3.2. The download happens once and stays cached. Use ollama list to see everything you have.
Step Three: Start Chatting
In the terminal
The command ollama run llama3.2 drops you into an interactive prompt. Type, press Enter, get a reply. A few built-in commands are worth memorising:
- /bye exits the session.
- /set system "You are a concise technical editor." changes the standing instructions that shape every reply.
- /set parameter temperature 0.4 makes output more predictable.
In a proper chat window
A terminal is fine, but a real interface is nicer. Open WebUI is the popular choice: it runs alongside the runner, detects your installed models automatically, and gives you saved conversations and multiple chats. It ships as a Docker container, so you need Docker Desktop installed first.
If Docker feels like a detour, use a desktop app instead. Jan, LM Studio, and Ollama's own app all provide a normal chat window and can use the models you have already downloaded.
Make It Yours
Once the basics work, three small adjustments separate a novelty from a tool you open every day.
- Write a real system prompt. "You are a careful assistant who answers in short paragraphs and asks for clarification when a request is vague" beats leaving the field blank.
- Lower the temperature for factual work. Around 0.2–0.4 suits summarising and editing; 0.7–0.9 suits brainstorming. High settings are more creative and less reliable.
- Bundle your preferences into a named model. A short Modelfile that names a starting model and a system prompt lets you create your own variant, then run it like any other, with your defaults baked in.
Worth knowing: the runner also exposes a local API on port 11434, so scripts and other apps on your machine can talk to it without a cloud key.
Where Local Models Fall Short
Small models are convenient, not omniscient. They invent citations, fumble arithmetic, and lose track of long documents. They cannot browse the web unless you give them a tool to do it, and their knowledge has a cutoff date. Treat them as a fast drafting and thinking partner, then verify anything that matters.
For health, money, or legal questions, a local assistant is a poor substitute for a qualified professional — talk to one.
And if you outgrow a 3B model? Pull something larger. Your laptop will tell you within a minute whether it can keep up.
Your Next Thirty Minutes
You now have offline chat. The fastest way to make it useful is to give it a job. Point it at a folder of your own notes and ask for a summary. Have it rewrite an email in a plainer tone. Ask it to explain a piece of code line by line.
Keep two models installed: a tiny one for quick questions and a larger one for when quality matters. Then switch off your Wi-Fi and send a prompt. If it answers, the build is finished, and nothing you typed left your laptop.
Photo: ricardorv30 / Pixabay
