A local LLM is a large language model that performs inference on hardware you control, such as a laptop, desktop workstation or private server. Your prompt and the model’s response do not need to pass through a commercial AI provider’s API.

That can improve privacy, remove per-token charges and allow AI to work without an internet connection. It does not automatically make the entire workflow offline or secure. The chat interface, document index, plugins, update service or web-search tool may still contact external systems.

The practical test is therefore broader than asking where the model file is stored. You need to understand which components run locally, how much RAM or VRAM the workload consumes, what can still transmit data and when a private cloud deployment is more sensible than forcing a large model onto personal hardware.

Quick answer: A local LLM runs its model inference on your own device or server. Smaller quantised models can work on ordinary computers, while larger models need substantial RAM, GPU memory and power. A setup is only genuinely offline when the interface, embeddings, storage and connected tools also operate without external services.

A local LLM is a deployment pattern, not a special type of model

The term local LLM describes where a model runs, not how the model was trained. The same open-weight model may run locally on a laptop, on a private server or through somebody else’s hosted API.

A basic local setup has three components:

• Model weights: the files containing the model’s learned parameters.
• Inference runtime: software such as llama.cpp, Ollama or another engine that loads the weights and generates output.
• User interface or application: the chat window, coding assistant, document tool or custom application through which prompts are submitted.

Tools such as LM Studio, Jan, and the GPT4All package combine several of these components into a friendlier desktop application. Developers may instead run a command-line runtime or expose a local API to another application.

LocalAI, with that exact capitalisation, is also the name of a specific open-source project. This article uses local LLM for the broader deployment concept rather than that individual product.

---

---

Use a five-layer test to decide whether an LLM is really local

A local model badge tells you very little about the rest of the workflow. Check each layer separately.

Layer	What can run locally	What may still leave the device
Model inference	Prompt processing and token generation	A Cloud model may be selected instead, or a fallback may route failed requests remotely
Chat and file storage	Conversation history, uploaded documents and settings	Account sync, Cloud backup, crash reports or browser storage synchronisation
Retrieval and embeddings	Document parsing, vector database and local embedding model	Documents or extracted text sent to an external embedding API
Tools	Local files, scripts, databases and restricted applications	Web search, email, calendars, remote APIs and third-party plugins
Application services	Local interface and local API	Telemetry, update checks, model discovery, authentication and licence validation

A workflow can therefore use local inference while still sending sensitive information elsewhere. One common example is a local chat application that generates answers on-device but sends documents to a Cloud embedding service before the model sees them.

The reverse is also possible. A browser interface may run locally while all inference happens through a remote API. The location of the interface is not evidence of where the model runs.

Local, offline, self-hosted and private Cloud mean different things

Term	Where inference runs	Internet required during use?	Main limitation
Local LLM	Your laptop, desktop or on-site server	Not necessarily	Connected tools and application services may remain online
Offline LLM	Local hardware	No, once all required files are installed	No live search, Cloud integrations or remote model access
Self-hosted LLM	Hardware or Cloud infrastructure you administer	Usually, if accessed remotely	You own patching, access control, monitoring and availability
Private cloud LLM	A dedicated server, tenancy or controlled Cloud environment	Yes	Data still leaves the user’s device and passes through cloud infrastructure
Air-gapped LLM	An isolated local system or network	No network path available	Model updates, maintenance and data transfer become more difficult

An offline AI system is therefore a stricter form of local deployment. You may need an internet connection to download the runtime and model initially, but inference can continue after the network is removed.

How much RAM or VRAM does a local LLM need?

Memory requirements depend on four main components:

• The number of model parameters
• The precision or quantisation used for the weights
• The selected context length
• The runtime overhead and number of simultaneous requests

A rough calculation for model weights is:

parameters × bits per parameter ÷ 8

An 8-billion-parameter model stored at 4 bits per parameter therefore needs roughly 4 GB for the raw weights. The actual model file and runtime memory will be higher because quantisation metadata, buffers, context and application overhead must also fit.

Model class	Approximate 4-bit weight size	Practical available memory target	Typical use
7B to 8B	4 GB to 5 GB	8 GB to 12 GB	Basic chat, extraction, summarisation and lighter coding
12B to 14B	7 GB to 10 GB	12 GB to 16 GB	Stronger writing, instruction following and coding
30B to 32B	18 GB to 22 GB	24 GB to 32 GB	More demanding reasoning and specialist workflows
70B	40 GB to 50 GB	48 GB to 64 GB or more	High-end local inference where quality justifies the hardware

These are planning ranges, not guaranteed specifications. Model architecture, quantisation format, context length and inference engine can move the requirement significantly.

Our AI model comparison can help with capability and cost selection, but local deployment adds another filter: the best model on a benchmark is irrelevant if it cannot run at a usable speed on your hardware.

RAM, VRAM and unified memory are not interchangeable

A discrete graphics card has its own VRAM. Keeping the full model in VRAM normally produces much faster generation than running it from system RAM. Some runtimes can split model layers between GPU and CPU memory, but heavy CPU offloading often turns an impressive specification into a slow interactive experience.

Apple Silicon uses unified memory shared between the CPU and GPU. That makes a larger portion of total memory available to the model, although macOS and other applications still need headroom. A 16 GB Mac does not provide the model with an untouched 16 GB pool.

CPU-only inference is possible, particularly with smaller quantised models. It is useful for experimentation, background document processing and low-volume tasks where response time is not critical. It becomes frustrating for long answers, large models or agent workflows that make repeated calls.

Context length is the memory cost buyers often miss

The model weights are only part of the calculation. A runtime also allocates memory for the context and attention state used while processing the conversation. Longer documents, large codebases and extended chat histories can therefore trigger an out-of-memory error even when the model loaded successfully at first.

Runtime behaviour varies, so check the Ollama context-length documentation or the equivalent documentation for the engine you use before setting an ambitious context window.

A recurring practical mistake is buying hardware based on the downloaded model size, then discovering that context, operating-system use, and GPU buffers consume the remaining memory. Another is setting the maximum advertised context on day one. That reserves resources for tokens the workflow may never need and can reduce speed or prevent full GPU loading.

What privacy advantages does local inference provide?

A properly configured local LLM can remove several exposure points associated with a managed AI service:

• Prompts and responses do not need to be submitted to an external inference API.
• Documents can remain on hardware under your physical or administrative control.
• You decide how conversations are retained, backed up and deleted.
• Processing can continue without an account, subscription or provider outage.
• Highly sensitive workloads can be separated from public internet access.
• There is no usage-based model fee for each prompt after the hardware and electricity are paid for.

This is particularly useful for source code, unpublished documents, internal research and personal records that should not be copied into a consumer chatbot.

Local processing is not a compliance shortcut. Access permissions, malware protection, full-disk encryption, audit logs, backups and retention controls still determine who can see the data. A badly secured laptop may provide weaker protection than a well-governed business AI service.

Hidden privacy risks can undermine an offline AI setup

Plugins and agent tools can transmit prompt data

A local model may call a search engine, remote database, email service or external application. The tool receives whatever query the agent sends, which may include text from the conversation or uploaded documents.

Grant tools narrowly. A summarisation assistant does not need email access, shell execution and a writable home directory simply because the software supports them.

Embeddings may be generated remotely

Document chat usually has a separate indexing stage. Text is split into chunks and converted into embeddings before being stored in a vector database. Confirm that the embedding model and vector database are local as well as the conversational model.

Telemetry and update checks still create network traffic

Desktop applications may check for updates, report crashes or collect anonymous usage information. This does not necessarily mean prompt contents are transmitted, but it means the system is not completely offline.

For a strict deployment, disable optional telemetry, control outbound network access and review the application’s privacy settings rather than trusting the word “local” on a download page.

Model and plugin downloads introduce supply-chain risk

Download models from known publishers and verify that the repository, file type and licence match what you expect. Be particularly cautious with applications that download custom code alongside model weights or instruct you to enable remote code execution.

Plugins deserve even closer review because they are executable software. A local model cannot protect files from a plugin that has already been granted access to them.

A local API can accidentally become a network service

Many runtimes expose an HTTP endpoint so other applications can use the model. Binding that endpoint to every network interface can make it reachable by other devices on the local network or, on a misconfigured server, the public internet.

Keep the service bound to loopback unless remote access is intentional. Where other devices need access, add authentication, firewall restrictions and encrypted transport rather than publishing an unauthenticated model port.

History and backups may leave the machine

Conversation databases, uploaded documents and generated files can be copied by Cloud backup software even when inference is local. Check browser synchronisation, operating-system backup folders, shared drives and automated workstation backups.

How to verify that offline AI really works offline

1. Install the model and runtime fully. Confirm that no additional files are downloaded when the first prompt is submitted.
2. Disconnect the network. Restart the application and run the same task rather than relying on a session that was already open.
3. Test every workflow component. Chat, document retrieval, embeddings, speech recognition and image analysis may use different providers.
4. Remove Cloud API keys temporarily. This exposes silent fallback behaviour that a simple internet-disconnection test may miss on a networked workstation.
5. Disable web search and external tools. A model cannot provide live information offline unless the relevant data has already been stored locally.
6. Check where histories and indexes are stored. Identify the database, document directory and vector store rather than assuming they disappear when a chat is deleted.
7. Inspect the listening address. A service on 127.0.0.1 is local to the machine. A service on 0.0.0.0 may be reachable from other network interfaces.
8. Repeat the test after updates. New integrations, cloud-model options and changed defaults can alter a previously local workflow.

Local LLM or private Cloud: choose by workload, not ideology

Decision factor	Local workstation	Private cloud or dedicated server
Best fit	One user, private documents, experimentation and intermittent use	Teams, remote access, applications and concurrent workloads
Privacy boundary	Data can remain on the user’s device	Data leaves the device but stays within controlled infrastructure
Performance	Limited by purchased hardware and thermal constraints	Wider choice of GPUs and easier scaling
Availability	Depends on the workstation being switched on	Can run continuously with monitoring and recovery
Administration	Simple for one machine	Requires identity, networking, patching and cost controls
Cost pattern	Upfront hardware plus electricity	Recurring compute, storage and data-transfer charges

Private Cloud is often the better compromise when the model is too large for a laptop or needs to serve several people. DIY AI’s comparison of GPU hosting for AI inference covers the infrastructure choices and the billing traps that appear once a model must remain available remotely.

Do not compare only the cost of a graphics card with an API token rate. Include electricity, idle capacity, maintenance, backups, setup time and the cost of replacing hardware. For occasional prompts, a managed API may remain cheaper for years. Local economics improve when usage is consistent, privacy has operational value, or the hardware also serves other workloads.

When offline AI becomes impractical

Running locally is a poor fit when the workload requires:

• Frontier-model quality that smaller open-weight models cannot reproduce
• Very large contexts for repositories, legal archives or extensive research collections
• High-quality multimodal processing across text, images, audio and video
• Many simultaneous users or agent processes
• Continuous availability from phones, offices and remote locations
• Frequent model changes without repeated downloads and testing
• Fast generation from models that exceed the available GPU memory
• Centralised user management, auditing and support

Offline models also lose access to live information unless you deliberately import it. Adding web search solves that limitation but changes the privacy boundary. The system is no longer fully offline, and search queries may disclose part of the task.

The strongest local model is not always the most useful local model. A responsive 8B or 14B model that completes a narrow task reliably can provide more value than a 70B model that takes minutes to answer, overheats the machine or leaves no memory for the application using it.

A sensible first local LLM setup

1. Start with a 7B or 8B instruction model at 4-bit quantisation. This gives you a useful hardware baseline without committing to an expensive workstation.
2. Use a mature runtime. Ollama and LM Studio offer approachable starting points, while llama.cpp provides more direct control.
3. Begin with a modest context length. Increase it only after a real task is being truncated.
4. Test your own prompts. Benchmark extraction, writing, coding or document tasks that represent the actual workload.
5. Measure first-token delay and generation speed. A model that fits but responds too slowly has not passed the hardware test.
6. Run the offline verification process. Check inference, embeddings, tools, storage and network listeners separately.
7. Upgrade against an identified bottleneck. Buy more memory for model capacity, a faster GPU for generation speed or a server for shared availability. Do not upgrade all three without knowing which problem you are solving.

Common local LLM mistakes

• Buying for the largest model available: model quality does not rise evenly with parameter count, while hardware cost and latency can rise sharply.
• Confusing system RAM with GPU memory: 64 GB of desktop RAM does not make an 8 GB graphics card behave like a 64 GB GPU.
• Judging fit by model-file size: context cache, runtime buffers and the operating system also require memory.
• Assuming a local interface means local inference: confirm the selected provider and model endpoint.
• Exposing the local API without protection: a private model can become a shared network service through one careless bind setting.
• Turning on every integration: web search, plugins and remote embeddings can defeat the original privacy objective.
• Ignoring the model licence: downloadable does not always mean unrestricted for commercial use, redistribution or hosted services.
• Expecting frontier performance from entry-level hardware: local deployment changes the privacy and cost model, not the underlying capability of the chosen model.

Local LLM FAQs

Can a local LLM work without the internet?

Yes. Once the runtime and model files have been downloaded, a local LLM can generate responses without internet access. Web search, Cloud embeddings, account authentication and connected tools may stop working unless they also have local replacements.

Is a local LLM completely private?

Not automatically. Inference may be private while telemetry, plugins, backups, document embeddings or web tools transmit data. Privacy depends on the complete application path, access controls and storage configuration.

How much RAM do I need to run a local LLM?

Eight to 16 GB is enough to begin experimenting with smaller quantised models. Models around 30B often need 24 to 32 GB of available memory, while a 70B model commonly requires 48 to 64 GB or more. Context length and runtime overhead can increase these figures.

Can I run a local LLM without a GPU?

Yes. CPU-only inference works with quantised models, but generation is usually slower. It is reasonable for experimentation and background processing, while interactive coding, agents and long responses benefit considerably from GPU acceleration.

Is self-hosted AI the same as local AI?

No. A self-hosted model may run on a rented Cloud server rather than the user’s device. You administer the deployment, but prompts still travel across a network and are processed on third-party infrastructure.

Are local LLMs free?

Many runtimes and open-weight models can be downloaded without a subscription, subject to their licences. Deployment still has costs, including hardware, electricity, storage, maintenance, and the time required to configure and secure it.

Should you run a local LLM?

Use a local LLM when keeping prompts and files under your control is more valuable than having the strongest available model, effortless remote access or unlimited scale. Start with a smaller quantised model, prove that it handles the task and verify every network-connected component before calling the workflow private.

Choose private Cloud or a managed API when several users need reliable access, the workload requires larger models or maintaining inference infrastructure would consume more time than it saves. The sensible boundary is not local at any cost. It is the smallest deployment that meets the required privacy, capability, speed and operating burden.