Which AI Models Can Run Locally? Memory, Costs and SME Use Cases

A model can look small on its specification sheet and still fail to fit your system: active parameter counts are not memory figures, while context and the runtime add further requirements. Make the decision only after a measured acceptance test.

In brief: AI models that can be run locally in 2026 range from Gemma 4 E2B/E4B and Qwen3.5-4B to gpt-oss-120b and Mistral Small 4, whose cited vendor hardware anchors are in the server class. Useful pilot tasks include extraction, classification, summarisation, coding, RAG, and image, document and speech processing. Two complete systems offered in Austria with 16 GB and 32 GB of dedicated VRAM were listed on the reference date at €3,415.85 net and €7,082.50 net, respectively. These are volatile retail snapshots, not a market price range or a compatibility guarantee. What matters is total cost of ownership (TCO) per professionally accepted output. For an SME, the right choice is therefore the smallest candidate that meets the SME’s requirements for quality, latency and rework. Size the hardware only after fixing the exact model artefact, quantisation, runtime, required context length and concurrency.

Four test tiers provide a useful shortlist: Gemma 4 E2B/E4B or Qwen3.5-4B, the 7–14B class, models around 20–32B, and large models from 70B total parameters upwards. This is neither a ranking nor a buying recommendation; test every candidate with the same business examples.

This article covers local inference: a ready-made model generates outputs on your organisation’s own or self-controlled hardware. It does not cover training large foundation models. “Local” does not automatically mean offline either—external tools, updates and data sources still need to be assessed separately.

Model, licence, product and price sources are current as at 15 September 2026. Vendor statements support shortlisting and do not replace tests on your hardware. Models, documentation, prices and terms can change.

Why model selection is an economic decision

The purchase price of a workstation does not tell you whether local AI is economical. What matters is the total cost over a defined period—the total cost of ownership (TCO)—and the number of outputs your business team actually accepts.

A small model can be inexpensive to load yet costly when employees must correct every output. A larger model may fit the task better but become uneconomical once hardware, electricity, operations and resilience are included. Conversely, an existing device may be sufficient for a pilot without making ongoing operation free.

The robust target metric is therefore not “price per GPU”, but:

Cost per professionally accepted output = TCO for the period / number of professionally accepted outputs.

Define the counting unit for each process before the test. For extraction, it might be one document approved by the responsible business reviewer; for coding, one patch merged after tests and review; for summarisation, one summary approved without a material omission. Demo calls without professional acceptance do not belong in the denominator.

Parameters, quantisation and RAM/VRAM: how to read the figures

Five terms prevent the most common selection errors:

• Total parameters are all weights contained in a model and normally required to be loaded.
• Active parameters describe only the subset computed per token in a mixture-of-experts (MoE) architecture. They are not a memory figure.
• Effective parameters are a vendor-specific term used, for example, for Gemma E2B/E4B. Always read them together with the total parameter count.
• Quantisation stores weights at a lower bit width and can thereby reduce memory requirements. According to its project documentation, llama.cpp supports quantisation from 1.5 to 8 bits as well as hybrid CPU/GPU offload; this creates neither a quality nor a speed guarantee for your device.
• Context and the KV cache require additional runtime memory for input and generated tokens. The runtime, buffers, multimodal encoders and concurrent sessions add further requirements.

For dense weights, the following calculation provides only a lower bound:

Weight memory ≈ parameters × bit width / 8

Twelve billion parameters at exactly 4 bits equal 6 GB arithmetically. That is neither total system RAM nor total VRAM. Format and runtime overhead, context and concurrency are not included. Do not apply this calculation to effective or active MoE parameters.

One concrete example illustrates the trap: with Gemma 4 26B A4B, about 4B parameters are activated per token, according to Google’s documentation, but all 26B weights are loaded for fast inference. Google’s loading estimates include 20% loading overhead, but expressly exclude supporting software and the context KV cache. They must therefore not be transferred to other model families.

System RAM, dedicated VRAM and unified memory are different memory architectures, not interchangeable figures on a specification sheet. CPU offload can make a model start when sufficient system RAM is available, but Mistral describes it as substantially slower in its offline documentation. A latency measurement determines whether that is sufficient for your process.

For the specifically loaded artefact, Ollama can report data including parameter size, quantisation level, context metadata, loaded size and size_vram. This telemetry describes the measured run—not every variant of the model.

Which AI models can you run locally? Four test tiers

The following matrix is a curated starting set, not an exhaustive model list. Memory figures are official vendor anchors with a narrow scope; missing values are intentionally left open.

Class and candidates	Parameters and modality	Official memory or hardware anchor	Useful in-house test	Hard limit and licence note
Small: Gemma 4 E2B/E4B, Qwen3.5-4B	Gemma E2B/E4B: text, image and audio input, text output; Qwen: 4B, image and text input, text output	Google gives approximately 2.9/4.5 GB for Gemma Q4_0, including 20% loading overhead but excluding software and the KV cache. No memory commitment is carried over for Qwen.	Extraction, classification, short drafts, and image and document tests	Test document type, language and accuracy yourself. Both model cards name Apache 2.0; re-check the exact artefact and terms.
7–14B: Gemma 4 12B	Text, image and audio input, text output	Google gives approximately 6.7 GB for Q4_0, including 20% loading overhead but excluding software and the KV cache.	More demanding structured-text and multimodal tasks	“Loads in 6.7 GB” does not mean “will safely fit in 8 GB”. Context and runtime add requirements. Model card: Apache 2.0; check the exact artefact.
20–32B: Gemma 4 26B A4B/31B, gpt-oss-20b, Devstral Small 2	gpt-oss: 21B total/3.6B active parameters, text-only, with function calling and structured outputs; Devstral: a dense 24B coding model	Gemma Q4_0: approximately 14.4/17.5 GB, including 20% loading overhead but excluding software and the KV cache; these figures apply only to the named Gemma 4 Q4_0 artefacts. Mistral places Devstral at 4-bit and 32K context in the 24 GB VRAM class. No memory figure is added for gpt-oss.	gpt-oss: structured outputs and tool calls with controlled permissions; Devstral: test it on your own code	Active parameters are not weight memory. The Devstral anchor applies only to 4-bit/32K. gpt-oss documentation names Apache 2.0; check licences and exact artefacts individually.
70B and larger: gpt-oss-120b, Mistral Small 4	gpt-oss: 117B total/5.1B active parameters and text-only; Small 4: 119B total/6B active per token, with text and image input	OpenAI describes gpt-oss-120b for a single NVIDIA H100. Mistral’s offline documentation states at least one H100 with 80 GB for Small 4 at FP8, and more memory for long contexts.	gpt-oss: text tasks with your own quality and latency criteria; Small 4: additionally image–text tasks	The cited vendor anchors are in the server class; measure alternative quantisation, offload or unified-memory configurations separately. Both sources name Apache 2.0; check the exact artefact.

Do not automatically start with the largest tier. Use the same dataset, acceptance criteria and latency target for every candidate. Only then can a model card become part of an investment case.

Curated test tiers based on the linked vendor sources, current as at 15 September 2026. A loading or hardware anchor is neither the total system requirement nor a compatibility guarantee.

Which tasks make sense locally? Measure instead of trusting a capability label

A label such as “vision”, “coding” or “function calling” establishes a technical capability, not fitness for your process. The acceptance test should differ by task:

Task	Measure in the pilot	Do not infer from the model card
Extraction and classification	Field accuracy, invalid-structure rate and rework time for each document type	autonomous approval or universal OCR quality
Drafting and summarisation	Factual errors, omissions and time to human approval	current or error-free facts
Coding	Project tests, patch acceptance and review time in your own repository	that a generic coding benchmark proves value for your project
RAG and company search	Source coverage and retrieval separately from citation fidelity, refusal behaviour and permissions	that an embedding model is a complete RAG system
Vision and OCR	Field and page accuracy by scan, layout and document class	that image capability means dependable document automation
Speech	Error rate and correction time by language, dialect, acoustic conditions and specialist vocabulary	that a VRAM figure implies equal quality across languages
Agentic workflow	Success rate for each tool step, permissions, abort, restart and human escalation	that function calling makes a process autonomously safe

RAG needs more than a chat model

With retrieval-augmented generation (RAG), internal content is first retrieved and assessed before a generative model answers. According to its own documentation, Qwen offers Qwen3 Embedding and Reranker in 0.6B, 4B and 8B variants with a 32K sequence length under Apache 2.0. Embeddings create semantic vectors; a reranker scores query–document pairs. Neither replaces the answer model, permissions, data rights or separate evaluation of retrieval and answer quality.

A specialist model may be more appropriate for speech

According to its upstream documentation, Whisper supports multilingual speech recognition, language identification and translation. The official variants state approximately 1–10 GB of required VRAM; performance remains highly dependent on language, acoustics and hardware. According to upstream, turbo was not trained for translation. The code and weights are licensed under MIT there—but the specific package you deploy still needs to be checked.

Hardware tiers with two Austrian price anchors

A meaningful purchase-price comparison requires the country, currency, tax basis, complete CPU/GPU/memory configuration, order status and date. The two prices below are documented point-in-time snapshots from Austria—not a general price range.

Tier	Documented configuration and source	Purchase price in Austria on 15 September 2026	Hard qualification
Existing device / small model	A real device with documented free memory, runtime, quantisation and sustained load; candidates could include Gemma 4 E2B/E4B or Qwen3.5-4B with reduced context	€0 additional purchase cost only if the device is genuinely free and available	Opportunity cost, setup, electricity, administration and rework still exist.
16 GB dedicated VRAM	Dell Austria: Alienware Aurora ACT1250, act1250_cto7: Core Ultra 7 265KF, RTX 5070 Ti with 16 GB GDDR7, 32 GB DDR5, 1 TB NVMe SSD, Windows 11 Home; shown as available to order	€3,415.85 net / €4,099.02 gross	Volatile retail/gaming configuration; not business equipment, a buying recommendation, a performance statement or a model/quantisation/context compatibility guarantee.
32 GB dedicated VRAM	Dell Austria: Alienware Area-51 AAT2250, aat2250_cto5: Core Ultra 9 285K, RTX 5090 with 32 GB GDDR7, 64 GB DDR5, 2 TB NVMe SSD, Windows 11 Home; shown as available to order	€7,082.50 net / €8,499.00 gross	Volatile retail/gaming configuration; not business equipment, a buying recommendation, a performance statement or a model/quantisation/context compatibility guarantee.
H100 or multi-accelerator class	gpt-oss-120b with OpenAI’s H100 anchor; Small 4 with Mistral’s FP8/H100 80 GB anchor	Individual, complete server quote required	Request GPU type and count, VRAM, CPU, RAM, SSD, OS, support, redundancy, cooling, power circuit/site, replacement strategy, country, currency, tax basis, delivery status and date in full.

The two published prices are only budget references and do not promise that a model will fit your runtime, context length, quantisation and concurrency. Before purchasing, obtain at least two current, comparable business quotes using the same checklist.

What local AI really costs: TCO, not just the graphics card

Local AI costs more than the price of a graphics card. It costs acquisition, evaluation, electricity, operations, downtime contingency and human rework—less residual value.

Electricity cost = measured average wall power (kW)
                   × operating hours
                   × own all-in electricity price (€/kWh)

TCO for the period = acquisition
                     + setup and evaluation
                     + electricity
                     + cooling and site
                     + administration, updates and monitoring
                     + support, spare parts and downtime reserve
                     + human review and rework
                     - residual value

Cost per professionally accepted output
  = TCO / number of professionally accepted outputs

Wogenfels cost model without an invented total or payback threshold. Use the same period, tax basis and company-specific figures for every option.

Compare an existing device, a hardware purchase and a cloud pilot over the same period and using the same tax basis. Every column needs the same cost categories; zero is permitted only when an item demonstrably does not apply.

TCO category	Existing device	Hardware purchase	Cloud pilot
Acquisition or use	Allocated device and opportunity cost	Full purchase price	The company’s own cloud-usage amount in euros for the same period, based on pilot consumption and invoice
Setup and evaluation	Internal and external labour	Internal and external labour	Internal and external labour
Electricity	Measured wall power	Measured wall power	Local load and separately billed infrastructure
Cooling and site	Actual allocated share	Actual allocated share	Additional items actually incurred
Administration, updates and monitoring	Actual effort	Actual effort	Actual effort
Support, spare parts and downtime reserve	Device-specific share	Supplier quote and internal reserve	Contract, internal reserve and consequences of outages
Human review and rework	Measured specialist time	Measured specialist time	Measured specialist time
Residual value	Subtract the allocated residual value	Subtract the expected residual value	Subtract only an actually existing residual value

Keep the time period and tax basis consistent across every quote and cost item. For electricity, use measured wall power and your organisation’s actual all-in business electricity price.

To make the calculation tangible, Statistics Austria reports an average of €0.293 gross per kWh for private use in the provisional year 2025. This is neither an SME tariff nor a forecast.

• 0.25 kW × 2,000 h × €0.293/kWh = €146.50 per year
• 0.60 kW × 8,760 h × €0.293/kWh = €1,540.01 per year

The second example assumes 24/7 operation. Both calculations exclude cooling, peak loads, idle use, UPS and outages. For your decision, replace both wall power and electricity price with measurements and invoice values from your business.

The five-step pilot before buying hardware

1. Define the task and acceptance criteria. Choose a specific process, representative examples, one unambiguous counting unit and a professional acceptance threshold. Record rework time and errors that require human escalation as well.
2. Choose the smallest plausible candidate. Start with the lowest model tier whose documented modality fits the task. A cloud pilot remains an alternative when suitable local hardware is not yet available.
3. Record the test configuration. Document the exact model artefact and its version or hash, quantisation, runtime version, hardware, context length and number of concurrent sessions. This is the only way to make the test repeatable.
4. Measure quality, latency and load together. Record the task-specific quality metrics, runtime, loaded size, the size loaded in VRAM (size_vram), wall power and human rework. Change only one factor in the configuration at a time.
5. Calculate TCO before deciding. Compare two complete business quotes, the existing-device option and a cloud pilot using the same TCO categories. Calculate the cost per professionally accepted output. The outcome may be a hardware purchase, reuse of an existing device, or a deliberate cloud/hybrid choice.

A useful measurement log therefore includes at least: model version or hash, quantisation, runtime version, hardware and memory architecture, context, concurrency, quality criteria, latency, wall power, and human review and rework time. Only this dataset makes variants comparable.

A reproducible Wogenfels pilot workflow. Measurements decide; the graphic promises neither an outcome nor a target architecture.

Licence, data protection and the boundary with training

Open weights are not automatically open source. The OSI’s Open Source AI Definition 1.0 requires freedoms to use, study, modify and share, as well as access to the preferred form for making modifications with sufficient information about data, code and parameters. This does not automatically classify any of the models named here. Before production use, re-check the licence, model card and terms of the exact artefact. This article mentions Apache 2.0 or MIT only where the respective primary source assigns it to the specific model or upstream project; that is not blanket commercial clearance.

Local does not automatically mean GDPR-compliant. If AI processes personal data, the GDPR remains applicable according to the Austrian Data Protection Authority; it specifically identifies data-protection principles and an appropriate legal basis as matters to assess. Local execution may reduce external data paths, but it does not replace an assessment of the concrete processing. For a deeper treatment of the architecture decision, see “Is local AI the only GDPR-compliant option? A fact check for SMEs”.

The hardware requirements in this article concern inference. Fine-tuning requires a separate estimate for memory, the data pipeline and operations. Training large foundation models is outside this comparison.

Conclusion: buy the evidence, not the largest parameter count

The right local AI option is the smallest candidate that passes your professional acceptance test—not always the smallest and not automatically the largest model. Parameter tiers and vendor information narrow the test set. The investment decision emerges only from the measured, documented test configuration, human rework and cost per professionally accepted output.

If you want to prepare that decision systematically, our AI solutions for businesses page provides the wider implementation context. To discuss a specific pilot, you can book a meeting. Bring the task, sample data you may lawfully use, details of existing hardware or supplier quotes, and your acceptance criteria. Those inputs can define a measurable pilot without promising a particular model or hardware tier in advance.

Source date and context

The source list contains 17 direct sources with an access date of 15 September 2026. Sixteen were directly available for content verification. EUR-Lex returned only a JavaScript/bot-protection page during the review, so it is used solely as the authoritative link to the GDPR text; this article derives no new granular legal interpretation from that page. Vendor sources are not independent performance or price comparisons. Re-check volatile model, licence, hardware and price details before publication or purchase.

• Runtimes: llama.cpp, Ollama API
• Gemma 4: model overview, model card
• OpenAI models: gpt-oss-20b, gpt-oss-120b
• Mistral: Mistral Small 4, offline models and hardware anchors
• Qwen: Qwen3.5-4B, Qwen3 Embedding and Reranker
• Speech: Whisper upstream
• Licence definition: Open Source AI Definition 1.0
• Cost reference: Statistics Austria—energy prices and taxes
• Data-protection framework: GDPR on EUR-Lex, Austrian Data Protection Authority on the GDPR and AI Act
• Austrian price anchors: Dell Alienware Aurora ACT1250, Dell Alienware Area-51 AAT2250